Advertisement

Implementation of Magnetic Resonance Imaging-Guided Radiation Therapy in Routine Care: Opportunities and Challenges in the United States

Open AccessPublished:May 25, 2022DOI:https://doi.org/10.1016/j.adro.2022.100953

      Abstract

      Purpose

      Magnetic resonance image (MRI)-guided radiation therapy with the 1.5 Tesla magnetic resonance linear accelerator (MR-Linac) is a rapidly evolving and emerging treatment. The MR-Linac literature mainly focused on clinical and technological factors in technology implementation, but it is relatively silent on health care system-related factors. Consequently, there is a lack of understanding of opportunities and barriers in implementing the MR-Linac from a health care system perspective. This study addresses this gap with a case study of the US health care system.

      Methods and Materials

      An exploratory, qualitative research design was used. Data collection consisted of 23 semistructured interviews ranging from clinical experts at the radiation therapy and radiology department to insurance commissioners in 7 US hospitals. Analysis of opportunities and barriers was guided by the Nonadoption, Abandonment, Scale-up, Spread and Sustainability framework for new medical technologies in health care organizations.

      Results

      Opportunities included high-precision MR-guidance during radiation therapy with potential continued technical advances and better patient outcomes. MR-Linac also offers opportunities for research, professional, and economic development. Barriers included the lack of empirical evidence of clinical effectiveness, technological complexity, and large staffing and structural investments. Furthermore, the presence of patients with disadvantaged socioeconomic background, and the lack of appropriate reimbursement as well as regulatory conditions can hinder technology implementation.

      Conclusions

      Our study confirms the current literature on implementing the MR-Linac, but also reveals additional challenges for the US health care system. Alongside the well-known clinical and technical factors, also professional, socioeconomic, market, and governing influences affect technology implementation. These findings highlight new connections to facilitate technology uptake and provide a richer start to understanding its long-term effect.

      Introduction

      Defining the outcomes of new medical technology upfront implementation is important to identifying its potential benefits and obstacles.
      • Greenhalgh T
      • Wherton J
      • Papoutsi C
      • et al.
      Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
      • Smith GL
      • Ganz PA
      • Bekelman JE
      • et al.
      Promoting the appropriate use of advanced radiation technologies in oncology: Summary of a National Cancer Policy Forum Workshop.
      • Rapport F
      • Clay-Williams R
      • Churruca K
      • Shih P
      • Hogden A
      • Braithwaite J.
      The struggle of translating science into action: Foundational concepts of implementation science.
      Insight into comparative effectiveness of new medical technology to current standard of care is critical to ensure patient access to high-quality health care at the lowest possible cost levels.
      • Makady A
      • Ham R ten, de Boer A
      • Hillege H
      • Klungel O
      • Goettsch W.
      Policies for use of real-world data in health technology assessment (HTA): A comparative study of six HTA agencies.
      • Clancy CM
      • Cronin K.
      Evidence-based decision making: Global evidence, local decisions.
      • Li SA
      • Jeffs L
      • Barwick M
      • Stevens B.
      Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: A systematic integrative review.
      • Vandeusen Lukas C
      • Engle RL
      • Holmes SK
      • et al.
      Strengthening organizations to implement evidence-based clinical practices.
      • Jacobs M
      • Boersma L
      • Dekker A
      • et al.
      What is the impact of innovation on output in healthcare with a special focus on treatment innovations in radiotherapy? A literature review.
      Many new medical technologies, however, lack a comprehensive effectiveness evaluation before being implemented in the clinical setting, hence, without proven effectiveness.
      • Clancy CM
      • Cronin K.
      Evidence-based decision making: Global evidence, local decisions.
      ,
      • Li SA
      • Jeffs L
      • Barwick M
      • Stevens B.
      Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: A systematic integrative review.
      ,
      • Bertram RM
      • Suter JC
      • Bruns EJ
      • O'Rourke KE
      Implementation research and wraparound literature: Building a research agenda.
      This is problematic because the implementation of unproven medical technologies may threaten the quality of health care and increase costs.
      • Calabrò GE
      • La Torre G
      • De Waure C
      • Villari P
      • Federici A
      • Ricciardi W
      • Specchia ML
      Disinvestment in healthcare: An overview of HTA agencies and organizations activities at European level.
      Consequently, payers and policymakers are reluctant to approve or reimburse costly medical innovations.
      Despite these concerns, especially with regard to medical technology still in development, it can be difficult to estimate the relative effectiveness, nor its long-term effect on the patient, provider and payer.
      • Adang EMM
      • Wensing M.
      Economic barriers to implementation of innovations in health care: Is the long run-short run efficiency discrepancy a paradox?.
      This is particularly relevant for convergent technologies, which tent to cross disciplinary boundaries and are becoming dominant in health-related domains.
      • Thune T
      • Mina A.
      Hospitals as innovators in the health-care system: A literature review and research agenda.
      For instance, the convergence between biology and informatics knowledge has led to bioinformatics and personalized medicine innovations.
      • Roco MC.
      Nanotechnology: Convergence with modern biology and medicine.
      The involvement of different disciplines means that convergent medical innovation and its potential value, cannot be understood and implemented without intertwined institutional practices.
      • Lounsbury M
      • Crumley ET.
      New practice creation: An institutional perspective on innovation: Michael Lounsbury and Ellen T. Crumley.
      ,
      • Moors EHM
      • Kukk Fischer P
      • Boon WPC
      • Schellen F
      • Negro SO
      Institutionalisation of markets: The case of personalised cancer medicine in the Netherlands.
      Early assessment of crucial factors of technology implementation is therefore needed for an initial understanding of its long-term impact.
      • Markiewicz K
      • Van Til JA
      • Ijzerman MJ.
      Medical devices early assessment methods: Systematic literature review.
      A modern example of a new developing and convergent technology in the field of radiation oncology is the 1.5 Tesla MRI-guided linear accelerator (MR-Linac) system, which combines high-precision, real-time MRI external beam radiation therapy (MRgRT).
      • Hall WA
      • Paulson ES
      • van der Heide UA
      • et al.
      The transformation of radiation oncology using real-time magnetic resonance guidance: A review.
      • Raaymakers BW
      • Jürgenliemk-Schulz IM
      • Bol GH
      • et al.
      First patients treated with a 1.5 T MRI-Linac: Clinical proof of concept of a high-precision, high-field MRI guided radiotherapy treatment.
      • Hehakaya C
      • Van der Voort van Zyp JR
      • Lagendijk JJW
      • et al.
      Problems and promises of introducing the magnetic resonance imaging linear accelerator into routine care: The case of prostate cancer.
      As a result, the irradiation plan can be adjusted at any time based on these changes. This allows more targeted delivery of radiation to the tumor and avoids the healthy tissue surrounding the tumor compared with most conventional radiation therapy techniques.
      • Adang EMM
      • Wensing M.
      Economic barriers to implementation of innovations in health care: Is the long run-short run efficiency discrepancy a paradox?.
      • Thune T
      • Mina A.
      Hospitals as innovators in the health-care system: A literature review and research agenda.
      • Roco MC.
      Nanotechnology: Convergence with modern biology and medicine.
      Therefore, the MR-Linac may reduce radiation therapy induced toxicity and improve tumor control outcomes.
      • Hall WA
      • Paulson ES
      • van der Heide UA
      • et al.
      The transformation of radiation oncology using real-time magnetic resonance guidance: A review.
      ,
      • Hehakaya C
      • Van der Voort van Zyp JR
      • Lagendijk JJW
      • et al.
      Problems and promises of introducing the magnetic resonance imaging linear accelerator into routine care: The case of prostate cancer.
      The MR-Linac also allows therapy in fewer sessions with a higher dose of radiation therapy, also called hypofractionation, hence, permitting a shorter treatment.
      • Kothari G
      • Loblaw A
      • Tree AC
      • van As NJ
      • Moghanaki D
      • Lo SS
      • Ost P
      • Siva S.
      Stereotactic body radiotherapy for primary prostate cancer.
      ,
      • Pathmanathan AU
      • van As NJ
      • Kerkmeijer LGW
      • et al.
      Magnetic resonance imaging-guided adaptive radiation therapy: A “game changer” for prostate treatment?.
      This can be beneficial in a case like localized prostate cancer

      Alongi F, Rigo M, Figlia V, Cuccia F, et al. 1.5T MR-guided and daily adapted SBRT for prostate cancer: Feasibility, preliminary clinical tolerability, quality of life and patient-reported outcomes during treatment. 2020;94:20200848.

      • Cuccia F
      • Corradini S
      • Mazzola R
      • et al.
      MR-guided hypofractionated radiotherapy: Current emerging data and promising perspectives for localized prostate cancer.
      • Cuccia F
      • Alongi F
      • Belka C
      • et al.
      Patient positioning and immobilization procedures for hybrid MR-Linac systems.
      because traditional external beam radiation therapy often varies from 5, 20, to 39 treatment sessions.
      • Nicosia L
      • Mazzola R
      • Rigo M
      • et al.
      Moderate versus extreme hypofractionated radiotherapy: A toxicity comparative analysis in low- and favorable intermediate-risk prostate cancer patients.
      Alongside technical and clinical opportunities, the MR-Linac may also offer positive professional and economic prospects as identified by a Dutch implementation study.
      • Hehakaya C
      • Van der Voort van Zyp JR
      • Lagendijk JJW
      • et al.
      Problems and promises of introducing the magnetic resonance imaging linear accelerator into routine care: The case of prostate cancer.
      For instance, technology adopters may gain a more efficient treatment, a higher hospital quality profile with potential financial benefits, and improve their competence and technical expertise as well as multidisciplinary collaboration. However, technology implementation may also deal with technical complexities, substantial staffing and structural investments, and the presence of patient referral patterns and professional silos. In addition, we still know very little about the actual patient benefit of MRgRT due to the current lack of empirical evidence of clinical effectiveness.
      The lack of effectiveness evidence has different implications across health care systems because the dynamics of stakeholders, organizational procedures, dominant existing routines, professional identities, and legal and regulatory standards in technology implementation vary between countries.
      • Greenhalgh T
      • Wherton J
      • Papoutsi C
      • et al.
      Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
      ,
      • Caccia-Bava MDC
      • Guimaraes T
      • Harrington SJ.
      Hospital organization culture, capacity to innovate and success in technology adoption.
      • Lettieri E
      • Masella C.
      Priority setting for technology adoption at a hospital level: Relevant issues from the literature.
      • Pope C
      • Halford S
      • Turnbull J
      • Prichard J
      • Calestani M
      • May C.
      Using computer decision support systems in NHS emergency and urgent care: Ethnographic study using normalisation process theory.
      For instance, developing and acting upon evidence-based practices is perceived to be more challenging in health care systems with large private payers such as those in the United States, than in publicly funded health care systems as in Europe or Canada.
      • Clancy CM
      • Cronin K.
      Evidence-based decision making: Global evidence, local decisions.
      A fragmented and disjointed public and private health care hybrid in the United States can result in substantial variations in health care delivery.
      • Clancy CM
      • Cronin K.
      Evidence-based decision making: Global evidence, local decisions.
      This is particularly relevant for MR-Linac systems as several have been installed worldwide, with a considerable number currently operational in the United States (US). Hence, it is interesting to study how to facilitate the implementation of the MR-Linac technology in specific health care systems such as the US. In the present study, we aim to identify the opportunities and barriers to the implementation of the MR-Linac technology in US hospitals.

      Methods and Materials

      We conducted a qualitative study with semistructured interviews, a method most appropriated to make sensitive issues, attitudes, opinions, and experiences of individuals explicit.
      • Bryman A.
      Social Research Methods.

      Data collection

      We used the Nonadoption, Abandonment, Scale-up, Spread and Sustainability (NASSS) framework of new technologies to explore the factors of success and failure of technology adoption in health care organizations.
      • Greenhalgh T
      • Wherton J
      • Papoutsi C
      • et al.
      Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
      The NASSS framework considers 7 domains: (1) the targeted clinical indication; (2) the technology to be implemented; (3) the value proposition; (4) the adopter system (patient, technology user and other staff); (5) the providing organization; (6) the wider institutional social context; and (7) organizational resilience and technology development over time.
      • Greenhalgh T
      • Wherton J
      • Papoutsi C
      • et al.
      Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
      We conducted 23 semistructured interviews in 2020 to identify the opportunities and barriers to MR-Linac implementation in 7 US hospitals (Table 1). A questionnaire was designed based on the NASSS framework (Appendix E1). Interviewees included radiation oncologists, urologists, radiologists, medical physicists, radiation therapy technologists, and dosimetrists, as well as nonclinical hospital administrators, including strategic and financial department managers as well as insurance councils. We included interviewees from academic and community-based hospitals, to avoid professional biases. The study was approved by the local medical university research ethics board and all study participants provided verbal consent.
      Table 1Overview of roles and affiliations of respondents
      RespondentPositionSeniorityAffiliationAdditional rolesMethodDuration (min)
      R1Head of Imaging and Oncology departmentFull professorAMC1Research on functional imagingIn person44
      R2Radiation therapy technologistSeniorAMC1Research on MR-LinacIn person43
      R3Insurance commissionerSeniorAMC1In person46
      R4Head of IT for OncologySeniorAMC1Research on functional imagingIn person44
      R5Medical physicistSeniorAMC1In person41
      R6Nuclear medicine physicianSeniorAMC1In person39
      R7Member Board of DirectorsSeniorAMC1In person44
      R8Member Board of DirectorsSeniorAMC2In person44
      R9Radiation oncologistSeniorAMC3In person39
      R10Radiation oncologistSeniorAMC4In person39
      R11Radiation oncologistSeniorAMC2Head of Radiation Oncology department, Research on MR-LinacIn person45
      R12Radiation oncologistSeniorAMC2Research on MR-LinacIn person53
      R13Radiation oncologistSeniorAMC2Research on MR-LinacIn person39
      R14Radiation oncologistSeniorAMC5Research on MR-LinacVirtual41
      R15UrologistSeniorAMC4Head of Urology departmentIn person40
      R16UrologistSeniorMC1In person29
      R17Radiation therapy technologistSeniorAMC1In person41
      R18RadiologistProfessorAMC1In person35
      R19Radiation oncologistSeniorAMC4Head of Radiation Oncology departmentVirtual39
      R20Radiation oncologistSeniorAMC2Research on MR-LinacIn person45
      R21Radiation oncologistFull professorAMC2In person48
      R22Radiation oncologistSeniorMC2In person41
      R23Market access associateSeniorManufacturing companyVirtual46
      Abbreviations: AMC = academic medical center; MC = (nonacademic) medical center.
      The research objective was explained in the written invitation and at the start of each interview. All interviews were audio-recorded and transcribed using Sonix transcription software. Interviews lasted between 45 and 60 minutes and were carried out face-to-face and via telephone and Skype. Audio recordings and transcript of interviews are anonymized as well as confidential and therefore not publicly available.

      Data analysis

      Interview transcripts were analyzed using NVivo software. We applied open coding based on the research objective and ended with axial coding, identifying areas of theoretical interest and common themes. Developed codes were validated by a second reviewer. To include variation in findings and increase construct validity, we triangulated the findings across different respondents and with literature.

      Results

      We identified 4 opportunity and 5 barrier categories for implementation of the MR-Linac technology in US hospitals.

      Opportunities

      We first describe the clinical opportunities before moving on to the technological, adopter, and economic opportunities. Figure 1 shows the percentages of the interview cohort who discussed the opportunities, by main theme and subtheme.
      Fig 1
      Fig. 1Frequency of interviewees in percentages who discussed opportunities to MR-Linac implementation in US hospitals, by main theme and subtheme.
      Fig 1
      Fig. 1Frequency of interviewees in percentages who discussed opportunities to MR-Linac implementation in US hospitals, by main theme and subtheme.

      Clinical opportunities

      Interviewees expected a large added-value from MR-Linac implementation in clinical practice as a result of technical advancements. Main clinical opportunities included: (1) potential reduced toxicity and improved tumor control due to more precise and targeted radiation therapy; (2) possibly improved understanding of individual treatment response; and (3) potentially improved patient comfort and quality of life. Furthermore, interviewees also discussed that superior tumor delineation with real-time imaging and hypofractionation could improve treatment compliance and hospital visits given the potential of less therapy fractions.

      Technological opportunities

      The MR-Linac was perceived by interviewees to be a sophisticated technology with potential technical advances which could allow for both diagnostic and therapeutic opportunities. For instance, one interviewee referred to the potential of adaptive treatment planning for adaptive contour propagation and rapid dosimetric reconstruction which would allow for smaller treatment uncertainty margins and avoidance of dose to healthy tissues. This explains the possibility of hypofractionation. There was general agreement among interviewees that these opportunities may evolve as software development continues (eg, application of deep learning, enabling potential technology developments).
      The opportunity of real-time adaptive MRI-imaging during the course of radiation therapy could allow for increased data collection including anatomic and functional imaging data of both the tumor and surrounding organs. This data is perceived to allow greater understanding to safely deliver the most effective dose to individual tumor biology rather than its stage and location. As a result, interviewees expected new treatment options in radiation oncology and in related medical disciplines.

      Professional opportunities

      Implementation of the MR-Linac allows different possibilities for professional development. First, using MR-Linac encourages the development of new competencies and welcomed novel responsibilities for staff members because of the technological knowledge required to run both an MRI machine and linear accelerator. For instance, radiation oncologists need to gain understanding of the potential information obtained from different MRI sequences and response assessment. Furthermore, interdisciplinary relationships could be improved because of the increased cooperation between radiology and radiation oncology experts. These opportunities are expected to allow retention of staff and additional recruitment. As an example, one interviewee stated: “I think rolling out new technology is always important for your recruitment and retention of the highest-level physicians in the hospital.”
      Given the ongoing technology development, interviewees felt they could also benefit from research opportunities with funding potentials, which positively affect their own careers. To elaborate, implementing the MR-Linac is perceived by technology adopters to access prestigious technology which could allow delivery of higher-quality care. Being perceived as a “pioneer” in this area by providing this new technology could also be greatly beneficial for clinical expert and hospital reputation.

      Economic opportunities

      Most interviewees expected long-term efficiency and operational benefits from implementing the MR-Linac. Automation of elements in the clinical workflow, such as planning and contouring, were perceived to increase efficiency. Furthermore, the possible absence of pretreatment planning within an MR-only online workflow could decrease the duration of the total care pathway. Another example, the increased potential for hypofractionation could allow operational efficiencies for both the provider and the patient, such as decreased treatment sessions as well as interruptions and hospital admissions.
      Operational efficiencies were also expected to evolve over time as technological development continued. Interviewees reported communication efficiencies, for example by the automation of tasks together with more consistent multidisciplinary communication and reporting. Furthermore, interviewees emphasized that the clinical benefit of decreased radiation related side effects and less treatment fractions could result in improved overall efficiency of hospital systems (eg, lower burden on anesthesia provision).
      To elaborate, a possible reduction in the workforce per treatment of technology use is expected. On the one hand, the ongoing software development is expected to enable an automated process with faster quantification of the dose, which may lead to faster treatment delivery with reduced staff time and requirement (eg, less presence of the radiation oncologist and physicist during actual treatment delivery). On the other hand, professional experience and learning curves can reduce the education time needed to use the technology effectively.
      Implementation of the MR-Linac was also thought to potentially improve economic outcomes of radiation oncology departments by providing state-of-the-art treatment, particularly in the US health care system where health services are privatized and hospitals often compete against one another for patient referrals. Hence, interviewees mentioned that implementing the MR-Linac technology could possibly attract more patients with cancer to the providing center. One interviewee stated: “Especially well insured patients have this belief that that as Americans, we should have the latest and greatest technologies, almost regardless of cost.”

      Barriers

      We identified 5 barriers for implementation of the MR-Linac technology in US hospitals. These barriers were technology, professional, organizational, market, and regulatory related. Figure 2 shows the percentages of the interview cohort who discussed the barriers, by main theme and subtheme.
      Fig 2
      Fig. 2Frequency of interviewees in percentages who discussed barriers to MR-Linac implementation in US hospitals, by main theme and subtheme.
      Fig 2
      Fig. 2Frequency of interviewees in percentages who discussed barriers to MR-Linac implementation in US hospitals, by main theme and subtheme.

      Technological barriers

      Current radiation therapy centers often lack MR-imaging facilities. Technology users therefore face substantial investments for implementing the MR-Linac in the construction of technical facilities, maintenance, information technology, safety assurance, human resource policy, and personnel training. To elaborate, the combined functionality of both the MRI device and the radiation delivery device raises technical complexity and necessitates the acquirement of new skills and additional understanding of MRI sequences.
      Furthermore, the current lack of comparative effectiveness data and the ongoing technical development complicates a clear identification of potential benefits and return of investment. Although the MR-Linac was expected to increase the efficiency and precision of existing radiation therapy treatment, interviewees were at the same time critical of the actual clinical effect given the current lack of comparative effectiveness data. The previously mentioned advantages, such as online high-resolution imaging during treatment and functional imaging have yet to be proven in clinical studies.
      The interviewees also indicated a potential lack of understanding by technology users of the costs associated with MR-Linac treatment, especially as software development continues and the true treatment outcomes will evolve. One interviewee stated: “What would the cost be? What numbers are needed to actually make it a beneficial investment from a capital standpoint while the technological development still continues? I think that's the main discussion.”

      Professional barriers

      Technology adopters may face hurdles from the burden of technology implementation in the radiation oncology department to staff training of staff as well as treatment delivery to patients. As mentioned previously, several interviewees indicated a high investment burden and strategic decisions at a hospital level in early stages of technology implementation. To elaborate, technology users would need to obtain formal training to safely use the technology and to acquire MRI knowledge and skills given the routine use of MRI. Software developments inherent to the MR-Linac's development require technology users to anticipate continuous learning. Hence, the implementation of technical advancements would need to be adopted into professional expertise before the actual use of the technology could flourish. One interviewee asked: “Do we have appropriate algorithms that are rooted in the science, that are also rooted in the clinical expertise of our medical staff?”
      Another challenge is the formal approval for the introduction of new treatment workflows into clinical practice due to the lack of empirical evidence of effectiveness. Interviewees illustrated the importance of incorporating findings, as research data become available, into local clinical guidelines for appropriate technology usage in a timely manner. Therefore, interviewees also emphasized the importance of comparative effectiveness evaluation for technology implementation. Yet, there could be conflicts in prioritizing clinical treatment versus effectiveness research, given limited resources allocation at hospitals, due to the ability to have only one MR-Linac machine and the care burden because of the COVID-19 pandemic. Clinical evaluation was therefore not widely accepted as a main research objective among all interviewees.

      Organizational barriers

      As stated previously, the technology supply model requires substantial structural and staffing investments for factors such as workflow, quality assurance, and the development of protocols. Alongside these implementation investments, the ongoing technology development requires radiation oncology departments to maintain training and educational programs. Furthermore, interviewees expected an increased necessary interaction between radiation oncology departments as well as supporting medical specialties such as radiology who would aid in target delineation.
      Interviewees also emphasized that the COVID-19 pandemic was also an important factor leading to changes in resource allocation at the hospital; hence, for more commitment of human resources to direct care provision rather than secondary tasks, such as expertise training and clinical evaluation of new technology. One interviewee noted: “Discussions of decreased reimbursements for radiation delivery are important. A lot of resources are being reallocated to fight the COVID-19 pandemic. So, it is actually a pretty uncertain time economically.”
      Interviewees perceived the fact that value assessment and clinical trials (eg, demonstrating empirical evidence of [cost] effectiveness) not being integrated into departmental strategy and culture including vision, goals and key performance indicators, as a significant barrier to identifying the potential value offered by technology implementation.

      Market barriers

      Interviewees felt that they may not be appropriately motivated or financially compensated when it came to proving the effectivenesst of the MR-Linac technology. For example, some felt that the external authorities focused more on safety aspects than on demonstrating clinical effectiveness or added-value for the patient. Furthermore, interviewees discussed that decreased financial reimbursement for new radiation therapy delivery techniques may not allow possible financial gains from implementing MRgRT (eg, a relative reduction in number of treatment fractions and hospital visits as a result of hypofractionation compared with traditional external beam radiation therapy).
      The interviewees also addressed the implementation challenge of the presence of patient groups with disadvantaged socioeconomic background for new, costly therapy. As an example, the main hospital in which this study was conducted was identified as being socioeconomically diverse. This location included both wealthy, insured patients which will be beneficial for costly treatment implementation such as the MR-Linac, compared with relatively poor or uninsured patients, which may face hinder to afford costly therapy. Furthermore, while well-insured patients may hold the belief that Americans should have the “latest and greatest” medical therapy options available to them regardless of cost, other more historically marginalized communities (eg, African Americans) may be more reluctant to try innovative treatment such as the MRgRT.
      One interviewee stated: “There are a lot of undertreated, underdiagnosed cancer patients here in this inner city. So, if you're using your resource dollars, you'll get the most benefit from a place like [city]. There is a negative to that. And part of that is patient's reluctance to come in and try new technology given historical health disparities with race and other socioeconomic status. It's both a huge opportunity, but also a challenge, especially here in [city].”

      Regulatory barriers

      Interviewees identified the lack of appropriate reimbursement arrangements as a barrier to implementing the MR-Linac. As stated previously, interviewees felt more motivated by federal regulators to demonstrate safety than clinical effectiveness for reimbursement of care. Interviewees also indicated a potential lack of consensus among clinical care providers and nonclinical reimbursement entities in regards to health care cost reimbursement. The definition of insured care costs would be more focused on how much the government or insurers were willing to pay to providers, rather than the costs incurred by providers to deliver treatment. The existence of such gaps between clinical care providers and reimbursement entities was experienced by interviewees to discourage the demonstration of value of new technology.
      Furthermore, as health care and reimbursement are structured differently across states and individual hospitals in the US, some interviewees recognized that this fragmentation could hinder collaboration between MR-Linac providers when implementing the technology as well as having reproducible results across institutions.

      Discussion

      Our findings will help US hospitals to identify key points in their strategy when implementing the MR-Linac. MR-guidance in radiation therapy has been perceived a technical advancement with potential for therapy improvement and better patient outcomes, as well as scope for research and professional development.
      • Hall WA
      • Paulson ES
      • van der Heide UA
      • et al.
      The transformation of radiation oncology using real-time magnetic resonance guidance: A review.
      ,
      • Tree AC
      • Huddart R
      • Choudhury A.
      Magnetic resonance-guided radiotherapy—Can we justify more expensive technology?.
      ,
      • Ibbott GS.
      The need for, and implementation of, image guidance in radiation therapy.
      Yet, the technological complexity, the substantial operational and staffing investments, and the lack of empirical evidence of clinical effectiveness raise implementation uncertainties. Our findings also show that the lack of appropriate reimbursement and regulatory structures may complicate the actual deployment of the potential technology value.
      • Hall WA
      • Paulson ES
      • van der Heide UA
      • et al.
      The transformation of radiation oncology using real-time magnetic resonance guidance: A review.
      ,
      • Aarons GA
      • Cafri G
      • Lugo L
      • Sawitzky A.
      Expanding the domains of attitudes towards evidence-based practice: The evidence based practice attitude scale-50.
      The convergence of diagnostic and therapeutic realms in the MR-Linac technology, confirms the essential task of solving the common gap between technical innovations in the field of radiation oncology and current national associated treatment guidelines as well as present reimbursement structures.
      • Lievens Y
      • Defourny N
      • Corral J
      • et al.
      How public health services pay for radiotherapy in Europe: An ESTRO-HERO analysis of reimbursement.
      • Atun R
      • Jaffray DA
      • Barton MB
      • et al.
      Expanding global access to radiotherapy.
      • Jacobs M
      • Kerkmeijer L
      • de Ruysscher D
      • Brunenberg E
      • Boersma L
      • Verheij M.
      Implementation of MR-linac and proton therapy in two radiotherapy departments in The Netherlands: Recommendations based on lessons learned.
      The opportunities and barriers identified are fairly similar to those in literature such as the Dutch study on the implementation of the MR-Linac.
      • Hehakaya C
      • Van der Voort van Zyp JR
      • Lagendijk JJW
      • et al.
      Problems and promises of introducing the magnetic resonance imaging linear accelerator into routine care: The case of prostate cancer.
      Both in the US and Dutch cases, findings were consistent about the technology aspects, knowledge, and competence needed, and what it expects from radiation oncology departments to use new interventional procedures on the MR-Linac. Hence, the use of new interventional procedures on the MR-Linac with continued software developments require ongoing staffing learning and workflow adaptations, approval from national regulatory authorities before clinical usage and necessary interactions with policy-makers and payers.
      There is also consistency in the professional prospects, such as the development of staffing roles and required ongoing learning, and a higher hospital quality profile with potential economic growth. Interviewees in both cases were, however, also critical of the actual clinical added-value of the MR-Linac given the current lack of comparative effectiveness data. Defining the clinical value and operational profits of the MR-Linac at this time would be more of a challenge, given the ongoing technology development and indefinite treatment outcomes at present.
      • Pathmanathan AU
      • van As NJ
      • Kerkmeijer LGW
      • et al.
      Magnetic resonance imaging-guided adaptive radiation therapy: A “game changer” for prostate treatment?.
      ,
      • van Herk M
      • McWilliam A
      • Dubec M
      • Faivre-Finn C
      • Choudhury A.
      Magnetic resonance imaging-guided radiation therapy: A short strengths, weaknesses, opportunities, and threats analysis.
      • Tocco BR
      • Kishan AU
      • Ma TM
      • Kerkmeijer LGW
      • Tree AC.
      MR-guided radiotherapy for prostate cancer.
      • Kontaxis C
      • Bol GH
      • Kerkmeijer LGW
      • Lagendijk JJW
      • Raaymakers BW.
      Fast online replanning for interfraction rotation correction in prostate radiotherapy.
      • Kiser KJ
      • Smith BD
      • Wang J
      • Fuller CD.
      Après mois, le déluge”: Preparing for the coming data flood in the MRI-guided radiotherapy era.
      The relative cost-effectiveness of the MR-Linac will vary across geographic settings due to specific treatment standards and patient populations in a particular health care system.
      The inference of the different geographic contexts in the opportunities and barriers to the implementation of the MR-Linac shows similarities, but also differences. To illustrate, in the US case, interviewees mentioned potential challenges regarding certain patient groups with a reluctant attitude toward new therapy. In this geographic setting, interviewees also explicitly stated that they perceive few external stimuli to evaluate clinical effectiveness. The Dutch case, in contrast to the US case, reflected more explicitly on the intra- and interorganizational network required in technology implementation, such as the cooperation with the referring physician because of competitive specialties and payers to ensure treatment access and reimbursement. Thus, the implementation of the MR-Linac technology may encounter opportunities and challenges that are more relevant to one specific health care system.
      The identified opportunities and barriers to implementing the MR-Linac could relate to each other. Market influences on technology implementation could be caused by regulation while also raising technological barriers.
      • Greenhalgh T
      • Wherton J
      • Papoutsi C
      • et al.
      Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
      ,
      • Ahn JH.
      The impact of the banking competition in funding and lending markets on lending technology.
      For instance, governmental decision-making informs reimbursement requirements and may therefore stimulate technology users to evaluate clinical effectiveness as well as to deliver new insured treatment regimes.
      • Tang M
      • Joensuu H
      • Simes RJ
      • et al.
      Challenges of international oncology trial collaboration—A call to action.
      Our findings also showed that technology users may not feel motivated to evaluate the clinical effectiveness by external bodies as well as face hinder in technology implementation due to the lack of appropriate reimbursement structures. Ultimately, securing proper reimbursement structures and regulatory approval for new interventional procedures, are essential for technology implementation.
      Although market barriers could result from regulatory barriers, cultural barriers may determine these regulatory barriers.

      Hancher L, Moran M. Introduction: Regulation and deregulation. 1989:129-136.

      Our findings confirm that the lack of effectiveness evaluation in the radiation oncology department's strategy hinders to prove the technology's value as well as the steering and development of appropriate treatment guidelines and staffing protocols. Regulation and policy such as financial incentives or cost sharing have an important albeit indirect effect on clinical decision making and staffing allocation.
      • Clancy CM
      • Cronin K.
      Evidence-based decision making: Global evidence, local decisions.
      Thus, the implementation of the MR-Linac includes factors at clinical and technical level, but also at cultural, organizational, market, as well as governance level which can influence each other.
      Our study provides the first early multifaceted assessment of opportunities and challenges to MR-Linac implementation in US hospitals, which inform local technology adopters to improve and facilitate treatment access.
      • Atun R
      • Jaffray DA
      • Barton MB
      • et al.
      Expanding global access to radiotherapy.
      ,
      • Grutters JPC
      • Govers T
      • Nijboer J
      • Tummers M
      • Van Der Wilt GJ
      • Rovers MM.
      Problems and promises of health technologies: The role of early health economic modeling.
      ,
      • Love-Koh J.
      How useful are early economic models? Comment on “problems and promises of health technologies: The role of early health economic modelling.
      Technology users may face influences during implementation that are outside their ordinary scope of work. Because these influences are interrelated, the engagement with other health care decision-makers such as insurers, regulators, and policymakers could be relevant. Future efforts should generate empirical evidence of MR-Linac's effectiveness to prove expectations and justify return on investment concerns. The Multi-OutcoMe EvaluatioN of radiation Therapy Using the MR-linac Study aims to generate empirical evidence of clinical effectiveness and safety, and to identify subgroups of patients who are most likely to benefit from the MR-Linac.
      • de Mol van Otterloo SR
      • Christodouleas JP
      • Blezer ELA
      • et al.
      The MOMENTUM study: An international registry for the evidence-based introduction of MR-guided adaptive therapy.
      Approaches to assess the potential value of new medical technology, particularly those in early stages of development and implementation are gaining more attention.
      • Lievens Y
      • Audisio R
      • Banks I
      • et al.
      Towards an evidence-informed value scale for surgical and radiation oncology: A multi-stakeholder perspective.
      • Garibaldi C
      • Jornet N
      • Tee Tan L
      • et al.
      National societies’ needs as assessed by the ESTRO National Society Committee survey: A European perspective.
      • Ijzerman MJ
      • Koffijberg H
      • Fenwick E
      • Krahn M.
      Emerging use of early health technology assessment in medical product development: A scoping review of the literature.
      • Abrishami P
      • Boer A
      • Horstman K.
      How can we assess the value of complex medical innovations in practice?.
      Early health economic analyses are useful for identifying areas in which new developing technologies could be cost-effective and conditions that must be met to achieve cost-effectiveness results.
      • Grutters JPC
      • Govers T
      • Nijboer J
      • Tummers M
      • Van Der Wilt GJ
      • Rovers MM.
      Problems and promises of health technologies: The role of early health economic modeling.
      ,
      • Ijzerman MJ
      • Steuten LMG.
      Early assessment of medical technologies to inform product development and market access: A review of methods and applications.
      This analysis is particularly relevant for developing medical technologies such as the MR-Linac when both the costs and the effects of the innovation are still largely unknown, thereby guiding research and development and identifying potential meaningful treatment strategies.
      The presented findings may be relevant for other convergent medical technology as the MR-Linac represents other technical trends in medicine, such as the application of artificial intelligence applications.
      • Kiser KJ
      • Smith BD
      • Wang J
      • Fuller CD.
      Après mois, le déluge”: Preparing for the coming data flood in the MRI-guided radiotherapy era.
      The converging characteristic and rapidly increasing role of digitization in medical technologies affects the user practices as well as the current organizational and regulatory settings, creating an additional dynamic context.
      • Deans KJ
      • Sabihi S
      • Forrest CB.
      Learning health systems.
      • Shah P
      • Kendall F
      • Khozin S
      • et al.
      Artificial intelligence and machine learning in clinical development: A translational perspective.
      • Shaw J
      • Rudzicz F
      • Jamieson T
      • Goldfarb A.
      Artificial intelligence and the implementation challenge.
      Our findings confirm this potential impact of MR-Linac implementation. In-depth knowledge about the functionality and configuration of convergent medical technology, and the redistribution of knowledge, responsibilities and care pathways as well as related regulation and market dynamics are important to further consider in future research for this type of technologies. Moreover, findings can be better interpreted across different health care systems when considering specific national dynamics, such as health care trends, socioeconomic, reimbursement, and regulatory structures.
      Strengths of the present study include a thorough qualitative study design, which allows for maximal stakeholder insight into questions asked. The study was conducted with insights from several US health care institutions and different types of stakeholders. Limitations of this study include difficulty contacting private and public insurance regulators, whose insights could have been useful regarding regulatory and policy factors affecting technology implementation. Furthermore, the COVID-19 pandemic during the study timeline limits the number of stakeholders interviewed and may influence the experiences and perspectives on MR-Linac implementation. Future research could include more implementation perspectives at another time point. Broader stakeholder perspectives across national borders can help to identify health care system-specific opportunities and barriers.

      Conclusions

      Our findings will help US radiation therapy oncology departments to identify main elements in their strategy when implementing the MR-Linac. Our study confirmed the current literature on MR-Linac implementation with technical and clinical prospects, but also reveals additional insights into professional, sociocultural, market, and regulatory challenges. We therefore address implementation factors that are overlooked in the current MR-Linac literature and which in particular are crucial in the US health care system. This leads to new connections to facilitate appropriate introduction of the MR-Linac in US hospitals and is a start to understanding its long-term effects in health care systems.

      Appendix. Supplementary materials

      References

        • Greenhalgh T
        • Wherton J
        • Papoutsi C
        • et al.
        Beyond adoption: A new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies.
        J Med Internet Res. 2017; 19: e367
        • Smith GL
        • Ganz PA
        • Bekelman JE
        • et al.
        Promoting the appropriate use of advanced radiation technologies in oncology: Summary of a National Cancer Policy Forum Workshop.
        Int J Radiat Oncol Biol Phys. 2017; 97: 450-461
        • Rapport F
        • Clay-Williams R
        • Churruca K
        • Shih P
        • Hogden A
        • Braithwaite J.
        The struggle of translating science into action: Foundational concepts of implementation science.
        J Eval Clin Pract. 2018; 24: 117-126
        • Makady A
        • Ham R ten, de Boer A
        • Hillege H
        • Klungel O
        • Goettsch W.
        Policies for use of real-world data in health technology assessment (HTA): A comparative study of six HTA agencies.
        Value Heal. 2017; 20: 520-532
        • Clancy CM
        • Cronin K.
        Evidence-based decision making: Global evidence, local decisions.
        Health Aff. 2005; 24: 151-162
        • Li SA
        • Jeffs L
        • Barwick M
        • Stevens B.
        Organizational contextual features that influence the implementation of evidence-based practices across healthcare settings: A systematic integrative review.
        Syst Rev. 2018; 7: 72
        • Vandeusen Lukas C
        • Engle RL
        • Holmes SK
        • et al.
        Strengthening organizations to implement evidence-based clinical practices.
        Health Care Manage Rev. 2010; 35: 235-245
        • Jacobs M
        • Boersma L
        • Dekker A
        • et al.
        What is the impact of innovation on output in healthcare with a special focus on treatment innovations in radiotherapy? A literature review.
        Br J Radiol. 2017; 9020170251
        • Bertram RM
        • Suter JC
        • Bruns EJ
        • O'Rourke KE
        Implementation research and wraparound literature: Building a research agenda.
        J Child Fam Stud. 2011; 20: 713-725
        • Calabrò GE
        • La Torre G
        • De Waure C
        • Villari P
        • Federici A
        • Ricciardi W
        • Specchia ML
        Disinvestment in healthcare: An overview of HTA agencies and organizations activities at European level.
        BMC Health Serv Res. 2018; 18: 1-7
        • Adang EMM
        • Wensing M.
        Economic barriers to implementation of innovations in health care: Is the long run-short run efficiency discrepancy a paradox?.
        Health Policy (New York). 2008; 88: 236-242
        • Thune T
        • Mina A.
        Hospitals as innovators in the health-care system: A literature review and research agenda.
        Res Policy. 2016; 45: 1545-1557
        • Roco MC.
        Nanotechnology: Convergence with modern biology and medicine.
        Curr Opin Biotechnol. 2003; 14: 337-346
        • Lounsbury M
        • Crumley ET.
        New practice creation: An institutional perspective on innovation: Michael Lounsbury and Ellen T. Crumley.
        Organ Stud. 2007; 28: 993-1012
        • Moors EHM
        • Kukk Fischer P
        • Boon WPC
        • Schellen F
        • Negro SO
        Institutionalisation of markets: The case of personalised cancer medicine in the Netherlands.
        Technol Forecast Soc Change. 2018; 128: 133-143
        • Markiewicz K
        • Van Til JA
        • Ijzerman MJ.
        Medical devices early assessment methods: Systematic literature review.
        Int J Technol Assess Health Care. 2014; 30: 137-146
        • Hall WA
        • Paulson ES
        • van der Heide UA
        • et al.
        The transformation of radiation oncology using real-time magnetic resonance guidance: A review.
        Eur J Cancer. 2019; 122: 42-52
        • Raaymakers BW
        • Jürgenliemk-Schulz IM
        • Bol GH
        • et al.
        First patients treated with a 1.5 T MRI-Linac: Clinical proof of concept of a high-precision, high-field MRI guided radiotherapy treatment.
        Phys Med Biol. 2017; 62: L41-L50
        • Hehakaya C
        • Van der Voort van Zyp JR
        • Lagendijk JJW
        • et al.
        Problems and promises of introducing the magnetic resonance imaging linear accelerator into routine care: The case of prostate cancer.
        Front Oncol. 2020; 10: 1741
        • Kothari G
        • Loblaw A
        • Tree AC
        • van As NJ
        • Moghanaki D
        • Lo SS
        • Ost P
        • Siva S.
        Stereotactic body radiotherapy for primary prostate cancer.
        Technol Cancer Res Treat. 2018; 17: 1-13
        • Pathmanathan AU
        • van As NJ
        • Kerkmeijer LGW
        • et al.
        Magnetic resonance imaging-guided adaptive radiation therapy: A “game changer” for prostate treatment?.
        Int J Radiat Oncol Biol Phys. 2018; 100: 361-373
      1. Alongi F, Rigo M, Figlia V, Cuccia F, et al. 1.5T MR-guided and daily adapted SBRT for prostate cancer: Feasibility, preliminary clinical tolerability, quality of life and patient-reported outcomes during treatment. 2020;94:20200848.

        • Cuccia F
        • Corradini S
        • Mazzola R
        • et al.
        MR-guided hypofractionated radiotherapy: Current emerging data and promising perspectives for localized prostate cancer.
        Cancers (Basel). 2021; 13: 1-13
        • Cuccia F
        • Alongi F
        • Belka C
        • et al.
        Patient positioning and immobilization procedures for hybrid MR-Linac systems.
        Radiat Oncol. 2021; 16: 1-14
        • Nicosia L
        • Mazzola R
        • Rigo M
        • et al.
        Moderate versus extreme hypofractionated radiotherapy: A toxicity comparative analysis in low- and favorable intermediate-risk prostate cancer patients.
        J Cancer Res Clin Oncol. 2019; 145: 2547-2554
        • Caccia-Bava MDC
        • Guimaraes T
        • Harrington SJ.
        Hospital organization culture, capacity to innovate and success in technology adoption.
        J Heal Organ Manag. 2006; 20: 194-217
        • Lettieri E
        • Masella C.
        Priority setting for technology adoption at a hospital level: Relevant issues from the literature.
        Health Policy (New York). 2009; 90: 81-88
        • Pope C
        • Halford S
        • Turnbull J
        • Prichard J
        • Calestani M
        • May C.
        Using computer decision support systems in NHS emergency and urgent care: Ethnographic study using normalisation process theory.
        BMC Health Serv Res. 2013; 13: 111
        • Bryman A.
        Social Research Methods.
        4th ed. Oxford University Press, Oxford, United Kingdom2012
        • Tree AC
        • Huddart R
        • Choudhury A.
        Magnetic resonance-guided radiotherapy—Can we justify more expensive technology?.
        Clin Oncol. 2018; 30: 677-679
        • Ibbott GS.
        The need for, and implementation of, image guidance in radiation therapy.
        Ann ICRP. 2018; 47: 160-176
        • Aarons GA
        • Cafri G
        • Lugo L
        • Sawitzky A.
        Expanding the domains of attitudes towards evidence-based practice: The evidence based practice attitude scale-50.
        Adm Policy Ment Heal Ment Heal Serv Res. 2012; 39: 331-340
        • Lievens Y
        • Defourny N
        • Corral J
        • et al.
        How public health services pay for radiotherapy in Europe: An ESTRO-HERO analysis of reimbursement.
        Lancet Oncol. 2020; 21: e42-e54
        • Atun R
        • Jaffray DA
        • Barton MB
        • et al.
        Expanding global access to radiotherapy.
        Lancet Oncol. 2015; 16: 1153-1186
        • Jacobs M
        • Kerkmeijer L
        • de Ruysscher D
        • Brunenberg E
        • Boersma L
        • Verheij M.
        Implementation of MR-linac and proton therapy in two radiotherapy departments in The Netherlands: Recommendations based on lessons learned.
        Radiother Oncol. 2022; 167: 14-24
        • van Herk M
        • McWilliam A
        • Dubec M
        • Faivre-Finn C
        • Choudhury A.
        Magnetic resonance imaging-guided radiation therapy: A short strengths, weaknesses, opportunities, and threats analysis.
        Int J Radiat Oncol. 2018; 101: 1057-1060
        • Tocco BR
        • Kishan AU
        • Ma TM
        • Kerkmeijer LGW
        • Tree AC.
        MR-guided radiotherapy for prostate cancer.
        Front Oncol. 2020; 10: 1-11
        • Kontaxis C
        • Bol GH
        • Kerkmeijer LGW
        • Lagendijk JJW
        • Raaymakers BW.
        Fast online replanning for interfraction rotation correction in prostate radiotherapy.
        Med Phys. 2017; 44: 5034-5042
        • Kiser KJ
        • Smith BD
        • Wang J
        • Fuller CD.
        Après mois, le déluge”: Preparing for the coming data flood in the MRI-guided radiotherapy era.
        Front Oncol. 2019; 9: 1-10
        • Ahn JH.
        The impact of the banking competition in funding and lending markets on lending technology.
        Rev Econ. 2016; 67: 1117-1139
        • Tang M
        • Joensuu H
        • Simes RJ
        • et al.
        Challenges of international oncology trial collaboration—A call to action.
        Br J Cancer. 2019; 121: 515-521
      2. Hancher L, Moran M. Introduction: Regulation and deregulation. 1989:129-136.

        • Grutters JPC
        • Govers T
        • Nijboer J
        • Tummers M
        • Van Der Wilt GJ
        • Rovers MM.
        Problems and promises of health technologies: The role of early health economic modeling.
        Int J Heal Policy Manag. 2019; 8: 575-582
        • Love-Koh J.
        How useful are early economic models? Comment on “problems and promises of health technologies: The role of early health economic modelling.
        Int J Heal Policy Manag. 2020; 9: 215-217
        • de Mol van Otterloo SR
        • Christodouleas JP
        • Blezer ELA
        • et al.
        The MOMENTUM study: An international registry for the evidence-based introduction of MR-guided adaptive therapy.
        Front Oncol. 2020; 10: 1328
        • Lievens Y
        • Audisio R
        • Banks I
        • et al.
        Towards an evidence-informed value scale for surgical and radiation oncology: A multi-stakeholder perspective.
        Lancet Oncol. 2019; 20: e112-e123
        • Garibaldi C
        • Jornet N
        • Tee Tan L
        • et al.
        National societies’ needs as assessed by the ESTRO National Society Committee survey: A European perspective.
        Radiother Oncol. 2020; 151: 176-181
        • Ijzerman MJ
        • Koffijberg H
        • Fenwick E
        • Krahn M.
        Emerging use of early health technology assessment in medical product development: A scoping review of the literature.
        Pharmacoeconomics. 2017; 35: 727-740
        • Abrishami P
        • Boer A
        • Horstman K.
        How can we assess the value of complex medical innovations in practice?.
        Expert Rev Pharmacoeconomics Outcomes Res. 2015; 15: 369-371
        • Ijzerman MJ
        • Steuten LMG.
        Early assessment of medical technologies to inform product development and market access: A review of methods and applications.
        Appl Health Econ Health Policy. 2011; 9: 331-347
        • Deans KJ
        • Sabihi S
        • Forrest CB.
        Learning health systems.
        Semin Pediatr Surg. 2018; 27: 375-378
        • Shah P
        • Kendall F
        • Khozin S
        • et al.
        Artificial intelligence and machine learning in clinical development: A translational perspective.
        NPJ Digit Med. 2019; 2: 69
        • Shaw J
        • Rudzicz F
        • Jamieson T
        • Goldfarb A.
        Artificial intelligence and the implementation challenge.
        J Med Internet Res. 2019; 21: e13659