FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare (2024)

1. Introduction

Despite major advances in the field of medical AI, the deployment and adoption of AI technologies remain limited in real-world clinical practice. In recent years, concerns have been raised about the technical, clinical, ethical and social risks associated with medical AI(Lekadir etal., 2022; Vollmer etal., 2020). In particular, existing research has shown that medical AI tools can be prone to errors and patient harm, biases and increased health inequalities, lack of transparency and accountability, as well as data privacy and security breaches(Challen etal., 2019; Celi etal., 2022; He etal., 2019; Haibe-Kains etal., 2020; Murdoch, 2021).

To increase adoption in the real world, it is essential that medical AI tools are trusted and accepted by patients, clinicians, health organisations and authorities. However, there is an absence of clear, widely accepted guidelines on how medical AI tools should be designed, developed, evaluated and deployed to be trustworthy, i.e. technically robust, clinically safe, ethically sound and legally compliant. To have a real impact at scale, such guidelines for trustworthy and responsible AI must be obtained through wide consensus involving international and inter-disciplinary experts.

In other domains, international consensus guidelines have made lasting impacts. For example, the FAIR guideline(Wilkinson etal., 2016) for data management has been widely adopted by researchers, organisations and authorities, as they provided a logical framework for standardising and enhancing the tasks of data collection, curation, organisation and storage. While it can be argued that the FAIR principles do not cover every aspect of data management, as they focus more on findability, accessibility, interoperability and reusability of the data, and less on privacy and security, they delivered a code of practice that is now widely accepted and applied.

For medical AI, initial efforts have focused on providing recommendations for the reporting of AI studies for different medical domains or clinical tasks (e.g. TRIPOD-AI(Collins etal., 2021a), CLAIM(Mongan etal., 2020a), CONSORT-AI(Liu etal., 2020a), DECIDE-AI(Vasey etal., 2022), PROBAST-AI(Collins etal., 2021a), CLEAR(Kocak etal., 2023)). These guidelines do not provide best practices for the actual development and deployment of the AI tools but promote standardised and complete reporting of their development and evaluation. Recently, several researchers have published promising ideas on possible best practices for medical AI(Larson etal., 2021a; Reddy etal., 2021; Park and Han, 2018; Walsh etal., 2021; Maier-Hein etal., 2022a; Bradshaw etal., 2022). However, these proposals have not been established through wide international consensus and do not cover the whole lifecycle of medical AI (i.e. from design, development and evaluation to deployment, usage and monitoring). In 2020, a comprehensive self-assessment checklist for trustworthy AI was defined through consensus by Europe’s High-Level Expert Group on Artificial Intelligence, but it covered AI in general and did not address the specific risks and challenges of AI in medicine and healthcare(Ala-Pietilä etal., 2020).

This paper addresses an important gap in the field of medical AI, by delivering the very first international, consensus guideline for trustworthy medical AI that covers the entire AI lifecycle (Figure1).

The FUTURE-AI consortium was initiated in 2021 and currently comprises 118 international and inter-disciplinary experts from 51 countries (Figure1), representing all continents (Europe, North America, South America, Asia, Africa, and Oceania). Additionally, the members represent a variety of disciplines (e.g. data science, medical research, healthcare, computer engineering, medical ethics, social sciences) and data domains (e.g. radiology, genomics, mobile health, electronic health records, surgery, pathology). To develop the FUTURE-AI framework, we drew inspiration from the FAIR principles for data management and defined concise recommendations structured according to six guiding principles, i.e. Fairness, Universality, Traceability, Usability, Robustness, and Explainability (Figure2).

2. Materials and Methods

The FUTURE-AI framework was defined over a 24-month period through a modified Delphi approach (Table1).

FUTURE-AI was initiated with an in-depth literature review on medical AI and trustworthiness, which resulted in the identification of key dimensions of relevance to trustworthy AI, including robustness, safety, security, fairness, transparency, traceability, accountability, generalisability, explainability, usability and responsible AI. To facilitate the use of the framework, some keywords were grouped, selected and re-ordered to obtain a reduced set of six guiding principles (Fairness, Universality, Traceability, Usability, Robustness, Explainability), which form the basis of the FUTURE-AI acronym.

Working groups composed of three experts each (including clinicians, data scientists and computer engineers) explored the six principles separately and proposed an initial set of best practices, by using AI for medical imaging as an initial use case. Furthermore, the working groups discussed the proposed recommendations, and removed overlaps and redundancies across the six principles, resulting in a first comprehensive set of 54 recommendations.

Subsequently, the FUTURE-AI consortium members provided systematic feedback on the first version of the FUTURE-AI guideline through a comprehensive survey. The experts could comment on each recommendation, rate its importance, propose missing recommendations, and provide additional feedback in free text. Based on the results of the survey, the list of recommendations was deemed too extensive, and was thus substantially reduced from 54 to 22 recommendations, while the scope was carefully broadened from AI for medical imaging to AI for healthcare.

The reduced set of recommendations was sent out to the FUTURE-AI consortium, together with a list of major disagreements that arose, for a second round of feedback and comments. This step resulted in a new version consisting of 30 recommendations, with a new “General” category in addition to the six guiding principles to account for recommendations that are transversal across all the dimensions of trustworthy AI.

Based on the machine learning technology readiness level (ML-TRL)(Lavin etal., 2022), the FUTURE-AI guideline was refined by distinguishing between medical AI tools at the research or proof-of-concept stage (i.e. ML-TRL 1 to 4) and those intended for clinical deployment (i.e. ML-TRL 5 to 9), as they require different levels of compliance. Hence, we asked the members of the consortium to rate the recommendations as recommended vs. highly recommended, for both proof-of-concept (low ML-TRL) and deployable AI tools (high ML-TRL). Finally, iterative discussions on the guideline, disagreements and manuscript were held, including during four dedicated online meetings, resulting in a final set of 28 consensus recommendations, which are listed in Table2.

3. FUTURE-AI guideline

In this section, we provide definitions and justifications for each of the six guiding principles and give an overview of the FUTURE-AI recommendations. Table2 provides a summary of the recommendations, together with the proposed level of compliance (i.e. recommended vs. highly recommended). More detailed descriptions are provided in Table 3 in the Appendix for readers who may require more information on any recommendation(s). Note that a glossary of the main terms used in this paper is provided in Table 4 in the Appendix, while the main stakeholders of relevance to the FUTURE-AI framework are listed in Table 5 in the Appendix.

4. Discussion

Despite the tremendous amount of research in medical AI in recent years, currently, only a limited number of AI tools have made the transition to clinical practice. While many studies have demonstrated the huge potential of AI to improve healthcare, significant clinical, technical, socio-ethical and legal challenges persist.

In this paper, we presented the results of an international effort to establish a consensus guideline for developing trustworthy and deployable AI tools in healthcare. Through an iterative process that lasted 24 months, the FUTURE-AI framework was established, comprising a well-structured, self-contained set of 28 recommendations, which covers the whole lifecycle of medical AI. By dividing the recommendations across six guiding principles, the pathways towards trustworthy AI are clearly characterised to facilitate their use throughout the AI tool’s lifecycle.

By the end of the process, all the recommendations were approved with less than 5% disagreement among all FUTURE-AI members. The FUTURE-AI consortium provided knowledge and expertise across a wide range of disciplines and stakeholders, resulting in consensus and wide support, both geographically and across domains. Hence, the FUTURE-AI guideline can benefit a wide range of stakeholders, as detailed in Table 5 in the Appendix.

FUTURE-AI is a risk-informed framework. It proposes to assess application-specific risks and challenges early in the process (e.g. risk of discrimination, lack of generalisability, data drifts, lack of acceptance by end-users, potential harm for patients, lack of transparency, data security vulnerabilities, ethical risks), then implement tailored measures to reduce these risks (e.g. collect data on individuals’ attributes to assess and mitigate bias). This is also a risk-benefit balancing exercise, as the specific measures to be implemented have benefits and potential weaknesses that the developers need to assess and balance. For example, collecting data on individuals’ attributes may increase the risk of re-identification, but can enable to reduce the risk of bias and discrimination. Hence, in FUTURE-AI, risk management (as recommended in Traceability 1) must be a continuous and transparent process throughout the AI tool’s lifecycle.

Furthermore, FUTURE-AI is an assumption-free, highly collaborative framework. It recommends to continuously engage with multi-disciplinary stakeholders to understand application-specific needs, risks and solutions (General 1). This is crucial to remove assumptions and investigate all possible risks and factors that may reduce trust in a given AI tool. For example, instead of making any assumption on possible sources of bias (e.g. sex or age), FUTURE-AI recommends that the developers engage with healthcare professionals, domain experts, representative citizens, and/or ethicists early in the process to investigate in depth the application-specific sources of bias, that may include factors well beyond standards attributes (e.g. breast density for AI applications in breast cancer).

For deployable AI tools, 24 recommendations out of 28 are rated as highly recommended (Table2). For research and proof-of-concept AI tools, only 12 recommendations are rated as highly recommended, but we advise that researchers use as many elements as possible from the FUTURE-AI guideline to facilitate future transitions towards real-world practice.

The FUTURE-AI guideline was defined in a generic manner to ensure it can be applied across a variety of domains (e.g. radiology, genomics, mobile health, electronic health records). However, for many recommendations, their applicability varies across medical use cases. Hence, the first recommendation in each of the FUTURE-AI framework’s principles is to identify the specificities to be addressed, such as the types of biases (Fairness 1), the clinical settings (Universality 1), or the need and approaches for explainable AI (Explainability 1).

The FUTURE-AI framework provides a set of general recommendations on how to enhance the trustworthiness of medical AI tools but does not impose any specific techniques for implementing each recommendation. While some examples of techniques are provided in the Appendix (Table 3), the final implementations should be defined by the developers, who should carefully select the most adequate methods given the application domain, clinical use case and data characteristics, as well as the advantages and limitations of each method.While we obtained a large consensus, some AI experts may disagree with some of the recommendations or may consider that some recommendations are either missing or not fully addressed. For example, while we propose mechanisms to enhance traceability and governance (e.g. AI logging), the issue of liability is yet to be addressed (e.g. who should be responsible for periodic auditing of the AI tools, who should be accountable when there is an error). Some of these key issues will require further investigations by multi-disciplinary researchers in the field of trustworthy AI, as well as by legal experts, regulators and authorities.

Aware of this limitation, we propose FUTURE-AI as a dynamic, living framework. Progressive development and adoption of medical AI tools will lead to new needs, challenges, and opportunities. To refine the FUTURE-AI guideline and learn from other voices, we set up a dedicated webpage (www.future-ai.eu) through which we invite the community to join the FUTURE-AI network and provide feedback based on their own experience and perspective. On the website, we include a FUTURE-AI self-assessment checklist, which comprises a set of questions and examples to facilitate and illustrate the use of the FUTURE-AI recommendations.

Additionally, we plan to organise regular outreach events such as webinars and workshops to exchange with medical AI researchers, manufacturers, evaluators, end-users and regulators.

References

• (1)
• a30 ([n. d.])[n. d.].European Commission, Directorate-General forCommunications Networks, Content and Technology, The Assessment List forTrustworthy Artificial Intelligence (ALTAI) for self assessment, PublicationsOffice, 2020, https://data.europa.eu/doi/10.2759/002360.
• a14 ([n. d.])[n. d.].ISO/IEC JTC 1/SC 42 - artificial intelligence. ISO.May 27, 2023. Accessed May 31, 2023.https://www.iso.org/committee/6794475.html.
• Ala-Pietilä etal. (2020)Pekka Ala-Pietilä,Yann Bonnet, Urs Bergmann,Maria Bielikova, Cecilia Bonefeld-Dahl,Wilhelm Bauer, Loubna Bouarfa,Raja Chatila, Mark Coeckelbergh,Virginia Dignum, etal.2020.The assessment list for trustworthyartificial intelligence (ALTAI).European Commission.
• Amann etal. (2020)Julia Amann, AlessandroBlasimme, Effy Vayena, Dietmar Frey,and VinceI Madai. 2020.Explainability for artificial intelligence inhealthcare: a multidisciplinary perspective.BMC medical informatics and decision making20, 1 (2020),1–9.
• Aquino etal. (2023)Yves SaintJames Aquino,WendyA Rogers, Annette Braunack-Mayer,Helen Frazer, KhinThan Win,Nehmat Houssami, Christopher Degeling,Christopher Semsarian, and StacyMCarter. 2023.Utopia versus dystopia: Professional perspectiveson the impact of healthcare artificial intelligence on clinical roles andskills.International Journal of MedicalInformatics 169 (2023),104903.
• Arnold etal. (2019)Matthew Arnold, RachelKEBellamy, Michael Hind, Stephanie Houde,Sameep Mehta, Aleksandra Mojsilović,Ravi Nair, KNatesan Ramamurthy,Alexandra Olteanu, David Piorkowski,etal. 2019.FactSheets: Increasing trust in AI services throughsupplier’s declarations of conformity.IBM Journal of Research and Development63, 4/5 (2019),6–1.
• Arras etal. (2022)Leila Arras, Ahmed Osman,and Wojciech Samek. 2022.CLEVR-XAI: A benchmark dataset for the ground truthevaluation of neural network explanations.Information Fusion 81(2022), 14–40.
• Barocas etal. (2017)Solon Barocas, MoritzHardt, and Arvind Narayanan.2017.Fairness in machine learning.Nips tutorial 1(2017), 2017.
• Bellamy etal. (2018)RachelKE Bellamy, KuntalDey, Michael Hind, SamuelC Hoffman,Stephanie Houde, Kalapriya Kannan,Pranay Lohia, Jacquelyn Martino,Sameep Mehta, Aleksandra Mojsilovic,etal. 2018.AI Fairness 360: An extensible toolkit fordetecting, understanding, and mitigating unwanted algorithmic bias.arXiv preprint arXiv:1810.01943(2018).
• Bodenreider etal. (2018)Oliver Bodenreider, RonaldCornet, and DanielJ Vreeman.2018.Recent developments in clinicalterminologies—SNOMED CT, LOINC, and RxNorm.Yearbook of medical informatics27, 01 (2018),129–139.
• Bradshaw etal. (2022)TylerJ Bradshaw, RonaldBoellaard, Joyita Dutta, AbhinavK Jha,Paul Jacobs, Quanzheng Li,Chi Liu, Arkadiusz Sitek,Babak Saboury, PeterJH Scott,etal. 2022.Nuclear medicine and artificial intelligence: bestpractices for algorithm development.Journal of Nuclear Medicine63, 4 (2022),500–510.
• Cabitza etal. (2021)Federico Cabitza, AndreaCampagner, Felipe Soares,LuisGarcía de Guadiana-Romualdo,Feyissa Challa, Adela Sulejmani,Michela Seghezzi, and Anna Carobene.2021.The importance of being external. methodologicalinsights for the external validation of machine learning models in medicine.Computer Methods and Programs inBiomedicine 208 (2021),106288.
• Celi etal. (2022)LeoAnthony Celi,Jacqueline Cellini, Marie-LaureCharpignon, EdwardChristopher Dee,Franck Dernoncourt, Rene Eber,WilliamGreig Mitchell, Lama Moukheiber,Julian Schirmer, Julia Situ,etal. 2022.Sources of bias in artificial intelligence thatperpetuate healthcare disparities—A global review.PLOS Digital Health 1,3 (2022), e0000022.
• Challen etal. (2019)Robert Challen, JoshuaDenny, Martin Pitt, Luke Gompels,Tom Edwards, and KrasimiraTsaneva-Atanasova. 2019.Artificial intelligence, bias and clinical safety.BMJ Quality & Safety 28,3 (2019), 231–237.
• Channa etal. (2021)Roomasa Channa, RisaWolf, and MichaelD Abramoff.2021.Autonomous artificial intelligence in diabeticretinopathy: from algorithm to clinical application.Journal of diabetes science and technology15, 3 (2021),695–698.
• Collins etal. (2021a)GaryS Collins, PaulaDhiman, Constanza LAndaur Navarro, JieMa, Lotty Hooft, JohannesB Reitsma,Patricia Logullo, AndrewL Beam,Lily Peng, Ben VanCalster,etal. 2021a.Protocol for development of a reporting guideline(TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognosticprediction model studies based on artificial intelligence.BMJ open 11,7 (2021), e048008.
• Collins etal. (2021b)GaryS Collins, PaulaDhiman, Constanza LAndaur Navarro, JieMa, Lotty Hooft, JohannesB Reitsma,Patricia Logullo, AndrewL Beam,Lily Peng, Ben VanCalster,etal. 2021b.Protocol for development of a reporting guideline(TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognosticprediction model studies based on artificial intelligence.BMJ open 11,7 (2021), e048008.
• Commission ([n. d.])European Commission.[n. d.].AI for Health Imaging — Programme — H2020 — CORDIS.https://cordis.europa.eu/programme/id/H2020_DT-TDS-05-2020
• DeGrave etal. (2021)AlexJ DeGrave, JosephDJanizek, and Su-In Lee.2021.AI for radiographic COVID-19 detection selectsshortcuts over signal.Nature Machine Intelligence3, 7 (2021),610–619.
• Dormann (2020)CarstenF Dormann.2020.Calibration of probability predictions frommachine-learning and statistical models.Global ecology and biogeography29, 4 (2020),760–765.
• Feng etal. (2022)Jean Feng, RachaelVPhillips, Ivana Malenica, AndrewBishara, AlanE Hubbard, LeoA Celi,and Romain Pirracchio. 2022.Clinical artificial intelligence qualityimprovement: towards continual monitoring and updating of AI algorithms inhealthcare.npj Digital Medicine 5,1 (2022), 66.
• Ferryman and Pitcan (2018)Kadija Ferryman andMikaela Pitcan. 2018.Fairness in precision medicine.(2018).
• Finlayson etal. (2019a)SamuelG Finlayson, JohnDBowers, Joichi Ito, JonathanL Zittrain,AndrewL Beam, and IsaacS Kohane.2019a.Adversarial attacks on medical machine learning.Science 363,6433 (2019), 1287–1289.
• Finlayson etal. (2019b)SamuelG Finlayson, JohnDBowers, Joichi Ito, JonathanL Zittrain,AndrewL Beam, and IsaacS Kohane.2019b.Adversarial attacks on medical machine learning.Science 363,6433 (2019), 1287–1289.
• Ganapathi etal. (2022)Shaswath Ganapathi, JoPalmer, JosephE Alderman, MelanieCalvert, Cyrus Espinoza, Jacqui Gath,Marzyeh Ghassemi, Katherine Heller,Francis Mckay, Alan Karthikesalingam,etal. 2022.Tackling bias in AI health datasets through theSTANDING Together initiative.Nature Medicine 28,11 (2022), 2232–2233.
• Gao etal. (2019)Yuan Gao, Yuanyuan Wang,and Jinhua Yu. 2019.Optimized Resolution-Oriented Many-to-One IntensityStandardization Method for Magnetic Resonance Images.Applied Sciences 9,24 (2019), 5531.
• Garrucho etal. (2022)Lidia Garrucho, KaisarKushibar, Socayna Jouide, Oliver Diaz,Laura Igual, and Karim Lekadir.2022.Domain generalization in deep learning based massdetection in mammography: A large-scale multi-center study.Artificial Intelligence in Medicine132 (2022), 102386.
• Gebru etal. (2021)Timnit Gebru, JamieMorgenstern, Briana Vecchione,JenniferWortman Vaughan, Hanna Wallach,HalDaumé Iii, and Kate Crawford.2021.Datasheets for datasets.Commun. ACM 64,12 (2021), 86–92.
• Ghassemi etal. (2021a)Marzyeh Ghassemi, LukeOakden-Rayner, and AndrewL Beam.2021a.The false hope of current approaches to explainableartificial intelligence in health care.The Lancet Digital Health3, 11 (2021),e745–e750.
• Ghassemi etal. (2021b)Marzyeh Ghassemi, LukeOakden-Rayner, and AndrewL Beam.2021b.The false hope of current approaches to explainableartificial intelligence in health care.The Lancet Digital Health3, 11 (2021),e745–e750.
• Ghorbani etal. (2019)Amirata Ghorbani, AbubakarAbid, and James Zou. 2019.Interpretation of neural networks is fragile. InProceedings of the AAAI conference on artificialintelligence, Vol.33. 3681–3688.
• Haibe-Kains etal. (2020)Benjamin Haibe-Kains,GeorgeAlexandru Adam, Ahmed Hosny,Farnoosh Khodakarami, Massive AnalysisQuality Control (MAQC) SocietyBoard of Directors Shraddha Thakkar 35 KuskoRebecca 36 Sansone Susanna-Assunta 37 Tong Weida 35 Wolfinger Russ D. 38Mason Christopher E. 39 Jones Wendell 40 Dopazo Joaquin 41 FurlanelloCesare42, Levi Waldron, Bo Wang,Chris McIntosh, Anna Goldenberg,Anshul Kundaje, etal.2020.Transparency and reproducibility in artificialintelligence.Nature 586,7829 (2020), E14–E16.
• He etal. (2019)Jianxing He, SallyLBaxter, Jie Xu, Jiming Xu,Xingtao Zhou, and Kang Zhang.2019.The practical implementation of artificialintelligence technologies in medicine.Nature medicine 25,1 (2019), 30–36.
• Hedström etal. (2023a)Anna Hedström, LeanderWeber, Daniel Krakowczyk, DilyaraBareeva, Franz Motzkus, Wojciech Samek,Sebastian Lapuschkin, and Marina M-CHöhne. 2023a.Quantus: An explainable ai toolkit for responsibleevaluation of neural network explanations and beyond.Journal of Machine Learning Research24, 34 (2023),1–11.
• Hedström etal. (2023b)Anna Hedström, LeanderWeber, Daniel Krakowczyk, DilyaraBareeva, Franz Motzkus, Wojciech Samek,Sebastian Lapuschkin, and Marina M-CHöhne. 2023b.Quantus: An explainable ai toolkit for responsibleevaluation of neural network explanations and beyond.Journal of Machine Learning Research24, 34 (2023),1–11.
• IEEE ([n. d.])IEEE. [n. d.].IEEE SA - IEEE Recommended Practice for the qualitymanagement of datasets for Medical Artificial Intelligence. IEEE StandardsAssociation. Accessed May 31, 2023.https://standards.ieee.org/ieee/2801/7459/.
• Kapoor and Narayanan (2022)Sayash Kapoor and ArvindNarayanan. 2022.Leakage and the reproducibility crisis in ML-basedscience.arXiv preprint arXiv:2207.07048(2022).
• Kaviani etal. (2022)Sara Kaviani, KiJin Han,and Insoo Sohn. 2022.Adversarial attacks and defenses on AI in medicalimaging informatics: A survey.Expert Systems with Applications198 (2022), 116815.
• Kocak etal. (2023)Burak Kocak, BettinaBaessler, Spyridon Bakas, RenatoCuocolo, Andrey Fedorov, LenaMaier-Hein, Nathaniel Mercaldo, HenningMüller, Fanny Orlhac, DanielPintodos Santos, etal. 2023.CheckList for EvaluAtion of Radiomics research(CLEAR): a step-by-step reporting guideline for authors and reviewersendorsed by ESR and EuSoMII.Insights into Imaging 14,1 (2023), 1–13.
• Kompa etal. (2021)Benjamin Kompa, JasperSnoek, and AndrewL Beam.2021.Second opinion needed: communicating uncertainty inmedical machine learning.NPJ Digital Medicine 4,1 (2021), 4.
• Königstorfer and Thalmann (2022)Florian Königstorfer andStefan Thalmann. 2022.AI Documentation: A path to accountability.Journal of Responsible Technology11 (2022), 100043.
• Larson etal. (2021a)DavidB Larson, HughHarvey, DanielL Rubin, Neville Irani,RTse Justin, and CurtisP Langlotz.2021a.Regulatory frameworks for development andevaluation of artificial intelligence–based diagnostic imaging algorithms:summary and recommendations.Journal of the American College ofRadiology 18, 3 (2021),413–424.
• Larson etal. (2021b)DavidB Larson, HughHarvey, DanielL Rubin, Neville Irani,RTse Justin, and CurtisP Langlotz.2021b.Regulatory frameworks for development andevaluation of artificial intelligence–based diagnostic imaging algorithms:summary and recommendations.Journal of the American College ofRadiology 18, 3 (2021),413–424.
• Lavin etal. (2022)Alexander Lavin,CiaránM Gilligan-Lee, AlessyaVisnjic, Siddha Ganju, Dava Newman,Sujoy Ganguly, Danny Lange,AtílímGüneş Baydin,Amit Sharma, Adam Gibson,etal. 2022.Technology readiness levels for machine learningsystems.Nature Communications 13,1 (2022), 6039.
• Lekadir etal. (2022)Karim Lekadir, GianlucaQuaglio, ATselioudis Garmendia, andCatherine Gallin. 2022.Artificial Intelligence in Healthcare-Applications,Risks, and Ethical and Societal Impacts.European Parliament (2022).
• Lemay etal. (2022)Andreanne Lemay, KatharinaHoebel, ChristopherP Bridge, BrianBefano, Silvia DeSanjosé, DidemEgemen, AnaCecilia Rodriguez, MarkSchiffman, JohnPeter Campbell, andJayashree Kalpathy-Cramer.2022.Improving the repeatability of deep learning modelswith Monte Carlo dropout.npj Digital Medicine 5,1 (2022), 174.
• Li etal. (2021)Xiaoxiao Li, Ziteng Cui,Yifan Wu, Lin Gu, andTatsuya Harada. 2021.Estimating and improving fairness with adversariallearning.arXiv preprint arXiv:2103.04243(2021).
• Lima etal. (2019)DanielM Lima, JoseFRodrigues-Jr, AgmaJM Traina, FabioAPires, and MarcoA Gutierrez.2019.Transforming two decades of ePR data to OMOP CDMfor clinical research.Stud Health Technol Inform264, August (2019),233–7.
• Liu etal. (2020a)Xiaoxuan Liu,SamanthaCruz Rivera, David Moher,MelanieJ Calvert, AlastairK Denniston,Hutan Ashrafian, AndrewL Beam,An-Wen Chan, GaryS Collins,Ara DarziJonathanJ Deeks, etal.2020a.Reporting guidelines for clinical trial reports forinterventions involving artificial intelligence: the CONSORT-AI extension.The Lancet Digital Health2, 10 (2020),e537–e548.
• Liu etal. (2020b)Xiaoxuan Liu,SamanthaCruz Rivera, David Moher,MelanieJ Calvert, AlastairK Denniston,Hutan Ashrafian, AndrewL Beam,An-Wen Chan, GaryS Collins,Ara DarziJonathanJ Deeks, etal.2020b.Reporting guidelines for clinical trial reports forinterventions involving artificial intelligence: the CONSORT-AI extension.The Lancet Digital Health2, 10 (2020),e537–e548.
• Maier-Hein etal. (2022a)Lena Maier-Hein, BjoernMenze, etal. 2022a.Metrics reloaded: Pitfalls and recommendations forimage analysis validation.arXiv. org 2206.01653(2022).
• Maier-Hein etal. (2022b)Lena Maier-Hein, BjoernMenze, etal. 2022b.Metrics reloaded: Pitfalls and recommendations forimage analysis validation.arXiv. org 2206.01653(2022).
• McLennan etal. (2020)Stuart McLennan, AmeliaFiske, LeoAnthony Celi, RuthMüller, Jan Harder, Konstantin Ritt,Sami Haddadin, and Alena Buyx.2020.An embedded ethics approach for AI development.Nature Machine Intelligence2, 9 (2020),488–490.
• McLennan etal. (2022)Stuart McLennan, AmeliaFiske, Daniel Tigard, Ruth Müller,Sami Haddadin, and Alena Buyx.2022.Embedded ethics: a proposal for integrating ethicsinto the development of medical AI.BMC Medical Ethics 23,1 (2022), 6.
• Mikołajczyk and Grochowski (2018)Agnieszka Mikołajczyk andMichał Grochowski. 2018.Data augmentation for improving deep learning inimage classification problem. In 2018international interdisciplinary PhD workshop (IIPhDW). IEEE,117–122.
• Mitchell etal. (2019)Margaret Mitchell, SimoneWu, Andrew Zaldivar, Parker Barnes,Lucy Vasserman, Ben Hutchinson,Elena Spitzer, InioluwaDeborah Raji,and Timnit Gebru. 2019.Model cards for model reporting. InProceedings of the conference on fairness,accountability, and transparency. 220–229.
• Mohseni etal. (2021)Sina Mohseni, NiloofarZarei, and EricD Ragan.2021.A multidisciplinary survey and framework for designand evaluation of explainable AI systems.ACM Transactions on Interactive IntelligentSystems (TiiS) 11, 3-4(2021), 1–45.
• Mongan etal. (2020a)John Mongan, Linda Moy,and CharlesE KahnJr. 2020a.Checklist for artificial intelligence in medicalimaging (CLAIM): a guide for authors and reviewers., e200029pages.
• Mongan etal. (2020b)John Mongan, Linda Moy,and CharlesE KahnJr. 2020b.Checklist for artificial intelligence in medicalimaging (CLAIM): a guide for authors and reviewers., e200029pages.
• Murdoch (2021)Blake Murdoch.2021.Privacy and artificial intelligence: challenges forprotecting health information in a new era.BMC Medical Ethics 22,1 (2021), 1–5.
• Ngiam and Khor (2019)KeeYuan Ngiam and WeiKhor. 2019.Big data and machine learning algorithms forhealth-care delivery.The Lancet Oncology 20,5 (2019), e262–e273.
• Oala etal. (2021a)Luis Oala, AndrewGMurchison, Pradeep Balachandran, ShrutiChoudhary, Jana Fehr, AlixandroWerneckLeite, PeterG Goldschmidt, ChristianJohner, EloraDM Schörverth, RoseNakasi, etal. 2021a.Machine learning for health: algorithm auditing &quality control.Journal of medical systems45 (2021), 1–8.
• Oala etal. (2021b)Luis Oala, AndrewGMurchison, Pradeep Balachandran, ShrutiChoudhary, Jana Fehr, AlixandroWerneckLeite, PeterG Goldschmidt, ChristianJohner, EloraDM Schörverth, RoseNakasi, etal. 2021b.Machine learning for health: algorithm auditing &quality control.Journal of medical systems45 (2021), 1–8.
• Park and Han (2018)SeongHo Park andKyunghwa Han. 2018.Methodologic guide for evaluating clinicalperformance and effect of artificial intelligence technology for medicaldiagnosis and prediction.Radiology 286,3 (2018), 800–809.
• Pleiss etal. (2017)Geoff Pleiss, ManishRaghavan, Felix Wu, Jon Kleinberg, andKilianQ Weinberger. 2017.On fairness and calibration.Advances in neural information processingsystems 30 (2017).
• Rafner etal. (2022)Janet Rafner, DominikDellermann, Arthur Hjorth, DoraVeraszto, Constance Kampf, Wendy MacKay,and Jacob Sherson. 2022.Deskilling, upskilling, and reskilling: a case forhybrid intelligence.Morals & Machines 1,2 (2022), 24–39.
• Reddy etal. (2021)Sandeep Reddy, WendyRogers, Ville-Petteri Makinen, EnricoCoiera, Pieta Brown, Markus Wenzel,Eva Weicken, Saba Ansari,Piyush Mathur, Aaron Casey,etal. 2021.Evaluation framework to guide implementation of AIsystems into healthcare settings.BMJ health & care informatics28, 1 (2021).
• Rouzrokh etal. (2022)Pouria Rouzrokh, BardiaKhosravi, Shahriar fa*ghani, ManaMoassefi, DianaV VeraGarcia, YashbirSingh, Kuan Zhang, GianMarco Conte,and BradleyJ Erickson. 2022.Mitigating bias in radiology machine learning: 1.Data handling.Radiology: Artificial Intelligence4, 5 (2022),e210290.
• Sahiner etal. (2023)Berkman Sahiner, WeijieChen, RaviK Samala, and NicholasPetrick. 2023.Data drift in medical machine learning:implications and potential remedies.The British Journal of Radiology(2023), 20220878.
• Selvan etal. (2022)Raghavendra Selvan, NikhilBhagwat, LasseF WolffAnthony, BenjaminKanding, and ErikB Dam.2022.Carbon footprint of selecting and training deeplearning models for medical image analysis. InInternational Conference on Medical Image Computingand Computer-Assisted Intervention. Springer, 506–516.
• Sperrin etal. (2022)Matthew Sperrin, RichardDRiley, GaryS Collins, and GlenPMartin. 2022.Targeted validation: validating clinical predictionmodels in their intended population and setting.Diagnostic and Prognostic Research6, 1 (2022),24.
• Sujan etal. (2019a)Mark Sujan, DominicFurniss, Kath Grundy, Howard Grundy,David Nelson, Matthew Elliott,Sean White, Ibrahim Habli, andNick Reynolds. 2019a.Human factors challenges for the safe use ofartificial intelligence in patient care.BMJ health & care informatics26, 1 (2019).
• Sujan etal. (2019b)Mark Sujan, DominicFurniss, Kath Grundy, Howard Grundy,David Nelson, Matthew Elliott,Sean White, Ibrahim Habli, andNick Reynolds. 2019b.Human factors challenges for the safe use ofartificial intelligence in patient care.BMJ health & care informatics26, 1 (2019).
• Tian and Zhang (2022)Yingjie Tian and YuqiZhang. 2022.A comprehensive survey on regularization strategiesin machine learning.Information Fusion 80(2022), 146–166.
• Tjoa and Guan (2020)Erico Tjoa and c*ntaiGuan. 2020.A survey on explainable artificial intelligence(xai): Toward medical xai.IEEE transactions on neural networks andlearning systems 32, 11(2020), 4793–4813.
• Varoquaux and Cheplygina (2022)Gaël Varoquaux andVeronika Cheplygina. 2022.Machine learning for medical imaging:methodological failures and recommendations for the future.NPJ digital medicine 5,1 (2022), 48.
• Vasey etal. (2022)Baptiste Vasey, MyuraNagendran, Bruce Campbell, DavidAClifton, GaryS Collins, Spiros Denaxas,AlastairK Denniston, Livia Faes,Bart Geerts, Mudathir Ibrahim,etal. 2022.Reporting guideline for the early-stage clinicalevaluation of decision support systems driven by artificial intelligence:DECIDE-AI.Nature medicine 28,5 (2022), 924–933.
• Vokinger etal. (2021)KerstinN Vokinger, StefanFeuerriegel, and AaronS Kesselheim.2021.Mitigating bias in machine learning for medicine.Communications medicine1, 1 (2021),25.
• Vollmer etal. (2020)Sebastian Vollmer, BilalAMateen, Gergo Bohner, FranzJKirály, Rayid Ghani, Pall Jonsson,Sarah Cumbers, Adrian Jonas,KatherineSL McAllister, Puja Myles,etal. 2020.Machine learning and artificial intelligenceresearch for patient benefit: 20 critical questions on transparency,replicability, ethics, and effectiveness.bmj 368(2020).
• Walsh etal. (2021)Ian Walsh, DmytroFishman, Dario Garcia-Gasulla, TiinaTitma, Gianluca Pollastri, JenniferHarrow, FotisE Psomopoulos, andSilvioCE Tosatto. 2021.DOME: recommendations for supervised machinelearning validation in biology.Nature methods 18,10 (2021), 1122–1127.
• Wilkinson etal. (2016)MarkD Wilkinson, MichelDumontier, IjJ Aalbersberg, G Appleton,M Axton, A Baak, NBlomberg, JW Boiten, LB da SilvaSantos,PE Bourne, etal.2016.The FAIR Guiding Principles for scientific datamanagement and stewardship. Sci Data 3: 160018.
• Zhang etal. (2022)Kuan Zhang, BardiaKhosravi, Sanaz Vahdati, Shahriarfa*ghani, Fred Nugen, SeyedMoeinRassoulinejad-Mousavi, Mana Moassefi,Jaidip ManikraoM Jagtap, Yashbir Singh,Pouria Rouzrokh, etal.2022.Mitigating bias in radiology machine learning: 2.Model development.Radiology: Artificial Intelligence4, 5 (2022),e220010.
• Zhou etal. (2021)Qian Zhou, Zhi-hang Chen,Yi-heng Cao, and Sui Peng.2021.Clinical impact and quality of randomizedcontrolled trials involving interventions evaluating artificial intelligenceprediction tools: a systematic review.NPJ digital medicine 4,1 (2021), 154.

APPENDIX

Appendix A TABLES

Appendix B FULL AUTHOR AFFILIATIONS

Karim Lekadir, Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Aasa Feragen, DTU Compute, Technical University of Denmark, Kgs Lyngby, Denmark
Abdul Joseph Fofanah, Department of Mathematics and Computer Science, Faculty of Science and Technology, Milton Margai Technical University, Freetown, Sierra Leone
Alejandro F Frangi, Centre for Computational Imaging & Simulation Technologies in Biomedicine, Schools ofComputing and Medicine, University of Leeds, Leeds, United Kingdom and Medical Imaging Research Center (MIRC), Cardiovascular Science and Electronic Engineering Departments, KU Leuven, Leuven, Belgium
Alena Buyx Institute of History and Ethics in Medicine, Technical University of Munich, Munich, Germany
Anais Emelie Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Andrea Lara Faculty of Engineering of Systems, Informatics and Sciences of Computing, Galileo University, Guatemala City, Guatemala
Antonio R Porras Department of Biostatistics and Informatics, Colorado School of Public Health, University of Colorado Anschutz Medical Campus, Aurora, CO, United States
An-Wen Chan Department of Medicine, Women’s College Research Institute, University of Toronto, Toronto, Canada
Arcadi Navarro Institució Catalana de Recerca i Estudis Avançats (ICREA) and Universitat Pompeu Fabra, Barcelona, Spain andBarcelona Beta Brain Research Center, Pasqual Maragall Foundation, Barcelona, Spain
Ben Glocker Department of Computing, Imperial College London, London, United Kingdom
Benard O Botwe School of Biomedical & Allied Health Sciences, University of Ghana, Accra, Ghana and Department of Midwifery & Radiography, School of Health & Psychological Sciences, City University of London, UK
Bishesh Khanal NepAl Applied Mathematics and Informatics Institute for research (NAAMII), Kathmandu, Nepal
Brigit Beger European Heart Network, Brussels, Belgium
Carol C Wu Department of Thoracic Imaging, University of Texas MD Anderson Cancer Center, Houston, United States
Celia Cintas IBM Research Africa, Nairobi, Kenya
Curtis P Langlotz Center for Artificial Intelligence in Medicine & Imaging, Stanford University School of Medicine, Stanford, United States
Daniel Rueckert Institute for AI and Informatics in Medicine, Klinikum rechts der Isar, Technical University Munich, Munich, Germany and Department of Computing, Imperial College London, London, UK
Deogratias Mzurikwao Muhimbili University of Health and Allied Sciences, Dar es Salaam, Tanzania
Dimitrios I Fotiadis Unit of Medical Technology and Intelligent Information Systems, Foundation for Research and Technology - Hellas (FORTH), Ioannina, Greece
Doszhan Zhussupov Almaty AI Lab, Almaty, Kazakhstan
Enzo Ferrante CONICET, Universidad Nacional del Litoral, Santa Fe, Argentina
Erik Meijering School of Computer Science and Engineering, University of New South Wales, Sydney, Australia
Eva Weicken Fraunhofer Heinrich Hertz Institute, Berlin, Germany
Fabio A González Computing Systems and Industrial Engineering Dept., Universidad Nacional de Colombia, Bogotá, Colombia
Folkert W Asselbergs Amsterdam University Medical Centers, Department of Cardiology, University of Amsterdam, Amsterdam, The Netherlands and Health Data Research UK and Institute of Health Informatics, University College London, London, United Kingdom
Fred Prior Department of Biomedical Informatics, University of Arkansas for Medical Sciences, Little Rock, United States
Gabriel P Krestin Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, the Netherlands
Gary Collins Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom
Geletaw S Tegenaw Faculty of Computing and Informatics, JiT, Jimma University, Jimma, Ethiopia
Georgios Kaissis Institute for AI and Informatics in Medicine, Klinikum rechts der Isar, Technical University Munich, Munich, Germany
Gianluca Misuraca Department of Artificial Intelligence, Universidad Politécnica de Madrid, Madrid, Spain
Gianna Tsakou Gruppo Maggioli, Research and Development Lab, Athens, Greece
Girish Dwivedi Department of Advanced Clinical and Translational Cardiovascular Imaging, The University of Western Australia, Perth, Australia
Haridimos Kondylakis Department of Electrical and Computer Engineering, Hellenic Mediterranean University and Foundation for Research and Technology - Hellas (FORTH), Crete, Greece
Harsha Jayakody Postgraduate Institute of Medicine, University of Colombo, Colombo, Sri Lanka
Henry C Woodruf The D-lab, Department of Precision Medicine, GROW - School for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands
Hugo JWL Aerts Artificial Intelligence in Medicine Program, Mass General Brigham, Harvard Medical School, Boston, United States
Ian Walsh Bioprocessing Technology Institute, Agency for Science, Technology and Research (A*STAR), Singapore, Singapore
Ioanna Chouvarda School of Medicine, Aristotle University of Thessaloniki, Thessaloniki, Greece
Irène Buvat Institut Curie, Inserm, Orsay, France
Islem Rekik BASIRA Lab, Imperial-X and Computing Department, Imperial College London, UK and ]Faculty of Computer and Informatics Engineering, Istanbul Technical University, Turkey
James Duncan Departments of Biomedical Engineering and Radiology & Biomedical Imaging, Schools of Engineering & Applied Science and Medicine, Yale University, New Haven, United States
Jayashree Kalpathy-Cramer Massachusetts General Hospital, Harvard Medical School, Massachusetts, United States
Jihad Zahir LISI Laboratory, Computer Science Department, Cadi Ayyad University, Marrakech, Morocco
Jinah Park School of Computing, Korea Advanced Institute of Science and Technology, Daejeon, South Korea
John Mongan Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, United States
Judy W Gichoya Department of Radiology & Imaging Sciences, Emory University, Atlanta, United States
Julia A Schnabel Institute of Machine Learning in Biomedical Imaging, Helmholtz Center Munich, Munich, Germany
Kaisar Kushibar Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Katrine Riklund Department of Radiation Sciences, Diagnostic Radiology, Umeå University, Umeå, Sweden
Kensaku Mori Graduate School of Informatics, Nagoya University, Nagoya, Japan
Kostas Marias Department of Electrical and Computer Engineering, Hellenic Mediterranean University and Foundation for Research and Technology - Hellas (FORTH), Crete, Greece
Lameck M Amugongo Department of Software Engineering, Namibia University of Science & Technology, Windhoek, Namibia
Lauren A Fromont Center for Genomic Regulation, The Barcelona Institute of Science and Technology, Barcelona, Spain
Lena Maier-Hein Div. Intelligent Medical Systems (IMSY), German Cancer Research Center (DKFZ), Heidelberg, Germany
Leonor Cerdá Alberich Biomedical Imaging Research Group, La Fe Health Research Institute, Valencia, Spain
Leticia Rittner School of Electrical and Computer Engineering, University of Campinas, Campinas, Brazil
Lighton Phiri Department of Library Information Science, University of Zambia, Lusaka, Zambia
Linda Marrakchi-Kacem National Engineering School of Tunis, University of Tunis El Manar, Tunis, Tunisia
Lluís Donoso-Bach Clinical Advanced Technologies institute (CATI), Hospital Clínic of Barcelona, Barcelona, Spain
Luis Martí-Bonmatí Medical Imaging Department, Hospital Universitario y Politécnico La Fe, Valencia, Spain
M Jorge Cardoso School of Biomedical Engineering & Imaging Sciences, King’s College London, London, United Kingdom
Maciej Bobowicz 2nd Division of Radiology, Medical University of Gdansk, Gdansk, Poland
Mahsa Shabani Faculty of Law and Criminology, Ghent University, Ghent, Belgium
Manolis Tsiknakis Department of Electrical and Computer Engineering, Hellenic Mediterranean University and Foundation for Research and Technology - Hellas (FORTH), Crete, Greece
Maria A Zuluaga Data Science Department, EURECOM, Sophia Antipolis, France
Maria Bielikova Kempelen Institute of Intelligent Technologies, Bratislava, Slovakia
Marie-Christine Fritzsche Institute of History and Ethics in Medicine, Technical University of Munich, Munich, Germany
Marius George Linguraru Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital Washington DC, United States
Markus Wenzel Fraunhofer Heinrich Hertz Institute, Berlin, Germany
Marleen De Bruijne Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, the Netherlands
Martin G Tolsgaard Copenhagen Academy for Medical Education and Simulation Rigshospitalet, University of, Copenhagen, Copenhagen, Denmark
Marzyeh Ghassemi Electrical Engineering and Computer Science (EECS) and Institute for Medical Engineering & Science (IMES), Massachusetts Institute of Technology, Cambridge, United States
Md Ashrafuzzaman Department of Biomedical Engineering, Military Institute of Science and Technology, Dhaka, Bangladesh
Melanie Goisauf BBMRI-ERIC, ELSI Services & Research, Graz, Austria
Mohammad Yaqub Mohamed Bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates
Mohammed Ammar Engineering Systems and Telecommunication Laboratory, University M’Hamed Bougara, Boumerdes, Algeria
Mónica Cano Abadía BBMRI-ERIC, ELSI Services & Research, Graz, Austria
Mukhtar M E Mahmoud Faculty of Computer Science and Information Technology, University of Kassala, Kassala, Sudan
Mustafa Elattar Center for Informatics Science, Nile University, Sheikh Zayed City, Egypt
Nicola Rieke Healthcare & Life Science EMEA, NVIDIA GmbH, Munich, Germany
Nikolaos Papanikolaou Computational Clinical Imaging Group, Champalimaud Foundation, Lisbon, Portugal
Noussair Lazrak Health, Environment and policy, New York University, New York, United States
Oliver Díaz Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Olivier Salvado Data61, The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra, Australia
Oriol Pujol Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Ousmane Sall Pôle Sciences, Technologies et Numérique, Université Virtuelle du Sénégal, Diamniadio, Senegal
Pamela Guevara Faculty of Engineering, Universidad de Concepción, Concepción, Chile
Peter Gordebeke European Institute for Biomedical Imaging Research, Vienna, Austria
Philippe Lambin The D-lab, Department of Precision Medicine, GROW - School for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands
Pieta Brown Orion Health, Auckland, New Zealand
Purang Abolmaesumi Department of Electrical and Computer Engineering, the University of British Columbia, Vancouver, BC, Canada
Qi Dou Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China
Qinghua Lu Data61, The Commonwealth Scientific and Industrial Research Organisation (CSIRO), Canberra, Australia
Richard Osuala Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Rose Nakasi Makerere Artificial Intelligence Lab, Makerere University, Kampala, Uganda
S Kevin Zhou School of Biomedical Engineering & Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, China
Sandy Napel Integrative Biomedical Imaging Informatics at Stanford (IBIIS), Department of Radiology, Stanford University, Stanford CA, United States
Sara Colantonio Institute of Information Science and Technologies of the National Research Council of Italy, Pisa, Italy
Shadi Albarqouni Department of Diagnostic and Interventional Radiology, University Hospital Bonn, Bonn, Germany
Smriti Joshi Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Stacy Carter Australian Centre for Health Engagement, Evidence and Values, School of Health and Society, University of Wollongong, New South Wales, Australia
Stefan Klein Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, the Netherlands
Steffen E Petersen William Harvey Research Institute, Queen Mary University of London, London, United Kingdom
Susanna Aussó Artificial Intelligence in Healthcare Program, TIC Salut Social Foundation, Barcelona, Spain
Suyash Awate, Computer Science and Engineering Department, Indian Institute of Technology Bombay, Mumbai, India
Tammy Riklin Raviv School of Electrical and Computer Engineering, Ben-Gurion University, Beer Sheba, Israel
Tessa Cook Department of Radiology, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA, United States
Tinashe E M Mutsvangwa Department of Human Biology, University of Cape Town, Cape Town, South Africa
Wendy A Rogers Department of Philosophy, and School of Medicine, Macquarie University, Sydney, Australia
Wiro J Niessen Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, the Netherlands
Xènia Puig-Bosch Artificial Intelligence in Medicine Lab (BCN-AIM), Department of Mathematics and Computer Science, Universitat de Barcelona, Barcelona, Spain
Yi Zeng Center for Long-term AI, Chinese Academy of Sciences, Beijing, China
Yunusa G Mohammed Department of Human Anatomy, Gombe State University, Gombe, Nigeria
Yves Saint James Aquino Australian Centre for Health Engagement, Evidence and Values, School of Health and Society, University of Wollongong, New South Wales, Australia
Zohaib Salahuddin The D-lab, Department of Precision Medicine, GROW - School for Oncology and Reproduction, Maastricht University, Maastricht, the Netherlands
Martijn P A Starmans Department of Radiology & Nuclear Medicine, Erasmus MC University Medical Center, Rotterdam, the Netherlands