Human-Centered Design and Explainable AI: Building Trust in Clinical AI Systems
DOI:
https://doi.org/10.58445/rars.3219Keywords:
Artificial Intelligence, Medical Ethics, Human Centered Design, Explainable AI, Digitall Divide, User InterfaceAbstract
With the rise of AI in healthcare, comes a challenge to trust due to the “black box” problem and algorithmic bias. This paper argues that a human-centered design approach is essential for mitigating these issues by creating transparent and fair systems by exploring how explainable AI can make logic understandable to healthcare professionals while design interventions can ensure equitable outcomes. It also provides an analysis of patient’s perspectives that reveals the key concerns about the loss of empathy, the non-negotiable need for physician oversight, and the imperative for data privacy.
References
Bohr A, Memarzadeh K. The rise of artificial intelligence in healthcare applications. Artificial Intelligence in Healthcare. 2020:25–60. doi: 10.1016/B978-0-12-818438-7.00002-2. Epub 2020 Jun 26. PMCID: PMC7325854.
Bajwa J, Munir U, Nori A, Williams B. Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J. 2021 Jul;8(2):e188-e194. doi: 10.7861/fhj.2021-0095. PMID: 34286183; PMCID: PMC8285156.
Chen Y, Clayton EW, Novak LL, Anders S, Malin B. Human-Centered Design to Address Biases in Artificial Intelligence. J Med Internet Res. 2023 Mar 24;25:e43251. doi: 10.2196/43251. PMID: 36961506; PMCID: PMC10132017.(jmir-2023-1)
Cecilia Panigutti, Andrea Beretta, Daniele Fadda, Fosca Giannotti, Dino Pedreschi, Alan Perotti, and Salvatore Rinzivillo. 2023. Co-design of Human-centered, Explainable AI for Clinical Decision Support. ACM Trans. Interact. Intell. Syst. 13, 4, Article 21 (December 2023), 35 pages. https://doi.org/10.1145/3587271 (3587271)
van Leersum, Catharina & Maathuis, Clara. (2025). Human Centred Explainable AI Decision-Making in Healthcare. Journal of Responsible Technology. 21. 100108. 10.1016/j.jrt.2025.100108. (van leersum mathius)
Hanhui Xu, Kyle Michael James Shuttleworth, Medical artificial intelligence and the black box problem: a view based on the ethical principle of “do no harm”,
Intelligent Medicine,(s2.0-s266) Volume 4, Issue 1,
,
Pages 52-57,
ISSN 2667-1026, https://doi.org/10.1016/j.imed.2023.08.001.
Director, S. “Does Black Box AI In Medicine Compromise Informed Consent?”. Philos. Technol. 38, 62 (2025). https://doi.org/10.1007/s13347-025-00860-1 (s13347-025)
Tjeerd A.J. Schoonderwoerd, Wiard Jorritsma, Mark A. Neerincx, Karel van den Bosch, Human-centered XAI: Developing design patterns for explanations of clinical decision support systems, International Journal of Human-Computer Studies,
Volume 154,
,
,
ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2021.102684. (s1071)
Norori N, Hu Q, Aellen FM, Faraci FD, Tzovara A. Addressing bias in big data and AI for health care: A call for open science. Patterns (N Y). 2021 Oct 8;2(10):100347. doi: 10.1016/j.patter.2021.100347. PMID: 34693373; PMCID: PMC8515002. (main 1)
Verganti, R., Vendraminelli, L., & Iansiti, M. (2020). Innovation and Design in the Age of Artificial Intelligence. Journal of Product Innovation Management, 37(3), 212-227. (20-091)
Iris Glassberg, Yael Brender Ilan, Moti Zwilling, The key role of design and transparency in enhancing trust in AI-powered digital agents,
Journal of Innovation & Knowledge,
Volume 10, Issue 5,
,
,
ISSN 2444-569X, https://doi.org/10.1016/j.jik.2025.100770. (11)
Saranya A., Subhashini R.,
A systematic review of Explainable Artificial Intelligence models and applications: Recent developments and future trends,
Decision Analytics Journal,
Volume 7,
,
,
ISSN 2772-6622, https://doi.org/10.1016/j.dajour.2023.100230. (12)
AUTHOR=Wang Liuping , Zhang Zhan , Wang Dakuo , Cao Weidan , Zhou Xiaomu , Zhang Ping , Liu Jianxing , Fan Xiangmin , Tian Feng TITLE=Human-centered design and evaluation of AI-empowered clinical decision support systems: a systematic review
JOURNAL=Frontiers in Computer Science VOLUME=Volume 5 - 2023
YEAR=2023 URL=https://www.frontiersin.org/journals/computer-sci ence/articles/10.3389/fcomp.2023.1187299 DOI=10.3389/fcomp.2023.1187299 (fcomp)
Chen Y, Clayton E, Novak L, Anders S, Malin B Human-Centered Design to Address Biases in Artificial Intelligence
J Med Internet Res 2023;25:e43251
URL: https://www.jmir.org/2023/1/e43251 DOI: 10.2196/43251
Tove Helldin, Christian Norrie,
Designing for human-centered AI—Lessons learned from a case study in the clinical domain, International Journal of Human-Computer Studies, Volume 205,
,
,
ISSN 1071-5819, https://doi.org/10.1016/j.ijhcs.2025.103623.
Fritsch SJ, Blankenheim A, Wahl A, Hetfeld P, Maassen O, Deffge S, Kunze J, Rossaint R, Riedel M,
Marx G, Bickenbach J. Attitudes and perception of artificial intelligence in healthcare: A cross-sectional survey among patients. Digit Health. 2022 Aug 8;8:20552076221116772. doi: 10.1177/20552076221116772. PMID: 35983102; PMCID: PMC9380417.(10.1177)
Esmaeilzadeh P, Mirzaei T, Dharanikota S. Patients' Perceptions Toward Human-Artificial Intelligence Interaction in Health Care: Experimental Study. J Med Internet Res. 2021 Nov 25;23(11):e25856. doi: 10.2196/25856. PMID: 34842535; PMCID: PMC8663518. (jmir-2021)
Witkowski, K., Dougherty, R.B. & Neely, S.R. Public perceptions of artificial intelligence in healthcare: ethical concerns and opportunities for patient-centered care. BMC Med Ethics 25, 74 (2024). https://doi.org/10.1186/s12910-024-01066-4 (s12910
Downloads
Posted
Categories
License
Copyright (c) 2025 Kaustubh Shaw
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.