BY RISHI SHAH
Artificial intelligence (AI) is heralded as a transformative force in medicine, with the potential to revolutionize diagnostics, personalize treatments, and optimize healthcare delivery [1]. Yet, AI is not an inherently neutral tool. Instead, it reflects and magnifies the values, biases, and inequities embedded in the systems that create it. In healthcare, this has profound implications: racial biases in AI risk exacerbating disparities, entrenching existing inequities, and undermining trust in medical institutions [2]. To interrogate the myth of algorithmic neutrality, we must explore how AI can perpetuate structural inequalities and propose actionable solutions for leveraging AI as a force for equity.
The Fallacy of Neutrality in AI
The notion that AI systems are neutral—objective intermediaries that merely process data without bias—is dangerously misleading [3]. Algorithms are shaped by human choices: the selection of training data, the design of models, and the priorities of those who build them. In medicine, this has led to AI systems that not only mirror but magnify the inequities present in healthcare.
The quest for neutrality often results in systems that aim to perform equally across populations, but this can obscure structural inequities. For instance, an algorithm might achieve high accuracy in diagnosing conditions across a broad population but fail to address the specific needs of marginalized groups. A more equitable approach is to embrace corrective bias, where AI actively prioritizes underrepresented and historically disadvantaged populations. Corrective bias acknowledges that fairness sometimes requires unequal treatment to achieve equitable outcomes [4]. For example, AI tools could be designed to over-index data from minority populations to compensate for historical underrepresentation.
The Power of Context: Rethinking Data in AI
AI’s reliance on large datasets often prioritizes quantity over quality and context. Most medical datasets strip away crucial contextual information—such as patients’ socioeconomic conditions, neighborhood environments, and lived experiences—that profoundly impact health outcomes [2]. This decontextualization reduces individuals to a set of clinical variables, ignoring the complex social determinants of health that drive disparities.
To create more equitable AI systems, we must shift towards “context-rich” data. This involves integrating narrative data—patients’ stories, community health histories, and social contexts—into AI training. Such data would allow AI to make more holistic assessments, capturing the complexities of health beyond biological factors. Imagine an AI system that not only predicts a patient’s risk for diabetes but also considers food insecurity, access to healthcare, and cultural dietary practices, offering tailored interventions that address these root causes [3].
AI Without Borders: Ethical Dilemmas in Global Health
The global health context adds another layer of complexity. Most AI systems are developed in high-income countries (HICs) using Western-centric datasets. When these systems are deployed in low- and middle-income countries (LMICs), they often fail to account for local healthcare realities, leading to misdiagnoses and inappropriate treatment recommendations [4]. This phenomenon, which can be termed global bias, perpetuates healthcare inequities on a global scale and risks reinforcing colonial legacies in medicine.
For instance, AI systems trained on Western populations may misinterpret symptoms in LMIC populations or overlook diseases more prevalent in these regions. Furthermore, the proprietary nature of many AI systems raises questions about data sovereignty and control. Who owns the data collected from LMICs? Who benefits from the insights generated? Without equitable frameworks, AI risks becoming another tool of exploitation rather than empowerment.
Decolonizing AI in medicine involves shifting power dynamics. This means developing systems collaboratively with LMIC communities, integrating local knowledge and practices, and ensuring that these communities benefit directly from the technology [1]. It also requires open-source AI models, where knowledge and innovation are shared freely, reducing dependence on costly proprietary systems and democratizing access to cutting-edge healthcare solutions.
Cultural Competence in Algorithmic Design
AI systems often lack cultural competence—the ability to understand and respect cultural differences in healthcare practices and patient communication. This shortcoming can lead to misinterpretation of symptoms and ineffective care. For example, an AI system trained on Western behavioral norms might misclassify culturally specific expressions of pain or mental distress, leading to misdiagnoses [5].
Embedding cultural competence in AI requires community involvement in data collection and model design. This could involve partnerships with local health organizations to ensure that AI systems reflect the diverse cultural contexts in which they are deployed. By incorporating culturally specific health data and practices, AI can provide more accurate and respectful care [4].
Redistribution of Health Knowledge: The Democratizing Potential of AI
AI has the potential to democratize healthcare knowledge by making advanced medical insights accessible to communities that historically lacked access to specialized care. For instance, AI-powered diagnostic tools could enable rural clinics to identify complex conditions without requiring a specialist. However, this potential is tempered by the risk of commodification. Many AI systems are proprietary, and controlled by corporations that profit from selling access to their tools. This risks deepening the divide between those who can afford premium healthcare insights and those who cannot [3].
To truly democratize healthcare, AI must shift towards an open-source model. Open-source AI systems would allow healthcare providers in resource-limited settings to access and adapt advanced tools without prohibitive costs. This would ensure that the benefits of AI are equitably distributed, helping to close the global health gap rather than widen it.
A Mirror and a Magnifier: AI’s Dual Role in Healthcare
At its core, AI serves as both a mirror and a magnifier of societal values. It reflects the biases and inequities embedded in our healthcare systems, but it also magnifies them, producing outcomes that demand urgent attention [2]. The diagnostic disparities and resource misallocations revealed by biased AI are not merely technological failures; they are indictments of the systemic inequities that pervade healthcare.
This dual role positions AI as a powerful diagnostic tool—not just for individual health but for the ethical and structural health of our systems. By critically examining these disparities, we are presented with an opportunity to redefine healthcare equity. AI’s revelations should catalyze systemic reform, prompting us to address the root causes of inequality with urgency and precision.
Artificial intelligence has the power to reshape medicine, but its benefits will remain unevenly distributed unless we confront its biases head-on. By rejecting the illusion of neutrality, embracing context-rich and culturally competent data, and decolonizing AI development, we can harness this technology to build a more equitable healthcare system. The future of medical AI should not only aim for accuracy and efficiency but actively work to correct historical and systemic imbalances. If we rise to this challenge, AI could become a transformative force for health equity, ensuring that all patients, regardless of race, geography, or socioeconomic status, receive the care they deserve.
Rishi Shah is a junior in Berkeley College.
——————————
References
- Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med. 2019 Jan;25[1]:44–56.
- Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019 Oct 25;366(6464):447–53.
- Char DS, Shah NH, Magnus D. Implementing Machine Learning in Health Care — Addressing Ethical Challenges. N Engl J Med. 2018 Mar 15;378(11):981–3.
- Gichoya JW, Banerjee I, Bhimireddy AR, Burns JL, Celi LA, Chen LC, et al. AI recognition of patient race in medical imaging: a modelling study. Lancet Digit Health. 2022 Jun 1;4(6):e406–14.
- Adamson AS, Smith A. Machine Learning and Health Care Disparities in Dermatology. JAMA Dermatol. 2018 Nov 1;154(11):1247–8.
Yale Global Health Review 