Artificial intelligence and machine learning techniques hold a great potential in solving complex world problems. They can facilitate clinical decision making by providing actionable insights through ‘learning’ from large volumes of patient data. Among others, deep learning algorithms were proved able to accurately identify head CT scan abnormalities requiring urgent attention, significantly increasing the efficiency of health services.

Encouraged by such exciting developments, AI is increasingly expected to be a promising means to realise high-performing medicine in the near future and is widely hoped to be the rescue for the overstretched health systems across the world in the aftermath of the COVID-19 pandemic. However, this promise comes with critical and alarming caveats for clinical decision making: there is a consensus that AI models, particularly those using data-driven technologies, are subject to, or themselves cause, bias and discrimination, exacerbating existing health inequalities. 

Health inequities 

Recent studies show that without proper mitigation on potential bias against underrepresented groups such as women and ethnic minorities, implementation of AI in healthcare can result in life or death consequences. A study by Straw and Wu showed that AI models built to identify people at high risk of liver disease from blood tests are twice as likely to miss the disease in women as in men.

This is because data-driven AI models make inferences by finding ‘patterns’ from the data they analyse, but disparities such as those on racial and ethnic basis have long existed in health and care. Without effective mitigation approaches, inferences learnt from such biased data are inevitably channeling embedded inequities into the decisions they make.

Apart from data-embedded structural health inequities, under-representation of minorities in health datasets creates a real technical challenge for machine learning to come up with sensible conclusions for such groups, creating another potential source of inequities exacerbated by AI. An insufficient number of samples of a minority group will cause computational models to draw inaccurate predictions for them. Unfortunately, such situations are pervasive for ethnic minority groups in healthcare datasets, which might not always reflect the actual population diversity: machine learning models trained on such data would draw inaccurate conclusions regarding the incidence or the risk of a disease within a specific population subset.

In addition to those embedded in data, biases could also arise from methodological choices for AI development and deployment. Obermeyer and colleagues analysed an algorithm widely employed in the US that uses health costs as a proxy for health needs and they demonstrated that it falsely concludes that Black patients are healthier than equally sick White patients. Technically speaking, bias might be unintentionally and easily induced in the feature selections (the choice of the input variables) and label determinations (the choice of target variables) of AI model developments.

Patient care

Besides potential bias and inequalities, already well-established as ethical concerns associated with the use of AI in health care, there are also numerous other challenges that may have significant impact on patient care. AI has the potential to impact not only diagnosis but also prevention, treatment, and disease management on systems-scale, thus raising broader questions about its role in public health, for example in anticipating epidemics and providing patient support. AI is data-driven and health care data are often difficult (or even impossible) to anonymise, raising worries about privacy and data protection for patients.

Questions of legal accountability and moral responsibility in AI-driven decision-making have been raised, drawing attention to the potential changes in doctor-patient relationships. A study by Hallowell et al. recently published in BMC Medical Ethics explores under what conditions trust could be placed in AI medical tools, highlighting that relational and epistemic trust is crucial, as clinician’s positive experience is directly correlated to future patient’s trust in AI tools. This study emphasises the need for deliberate and meticulous steps in designing trustworthy, confidence-worthy AI processes. 

Increasingly data-heavy medical practice might require novel skills from the next generation of medical professionals with consequences for the future of medical education. In order to address these issues, implementation of critical frameworks such as embedded ethics in the development of medical AI is required.

The need for tools to mitigate or overcome ethical issues

All in all, the use of AI in medicine, although it may bear high reward, is currently associated with high risk, as its consequences and implications are high-stakes including widening the social gap in health services and further fragmenting an already-divided society. Some tools are already available for general bias audit or particularly designed for healthcare application, and focused research communities are emerging, such as the Health Equity group at the Alan Turing Institute and the independent community of Data Science for Health Equity.

Metrics such as fairness, accountability, methods for bias mitigation, and explainability or interpretability are major aspects that impact the perception and use of AI in ethically-sensitive fields such as medicine, and methods endowed with these features might play a significant role in overcoming mistrust, although they could not guarantee trust.

However, we still do not have a clear picture about data-embedded and AI-induced bias in healthcare and their implications in our society: the biases of AI models are not quantified and reported with the same enthusiastic attention as accuracies, let alone apply effective mitigation approaches before deployments. Similarly, studies about the impact the use of AI has in healthcare, from multiple practical and theoretical perspectives, are still relatively limited compared to the impressive expansion of this field: more discussion is needed for a fair and critical consideration of these new technologies.

For these reasons, a cross-journal collection, Ethics of Artificial Intelligence in Health and Medicine, has been launched in collaboration with BMC Medical Ethics and BMC Medical Informatics and Decision Making. The collection welcomes studies focused on technical assessment and evaluation of AI-based medical decision making methods with regards to these ethics-relevant features, as well as more theoretical considerations on the medical use of AI-based methods. This includes but is not limited to the presentation of novel AI-based methods able to fulfil such ethical requirements and tools able to mitigate issues like bias, the elaboration of novel ethics-relevant metrics, research on attitudes and perceptions of physicians and the public on AI implementation, ethics surrounding AI-associated privacy and surveillance, and ethical challenges surrounding implementation of medical AI.