The Governance of AI in Healthcare: Exploring the Relationship between Vulnerability and Protection

The use of artificial intelligence (AI) technologies, including GenAI, is bringing about significant, if not, revolutionary changes to the organisation, administration and delivery of healthcare. While this is likely to be welcomed at a systems level, it poses a range of ethical and regulatory dilemmas in the context of interactions between patients and their treating healthcare professionals. Such interactions have been traditionally characterised by situated vulnerability on the part of patients against a background of informational and power asymmetries enjoyed by such professionals, as well as the wider healthcare systems in which they operate. Longstanding ethical and regulatory obligations have structured these interactions to ensure that such asymmetries are addressed, trust is enhanced and the risk of harm is minimised. However, the encroachment of AI technologies into this interactive clinical space is likely to change the dynamics of these interactions in potentially detrimental ways for patients through the introduction of algorithmic bias, informational gaps, and a lack of accountability in healthcare decision-making. Likewise for healthcare professionals, problems with a lack of explainability, combined with inadequate AI model training, may also hamper the provision of clinical advice and treatment in circumstances where they remain ultimately liable for harm caused to patients. In this context, an important aspect of academic and policy debates has been how best to embed an ethically principled approach in the design and use of AI technologies in healthcare to address such concerns. While recognising the merits of such an approach, it is suggested that it lacks a nuanced appreciation of the situated vulnerabilities that may arise in the context of using such technologies in interactions between patients and healthcare professionals. Drawing on vulnerability theory, this paper suggests that a better understanding of how to conceptualise the relationship between vulnerability and protection would assist in offering an enhanced assessment of the risks and benefits associated with the use of AI technologies in such interactions, as well as determining the level of regulatory protection that should be employed to mitigate the risks of harm associated with their use.