Beyond the Error: Frictions in Healthcare Professionals’ Repair of AI Systems

Résumé fourni par la source

Artificial intelligence (AI) systems in healthcare are prone to errors and inaccuracies, necessitating healthcare professionals (HCPs) to attend to AI systems and repair their errors based on clinical expertise. Yet we lack knowledge of the burdens imposed by repair work on HCPs, limiting decision-makers’ options for supporting HCPs during disruptive moments of AI repair as embedded in care practices. To bridge this gap, we build on ethnographic fieldwork in a Danish hospital ward using AI to monitor geriatric patients’ movements. We used participant observation and interviews to investigate how the AI’s errors created frictions that both generated visions of good repair work and impeded that very same repair. We identified three types of repair work. First, identifying errors generated knowledge of staff and patient whereabouts but was made difficult by cumbersome AI-embedding devices. Second, correcting errors animated HCPs to engage with error sources yet was challenged by patient flow and organization limiting when errors can be corrected. Third, communicating errors fostered visions of learning and AI improvement which were impeded by perceived lack of changes in AI performance. This shows repair of healthcare AI as beyond the individual error, intertwined with HCPs’ expertise, ward conditions, and institutionalized integration of AI. Meaningful repair work requires both staff-level coordination and systemic responsibility allocation for error management and follow-up.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.