HEART Attacks: Healthcare Evaluation of Adversarial RobusTness

Le résumé fourni par la source

Large Vision-Language Models (LVLMs) have shown potential for integrating visual and textual information, yet they continue to exhibit limitations in visual reasoning and robustness, a particular concern in high-stakes domains such as healthcare. Prior robustness evaluations remain limited in breadth, typically focusing on a narrow set of medical modalities and tasks. Moreover, they study failure modes within a single modality (e.g., text negation or image swaps) overlooking one of the most sensitive aspects of LVLMs: how models leverage and combine information across modalities. To address this gap, we introduce BAIT (Blind-trust Adversarial Injection Toolkit), a general framework that converts vision datasets into controlled multimodal adversarial evaluations by injecting assistive or adversarial cues as prompt text and in-image overlays. Using BAIT, we build HEART (Healthcare Evaluation of Adversarial RobusTness), an adversarial benchmark created from several expert-annotated medical sources, covering multiple imaging modalities (e.g., radiology, ophthalmology, dermatology, etc.). We evaluate state-of-the-art LVLMs on HEART and observe that, despite moderate performance under non-adversarial conditions, even the most robust models exhibit catastrophic failures under adversarial conditions, with accuracy dropping below random chance due to LVLMs prioritizing injected misinformation over visual evidence. As a mitigation strategy, this work shows that out-of-distribution adversarial supervised fine-tuning and explicit system prompt warnings provide limited resilience to BAIT-style attacks. Together, BAIT and HEART define a scalable and flexible scheme for assessing the robustness of LVLMs in the context of multimodal information conflict, enabling the development of more reliable LVLMs for use in healthcare as well as in other multimodal domains.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.