When to trust the answer: question-aligned semantic nearest neighbor entropy for safer surgical VQA

Résumé fourni par la source

PURPOSE: Future human-in-the-loop surgical visual question answering (VQA) requires detecting unreliable answers before use. In this preclinical methodological study, we distinguish answer consistency from question validity: A model may generate mutually consistent answers that still fail to address the clinical question. METHODS: We propose question-aligned semantic nearest neighbor entropy (QA-SNNE), a black-box failure-detection score that adds question-answer alignment to semantic entropy through bilateral gating. QA-SNNE modulates pairwise similarities among sampled answers by their question relevance, using embedding, entailment, or cross-encoder alignment. To assess robustness under language variation, we construct an out-of-template rephrased version of a benchmark surgical VQA dataset, in which only the question wording is modified while images and ground-truth answers remain unchanged. RESULTS: We evaluate QA-SNNE on five VQA models across two benchmark surgical VQA datasets, considering both zero-shot and parameter-efficient fine-tuned settings, including out-of-template questions. QA-SNNE improves AUROC on EndoVis18-VQA for two of three zero-shot models in-template (e.g., +15% for Llama3.2 and +21% for Qwen2.5) and achieves up to +8% AUROC relative improvement among zero-shot models under out-of-template rephrasing, with mixed results on external validation. CONCLUSION: By making uncertainty estimation explicitly question-aware, QA-SNNE provides a practical and model-agnostic safeguard for preclinical surgical VQA evaluation. Linking semantic uncertainty to question relevance enables more reliable failure detection under variation in question phrasing.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.