2106 Structural-Equation-Based Correction of Attending Rater Bias in Video-Based Assessment of Resident Craniotomy Performance: A Pilot Study

Le résumé fourni par la source

INTRODUCTION: Video assessment provides scalable, objective benchmarking of operative skill, a critical need as ACGME milestones and competency-based promotion demand defensible metrics. In neurosurgery, the fine motor precision required and the consequences of error mean that residents receive autonomy in much smaller, high-stakes increments than in most other fields, making truly accurate skill measurement indispensable. However, uncontrolled evaluator bias can distort scores and undermine the decision to entrust a trainee with advancing autonomy. We propose using structural equation modeling (SEM) to create an unbiased measure of resident skill by isolating rater effects during video assessment of neurosurgery. METHODS: Five de-identified craniotomy videos (three residents) were independently scored on an eight-item, OSATS-derived 100-point scale by six faculty evaluators (two attendings, four chiefs), resulting in 18 rating events. A two-factor SEM was fitted in Stata: Skill (resident ability) and Bias (evaluator leniency), with each variance fixed at 1 for identification. Standardized loadings and coefficients were tested using z-statistics. RESULTS: Skill loadings were robust across all items (λ = 0.63–0.89, p < 0.001). Four technical domains—Instrument Handling, Flow, Hemostasis, Non-Dominant Hand—showed significant rater bias (λ = 0.51–0.69, p = 0.041–0.010); global tissue-handling items did not (p > 0.30). Bias adjustment reduced item-level residual variance by up to 74% and decreased overall unexplained variance 36%. Bias-corrected composite scores ranked residents exactly as evaluators’ perceived postgraduate level. CONCLUSIONS: SEM isolated and corrected attending bias even in a sparse dataset, providing a practical route to fairer neurosurgical skills assessment. The approach supports milestone-based promotion, national benchmarking, and high-quality labels for forthcoming AI feedback tools. Ongoing institutional accrual will enable cut-score definition, but this proof-of-concept already merits a broader discussion of objective, scalable neurosurgeon-skill analytics.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.