Can AI Improve Medical School Admissions? A Reliability Analysis of AI-Generated Multiple Mini Interview (MMI) Stations. (Preprint)

Le résumé fourni par la source

BACKGROUND Multiple Mini Interviews (MMIs) are widely used in medical school admissions to assess applicants’ non-academic attributes in a structured and reliable manner. However, the development of high-quality MMI stations is resource-intensive and dependent on expert input. OBJECTIVE This study explores the utility of artificial intelligence (AI) in the generation MMI stations for the Direct and Graduate Entry Medicine Program admissions process for domestic applicants at Monash Medical School. METHODS A total of 56 MMI stations from the 2025 admissions cycle were evaluated, including 17 AI-generated and 39 traditionally developed stations, administered across 824 domestic applicants for a total of 4,897 applicant-station interactions. We assessed station quality through both reliability (using Cronbach’s alpha to examine internal consistency) and discrimination capability (using standard deviation and range of scores) at the station level. RESULTS AI-generated stations exhibited slightly higher reliability (α = 0.8181) compared to existing stations (α = 0.8081), though this difference was not statistically significant. Both AI-generated and traditionally developed stations demonstrated variable discrimination capability, with some stations from each development method showing excellent combinations of high reliability and strong discriminatory power, while others exhibited ceiling effects that limited their discriminatory power. Importantly, a greater proportion of AI-generated stations achieved excellent reliability (α > 0.85), and a lower proportion demonstrated poor reliability (α < 0.75), compared to traditional stations, suggesting that AI-generated development may enhance the consistency and quality of MMI stations. CONCLUSIONS Our findings presented here highlight the utility of AI as a useful tool for MMI station generation, offering a scalable approach that may reduce the resource burden on faculty while maintaining or enhancing psychometric quality for applicants. Ongoing quality assurance and evaluation remain essential to ensure fairness and validity across the admissions process.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.