Do outlier assessors provide useful narrative comments?

Le résumé fourni par la source

Introduction Outlier stringent/lenient and range-restricted assessors provide low utility assessment scores, reflecting assessor tendency more than learner performance. The quality of narrative comments generated by these assessors remains unknown.Methods End-of-shift assessments from an academic emergency department were scored using the Quality of Assessment of Learning (QuAL) score. The mean Qual score was calculated for each assessor. Stringency/leniency and range restriction were quantified using the mean-delta method and standard deviation of awarded scores respectively. Outlier vs. non-outlier assessor mean QuAL scores were compared using T-tests. Linear regression was conducted with mean QuAL score as the dependent variable, and assessor range-restriction as the independent variable.Results 2034 assessments were completed by 81 assessors. We identified 20 outlier assessors (10 stringent, 10 lenient). Assessor mean QuAL scores ranged from 1.2 to 5 (mean = 3.9, SD = 0.9). Mean QuAL score of non-outliers (4.0) was significantly higher than outlier lenient assessors (3.1, p = 0.02) but not different from outlier stringent assessors (3.8, p = 0.5). Range restriction was negatively correlated with mean QuAL scores (p = 0.003, R2=0.11)Conclusions Outlier lenient and range-restricted assessors provided lower-than-average quality narrative assessments. These assessors, where both numeric and narrative assessments are of limited utility, are ideal candidates for targeted interventions to improve assessment quality.

Ce résumé expose les affirmations des auteurs. BNTIC ne l’interprète pas comme une validation indépendante des résultats.