Scheduled Presentations by Date

“How Consistent is Competent? Examining Variance in Technical Skills Assessment” Mathilde Labbé, Meredith Young, Marco A Mascarella, Murad Husein, Philip C Doyle, Lily HP Nguyen



Background:The direct observation and assessment of residentperformance over time is a core tenant of competency-based medical education (CBME). One premise of CBME is that sufficient random assessments of a given skill will reflect a trainee’s level of competence. However, the variability in performance of a particular technical skills for a particular trainee for a given procedure across time remains largely undescribed.


Methods:We conducted a secondary analysis of assessment data generated by a prospective cohort of Otolaryngology – Head and Neck Surgery (OTL-HNS) surgeons with different levels of expertise. Assessment data was generated by trained raters scoring blinded recordings of multiple surgeries using a myringotomy and tube insertion assessment tool. Each participant performed the procedure at least four times. Analysis focused on the variability among groups and for each individual over time as well as a univariate comparison of standard deviations (SD).


Results:A total of 36 assessments from four junior residents, two senior residents and three attending surgeons were reviewed. The SD of scores of junior residents was highly variable (5.8 as compared to 1.8 for senior residents and attending surgeons (F-test, p<0.05)). The range of total scores for each individual varied from 7 to 23 in juniors, 2 to 9 in seniors and 1 to 6 in attending surgeons.


Conclusion: OTL-HNSresidents display highly variable performances for a given procedure, and variability appears to decrease with increasing expertise. Operative skill variability can potentially be underrepresented in CBME; emphasis on an adequate amount of repetitive evaluations for more junior residents is needed to capture the variability of performance present in order to support judgments of competence or entrustability.

Learning Objectives

  1. To understand how variability of performance affects assessment of residents in Competency Based Medical Education (CBME).
  2. To explore options for mitigating the potential unintended consequences of such variability.

Download Presentation Slides (PDF)

No presentation materials have been uploaded for this presentation. Please check back again later.

Scheduling Details

This 7 minute presentation is scheduled to take place on June 2, 2019 at 15:44 in the Salon 9/10.

Authors' Contact Details

Corresponding Author: Dr. Lily HP Nguyen

Senior Author: Dr. Lily HP Nguyen

The corresponding author of this presentation has not yet consented to having their email address published online.