Background/Purpose:

We developed an objective structured clinical examination (OSCE) station to guide preceptors’ observations of trainees’ performance of simulated knee arthrocentesis and to organize feedback in preparation for performing these procedures in patient care. The aim of this project was to examine the evidence for validity of this OSCE.

Methods:

Content:

An orthopedic surgeon, a rheumatologist, and a primary care provider with expertise in musculoskeletal (MSK) medicine developed a checklist to guide rater observations in evaluating arthrocentesis technique. Content was proposed by faculty, supplemented by literature review, and finalized through consensus.

Response Process

A multi-disciplinary cohort of 71 learners (53 postgraduate trainees, 12 physician assistant students, 5 advanced practice nursing students, and one university undergraduate student) participated in the OSCEs in 2016-2017. To promote accuracy of the simulated patient (SP) responses to assessment prompts, one faculty member served as the SP and another as rater; ratings were recorded in real time.

Internal Structure

Two faculty members independently rated a portion of the cases. Percent agreement was calculated and Cohen’s kappa corrected for chance agreement on binary outcomes.

Relations to other variables

Relationship to self-assessment of confidence and competence to perform knee arthrocentesis was explored by written surveys utilizing a 5-point Likert scale. Response scores were compared with OSCE scores through Pearson’s correlation coefficient.

Results:

Checklists were developed for knee arthrocentesis (19 items). The checklist was scored by assigning one point for each error observed, in order to capture multiple errors; thus a score of 0 indicated no errors were noted. Mean score was 3.7 (range = 0-8; s.d. = 1.9). Frequency distribution of scores is reported in the Figure:

Inter-rater agreement was near perfect at 93% (k = 0.8), as shown in the Table:

Observed Agreement = (116 + 31)/158 = 0.93

Chance Agreement = 0.64

Cohen’s kappa = 0.8

Pearson’s coefficient indicated no correlation between self-assessment and OSCE performance (-0.002).

Conclusion:  

Validity evidence supports use of this OSCE in educational programs preparing learners for clinical settings where MSK procedures may be performed. Evidence for validity includes systematic development of content, response process control, and demonstration of acceptable interrater agreement. Lack of correlation with self-assessments suggests that the OSCE measures a construct different than self-perceived ability.