Validity Evidence for Two Objective Structured Clinical Examination Stations to Assess Core Examination Skills of the Shoulder and Knee

Michael J. Battistone¹, Andrea M. Barker^2,3, J. Peter Beck⁴, Robert Z. Tashjian⁵ and Grant W. Cannon⁶, ¹Rheumatology, Salt Lake City VA Medical Center and University of Utah, Salt Lake City, UT, ²General Internal Medicine, Salt Lake City VA Medical Center and University of Utah, Salt Lake City, UT, ³Family and Preventative Medicine, University of Utah School of Medicine, Salt Lake City, UT, ⁴Orthopaedics, Salt Lake City VA Medical Center and University of Utah, Salt Lake City, UT, ⁵Division of Orthopaedics, Salt Lake City VA and University of Utah, Salt Lake City, UT, ⁶Division of Rheumatology, Salt Lake City VA Medical Center and University of Utah, Salt Lake City, UT

Meeting: 2015 ACR/ARHP Annual Meeting

Date of first publication: September 29, 2015

Keywords: clinical skills, Education, educational innovation and musculoskeletal curriculum, medical, Validity

Session Information

Date: Monday, November 9, 2015

Title: Education Poster I

Session Type: ACR Poster Session B

Session Time: 9:00AM-11:00AM

Validity
Evidence for 2 Objective Structured Clinical Examination Stations to Assess
Core Examination Skills of the Shoulder and Knee Background/Purpose: A
multi-disciplinary group developed two objective structured clinical
examination (OSCE) stations to train and assess trainee’s skills in the
evaluation and management of shoulder pain and knee pain. Our objective was to
examine the validity of these two OSCEs. Methods: Content Two
orthopaedists, two rheumatologists, and a primary care provider developed checklists
of physical exam maneuvers and criteria for guiding rater observations. Content
was proposed by faculty, supplemented by literature review, and finalized through
a modified Delphi process. Simulated cases representing common causes of
shoulder pain and knee pain were constructed. Response Process A
multi-disciplinary cohort of 69 trainees participated in the OSCEs in 2014-15. To
promote accuracy of the simulated patient (SP) responses to assessment prompts,
one faculty member served as SP and another as rater; ratings were recorded in
real time. Internal Structure Two
faculty members independently rated a portion of the cases. Percent agreement
was calculated and Cohen’s kappa corrected for chance agreement on binary
outcomes. Relationship to Other Variables Relationship
to self-assessment of ability to evaluate shoulder pain and knee pain was
explored by written surveys utilizing a 5-point Likert scale. Responses were stratified
into 3 categories – low, medium, and high – and compared with similarly
stratified OSCE scores. Results: Checklists
were developed for the shoulder (21 items) and knee (26 items); scoring rubrics
converted each checklist to a 5-point scale. Using the examination approach in
the checklists, trainees correctly identified rotator cuff pathology 61/69 (88%)
and meniscal disease 62/69(89%) of the time. Inter-rater agreement was 89% for
the knee (k = 0.55) and 96% for the shoulder (k = 0.51). Relationship of
stratified self-assessment and OSCE scores is shown in Table 1; Pearson’s
coefficient indicated no correlation for either shoulder (0.02) or knee (-0.07). Table
1. Conclusion: The
two station OSCE is responsive and consistency in test performance across a
range of trainees over time, with good inter-rater reliability. Lack of
correlation with self-assessment suggests that these OSCEs measure a construct
that is different than learners’ self-confidence to support the continuing use
of these two shoulder and knee OSCE stations in a structured educational
program. The next steps will search for additional validity evidence of these
experiences.

Disclosure: M. J. Battistone, None; A. M. Barker, None; J. P. Beck, None; R. Z. Tashjian, None; G. W. Cannon, None.

To cite this abstract in AMA style:

Battistone MJ, Barker AM, Beck JP, Tashjian RZ, Cannon GW. Validity Evidence for Two Objective Structured Clinical Examination Stations to Assess Core Examination Skills of the Shoulder and Knee [abstract]. Arthritis Rheumatol. 2015; 67 (suppl 10). https://acrabstracts.org/abstract/validity-evidence-for-two-objective-structured-clinical-examination-stations-to-assess-core-examination-skills-of-the-shoulder-and-knee/. Accessed .

« Back to 2015 ACR/ARHP Annual Meeting

ACR Meeting Abstracts - https://acrabstracts.org/abstract/validity-evidence-for-two-objective-structured-clinical-examination-stations-to-assess-core-examination-skills-of-the-shoulder-and-knee/