Background/Purpose: Self-assessment is the most frequently used method in the evaluation of continuing medical education (CME) programs. This multi-institutional project was designed to examine convergent validity of learners’ self-assessment competency ratings with their scores on a 2-station objective structured clinical examination (OSCE) during a musculoskeletal (MSK) CME program.
Methods: The VA Salt Lake City Health Care System developed a 3-day MSK Mini-Residency program in collaboration with six other VA facilities. This program for primary providers emphasizes physical examination and clinical management of common shoulder and knee disorders. Curriculum was introduced through focused didactics followed by small group hands-on practice sessions involving simulated patients. At the conclusion of the program, participants completed a self-reported competency assessment utilizing a 5-point Likert scale and an OSCE for both the shoulder and knee with a rater unaware of the self-reported competency. Self-assessment ratings were classified as low (<4.0), medium (4.0-4.5), and high (>4.5). OSCE ratings were classified with the same numerical boundaries based on a weighted scoring system for each component of the physical exam and then percentage of exam completed correctly (low <80%, medium 80%-90%, high >90%).
Results: A total of 107 and 109 participants completed all components of the shoulder and knee assessments, respectively. Over half the participants were rated high on both self-assessment and OSCE for the shoulder, though correlation was weak (Pearson coefficient = 0.28). For the knee, the OSCE ratings were more variable with approximately one-third of participants in each category while the majority of the participants assessed themselves as being highly competent. Correlation was again weak (Pearson coefficient = 0.05).
Table 1. Self-assessment ratings compared to OSCE ratings for shoulder.
|
SHOULDER
|
OSCE Rating |
|
|||
|
Low |
Med |
High |
|||
|
Self-Assessment Rating |
Low |
0 |
1 (0.9%) |
0 |
1 (0.9%) |
|
Med |
5 (4.7%) |
8 (7.5%) |
10 (9.3%) |
23 (21.5%) |
|
|
High |
9 (8.4%) |
18 (16.8%) |
56 (52.3%) |
83 (77.6%) |
|
|
|
14 (13.1%) |
27 (25.2%) |
66 (61.7%) |
107 |
|
Table 2. Self-assessment ratings compared to OSCE ratings for knee.
|
KNEE |
OSCE Rating |
|
|||
|
Low |
Med |
High |
|||
|
Self-Assessment Rating |
Low |
0 |
0 |
1 (0.9%) |
1 (0.9%) |
|
Med |
13 (11.9%) |
7 (6.4%) |
8 (7.3%) |
28 (25.7%) |
|
|
High |
26 (23.9%) |
28 (25.7%) |
26 (23.9%) |
80 (73.4%) |
|
|
|
39 (35.8%) |
35 (32.1%) |
35 (32.1%) |
109 |
|
Conclusion: Following an intense MSK CME program, the majority of participants rated themselves as highly competent for the shoulder and knee. However, correlation of self-reported competency and OSCE rating was weak, particularly for the knee. Greater variability in the knee OSCE rating and correlation coefficient may be due to the exam maneuvers being more technically difficult to perform as they require more “hands-on” skills of the examiner. These data suggest that an OSCE should be included in evaluation of participants in CME programs to assess competency in the clinical skills being taught.
Disclosure:
A. M. Barker,
None;
M. J. Battistone,
None;
J. P. Beck,
None;
J. Butler,
None;
M. Grotzke,
None;
T. A. Huhtala,
None;
A. C. Cannella,
None;
D. I. Daikh,
None;
M. A. Fang,
None;
A. A. Lazzari,
None;
P. Roldan,
None;
J. M. Von Feldt,
None;
G. W. Cannon,
None.
« Back to 2013 ACR/ARHP Annual Meeting
ACR Meeting Abstracts - https://acrabstracts.org/abstract/objective-assessment-of-musculoskeletal-physical-examination-skills-during-continuing-education-programs-for-primary-care-providers-adds-significant-information-not-obtained-through-self-assessment/