Background/Purpose: Rheumatology objective structured clinical examinations (ROSCEs) are assessment tools to evaluate learner performance in simulated clinical experiences. ROSCEs allow standardization of learner assessment in competencies that are challenging to measure in other ways. While several regional groups in the United States conduct ROSCEs, none have been validated for use across programs. We aimed to establish agreement among subspecialty experts regarding the important benchmarks for rheumatology fellows to achieve in 11 theoretical ROSCE stations.
Methods: First, each author independently created a grid listing important attributes to assess in 11 possible ROSCE stations. The grids were combined (by LCS) into a 173-item checklist survey to assess the importance of each item for future use in ROSCEs. IRB exemption was granted to send the survey by email to 37 expert rheumatologists, defined as individuals who have helped create and administer regional ROSCEs. Participants in the expert panel survey rated the importance of each item using a 5-point Likert scale (1=not important to 5=very important). Minimum and maximum scores, the mean, standard deviation, and lower 95% confidence level for the mean were calculated for each survey item. Consensus for high importance was pre-defined as a lower bound of the 95% CI ≥ 4.0.
Results: 11 individuals completed the expert panel survey. 123 of the 173 items (71%) met statistical cutoff for consensus to retain, distributed among the 11 proposed stations. Several items were rejected that had population means of ≥ 4.0 but did not meet the pre-determined definition for consensus. The percentage of retained items for individual stations ranged from 23.5 to 100% (Table). All items were retained for the core interpersonal and professionalism items included as part of general patient counseling and for x-ray interpretation tasks. Only 23% of items were retained for a station involving rehabilitation medicine, and 46% of items for a microscope use/synovial fluid analysis station.
Conclusion: In this single round expert panel survey, we established national consensus on 123 items to assess on 11 proposed ROSCE stations. This study represents the first use of such rigorous methods to establish checklist content agreement for an OSCE in any medical field. Next steps include 1) defining behavioral anchors (milestones) for performance on individual stations and 2) finalizing checklists and station manuals that can be shared among rheumatology programs nationally to improve the validity of regional OSCEs and initiate use in programs without regional collaborators. Use of these checklists by regional rheumatology groups will create the framework for collection of summary data and perhaps eventual creation of a high-stakes rheumatology OSCE.
Table: Item retention summary. Items were retained if statistical analysis revealed a lower bound of the 95% CI ≥ 4.0.
|
Station |
Number of items assessed |
Number of items retained |
Agreement score range |
Percentage of items retained |
|
Phone call from primary care provider |
18 |
10 |
1.98 – 5 |
55.5% |
|
X-rays |
19 |
19 |
4.19 – 5 |
100% |
|
Non-formulary request letter to insurance company |
16 |
12 |
3.03 – 5 |
75% |
|
Osteoporosis/DXA interpretation |
17 |
14 |
2.98 – 4.74 |
82.4% |
|
Rehabilitation |
17 |
4 |
2.51 – 4.6 |
23.5% |
|
General counseling |
13 |
13 |
4.08 – 5 |
100% |
|
Counseling: anti-Ro+ woman pre-conception |
13 |
11 |
3.13 – 5 |
84.6% |
|
Counseling: new scleroderma |
11 |
8 |
3.13 – 4.6 |
72.7% |
|
Counseling: new biologic for RA |
14 |
9 |
3.34 – 5 |
64.3% |
|
Knee arthrocentesis |
20 |
16 |
3.18 – 5 |
80% |
|
Microscope synovial fluid |
15 |
7 |
2.70 – 5 |
46.7% |
|
Total |
173 |
123 |
1.98 – 5 |
71.1% |
Disclosure:
L. G. Criscione-Schreiber,
None;
M. B. Bolster,
American College of Rheumatology,
6,
Rheumatology Research Foundation,
6,
American College of Rheumatology,
6;
B. L. Jonas,
None;
R. Sloane,
None;
J. Hawley,
None;
K. S. O’Rourke,
None.
ACR Meeting Abstracts - https://acrabstracts.org/abstract/expert-panel-consensus-on-content-of-a-rheumatology-objective-structured-clinical-examination/