Background/Purpose: The Pennsylvania Rheumatology Objective Structured Clinical Examination (ROSCE) is an annual assessment of rheumatology fellows’ skills in different standardized patient (SP) encounters involving a variety of topics related to communication issues. We previously mapped our ROSCE assessment tools to align with Accreditation Council of Graduate Medical Education (ACGME) internal medicine subspecialty milestones in 2018. In 2020, we conducted our ROSCE entirely online. Seventeen trainees from 8 rheumatology programs in the region participated in 6 SP scenarios and were observed by faculty raters. We aim to better tailor our cases and evaluation forms to address milestones of greatest utility to participants. In this study, we surveyed 2020 ROSCE fellow participants and program directors (PDs) regarding their perceptions of the most important competencies for rheumatology fellows to master in order to become excellent clinical rheumatologists.

Methods: We created an online survey using survey monkey asking participants to rank 11 ACGME subspecialty milestones in order of their perceived importance to attaining clinical excellence in rheumatology (Table 1). Content validity was determined by consensus of 5 rheumatology fellowship PDs and key clinical faculty. Participants were invited via email; all 2020 ROSCE fellow participants and regional PDs were invited to respond. Milestones rated in the top 3 and bottom 3 for both faculty and fellows are reported in aggregate. We also mapped our 2020 ROSCE evaluation questions to current ACGME subspecialty milestones.

Results: Five fellow participants and 9 PDs completed the survey (response rate 29% and 56%, respectively). Results were similar for faculty and fellows, with MK2, PC1, PC3, PC4, and PROF2 ranked in the top 3 by at least 25% of both groups (Figure 1a), addressing test interpretation, history-taking, forming a plan, therapeutics, and ethical behavior. Conversely, milestones ranked more frequently in the lowest 3 related to jargon use, system error recognition, transitions across health systems, and cost-effective care (Figure 1b). The 2020 ROSCE evaluation questions included 39 Likert scale questions (in addition to narrative comments) (Table 1). All milestones rated as important by faculty and fellows are represented in evaluations except PROF2.

Conclusion: We surveyed rheumatology faculty and fellows to gauge their perceptions of the most important areas to assess in the ROSCE. We observed a low response rate among fellows, possibly due to survey fatigue. In general, faculty and fellow preferences aligned with respect to “top” and “bottom” ranked milestones, though numbers were small. Nearly all top-ranked milestones, dealing predominantly with patient care, are addressed in our ROSCE cases and evaluations, suggesting that it is well-suited to address these areas. PROF2, addressing ethical conduct, was reported to be of high importance but was not included in prior ROSCE cases. We aim to further tailor our ROSCE cases and evaluations to align with these preference needs, thus providing more meaningful information to PDs and fellows for formative feedback.

« Back to ACR Convergence 2021

ACR Meeting Abstracts - https://acrabstracts.org/abstract/determining-faculty-vs-fellow-preferences-for-rosce-milestone-assessments-is-it-time-for-a-change-in-needs-assessment/