Background/Purpose: The American College of Rheumatology has developed 14 Entrustable Professional Activities (EPAs) for Adult Rheumatology that define broad clinical competency goals for fellowship. At the same time, the Accreditation Council for Graduate Medical Education (ACGME) requires programs to assess and report fellow progress toward a set of comprehensive training Milestones. While these tools provide a helpful framework for guiding and monitoring fellow progress, they are either too general (EPAs) or too comprehensive (Milestones) to serve as ready-made assessment tools for individual clinical rotations. Here, we describe a strategy for creating practical rotation-specific assessment tools that reflect EPAs and produce useful Milestone data.

Methods: In academic year (AY) 18/19, clinician-educator faculty/senior fellows and site directors from each of 3 training hospitals at a large, urban, university-based adult rheumatology program were asked to craft evaluation objectives for continuity clinic and inpatient consult rotations. Objectives were 1) EPA-representing, 2) task-oriented, 3) readily observable, and 4) frequently or uniquely observed during the rotation. Teams also mapped each objective to Internal Medicine Subspecialty Milestones. A Milestone was mapped to an objective if it met 2 or more of the following: 1) it covered all/most task activities; 2) was critical to task; 3) was unique to task; or 4) was not better represented by another objective (for any rotation). Objectives and mapping were reviewed and approved by the Program Evaluation Committee. Using these objectives, fellows were evaluated every 1 – 2 months using a 5-point competency-based scale. Evaluations (scores + comments) were made available to fellows in real time. For semi-annual reviews, each Milestone level was calculated from the average of scores of all mapped objectives and verified/adjusted by the Clinical Competency Committee (CCC). Fellow perception of evaluation process was assessed via anonymous internal and ACGME surveys of our program.

Results: We developed 9 – 13 objectives for each of six core clinical rotations (Table 1) and mapped them to Milestones. The new evaluation objectives and mapping algorithm were introduced in AY 19/20. Based on internal anonymous surveys, fellow satisfaction with the evaluation process increased from 4.0±0.8 out of 5.0 to 4.8±0.4 out of 5.0. On anonymous ACGME surveys, fellow perception of evaluation processes in general rose from 78.6% compliant in AY 18/19 to 95.8% compliant in AY 20/21. Analysis of Milestone maps and Milestone performance trends over time identified some gaps in overall assessment and areas of curricular strength/weakness (not shown). We are now mapping objectives to the new ACGME Milestones for adult rheumatology (Table 2).

Conclusion: We describe a practical approach for developing rotation-specific, task-oriented evaluation objectives that can generate Milestone data useful for tracking progress of individual trainees as well as curriculum effectiveness. In addition, fellow satisfaction with the evaluation process appeared to improve. We believe our approach is highly adaptable and could be useful to a wide range of training programs.

« Back to ACR Convergence 2021

ACR Meeting Abstracts - https://acrabstracts.org/abstract/a-practical-approach-to-competency-based-evaluations-and-milestone-mapping-for-adult-rheumatology-fellowship-programs/