Background/Purpose: The New York City Rheumatology Objective Self Assessment
Clinical Exam (NYC-ROSCE) annually assesses 1^st and 2^nd
year fellow competencies in areas such as professionalism and communication
skills in a patient-interactive setting.  This year’s exam used
inter-professional evaluators to allow for a complete 360-degree comparison of ratings
from physician-evaluators (MD), nurse-evaluators (RN) and patient-actors (PT).

Methods: The 2015 NYC-ROSCE
included 5 patient-centered stations focusing on rheumatic diseases.  The
stations focus on psychosocial dilemmas that occur in the context of clinical
care in order to assess fellow professionalism and other elements of the
patient-doctor relationship. Rheumatology fellows (n=25), faculty MD-evaluators
(n=25) and RN-evaluators (n=25) from 7 rheumatology training programs
participated. Professional actors (n=25) were trained to role-play patients.
Written quantitative assessments of the fellows were made at each station (9-point
Likert scale) by MD-, RN- and PT-evaluators in the areas of maintaining
composure, professionalism, and empathy. Immediate oral feedback was given to
each fellow by all three evaluators following each encounter. Repeated measures
ANOVA was performed to compare the mean ratings among the evaluators in all
three areas. Tukey’s method was used to preserve the overall Type I error when
making multiple comparisons. Spearman correlation coefficient was calculated to
assess the relationship between ratings for professionalism and empathy at each
station for each type of evaluator.

Results: When assessing the
areas of professionalism, composure and empathy, there was a statistically significant
difference between evaluators’ mean ratings, with RN-evaluators rating fellows
the lowest and patient-actors the highest, suggesting that physicians, nurses
and patients may apply different criteria when assessing competencies during
observed encounters (table1). The ratings for professionalism and empathy were highly
correlated for all evaluators at the majority of stations (data not shown).

Conclusion:
The
introduction of the RN-evaluator to the MD-evaluator and patient-actor to our
yearly ROSCE offered a unique inter-professional 360-degree perspective of the
patient-physician encounter. When assessing specific competencies, the MD-,
RN-evaluators’ and patient-actors’ scores were disparate, likely reflecting
their differing professional experiences and the way in which those affect how
they value/rate the same event. The ultimate “true” evaluation is likely a composite
of these individual perspectives. As this is the first subspecialty
inter-professional ROSCE to compare raters, further research will be needed to
understand the significance of the differing assumptions and expectations of
MDs, RNs and patients vis-à-vis the patient-physician interchange.

Table
1. Multiple Comparisons of Mean Ratings among Evaluators