Background/Purpose: Despite the prevalence of musculoskeletal conditions, the majority of internal medicine trainees at the University of Maryland complete residency with little exposure to rheumatology and score in 16th to 35th percentiles on the rheumatology portion of the annual in-training-exam. In order to improve trainee knowledge and clinical decision making in rheumatology, we created a web-based, self-directed core rheumatology curriculum; the first module addresses the topic of gout. Web-based learning is a standardized, convenient, and accessible method to deliver didactics. We assessed the efficacy of the gout module by using the script concordance testing (SCT) method.

Methods: This prospective observational study included internal medicine trainees at the University of Maryland Medical Center. Participants completed a baseline SCT and an electronic module on gout pathophysiology, clinical presentation, and therapeutic management. Immediate post-testing was performed with an identical SCT. A validated SCT was created for each module in the curriculum. Pre- and post-test scores were compared to the scores of the expert panel, which included 2 rheumatology fellows, 9 academic rheumatologists, and 1 community rheumatologist. A one-way ANOVA was used to compare trainee and expert groups as well as pre- and post-test scores. Cronbach’s alpha was used to calculate test reliability. An effect size was calculated using Cohen’s d. A one-way ANOVA was used to compare the effect of post-graduate year and academic year on SCT score.

Results: One hundred twelve trainees completed paired pre- and post-tests for analysis. The 20-case SCT achieved high reliability (Cronbach alpha > 0.75). At baseline, the trainees’ average SCT score was 31 points (M=31.71, SD=3.15); whereas the experts’ average SCT score was 40 points (M=40.65, SD=1.72). After the didactics, trainees’ SCT scores increased on average 2.56 points F (1, 232) = 43.12; (p<.0001). Cohen’s d showed a strong effect size (d=1.48). Expert SCT scores were on average 8.9 points higher than trainee pre-test scores; whereas at post-test experts’ SCT scores were on average 6.3 points higher. Both of these differences were statistically significant (p<.0001). There were no significant score differences between second and third year residents on pre- or post-test (p>.05). In subgroup analysis by academic year, 2015 trainees scored on average 1.71 points lower on the pre-test compared to 2014 trainees (p=.01). There were no significant differences between 2014 and 2016 trainees on the pre-test (p>.05). There were no significant differences between academic years on the post-test SCT (p>.05).

Conclusion: Trainee test scores significantly increased after the educational intervention in this study. There were no significant differences in post-test SCT scores when analyzed by academic year or post-graduate year. Expert SCT scores were higher at baseline and remained higher after the didactics, which lends support to the construct validity of the tool as the experts had higher clinical competence. Re-testing after 12 months to evaluate durability of knowledge is currently ongoing. 

« Back to ACR/ARHP Annual Meeting 2016

ACR Meeting Abstracts - https://acrabstracts.org/abstract/use-of-script-concordance-testing-to-evaluate-the-efficacy-of-a-web-based-module-on-gout-three-years-of-experience/