Background/Purpose: Shoulder pain is common.  Subacromial bursa and intra-articular glucocorticoid injections can provide pain relief while avoiding the risks associated with systemic therapies.  The American Board of Internal Medicine recommends internists achieve competence in arthrocenteses, but medicine residents at our institution report little practice doing them.  Furthermore, validated assessment methods for these procedures are not published, making it difficult to decide when trainees achieve competence.  We aim to develop and gather validity evidence for a tool that assesses the performance of shoulder injections.

Methods: A multi-disciplinary team of internists, physiatrists, and rheumatologists created a checklist and global rating score to assess the performance of shoulder injections.  We used a modified Delphi to reach consensus on what to include as checklist items and assigned points to each one, weighing key maneuvers more heavily.  We taught medicine interns the procedure on simulation models and observed them performing subacromial bursa injections for impingement syndrome during objective structured clinical examinations (OSCE).  We graded their performances against our checklist, our global rating score, and a modified version of the Global Rating Scale for the Evaluation of Technical Skills in the Operating Room (GRSETSOR) that excluded items unique to surgery.  Two graders evaluated 30% of the OSCEs.  We totaled our checklist and global rating scale scores to compare with total GRSETSOR scores with Pearson’s correlation coefficient in Excel.  We calculated the intraclass correlation coefficient (ICC) to measure inter-rater reliability with SPSS (2019, Chicago, IL); 13 dual-graded OSCEs were needed to measure an ICC of 0.7 with 95% confidence.

Results: A global rating score and ten items related to pre-procedural, procedural, and post-procedural shoulder injection tasks met consensus for inclusion in our checklist.  Checklist items totaled 28 possible points, and the global rating score ranged from one to nine points, set with behavioral anchors.  45 interns participated in our study, consenting to let us use their OSCE scores to calculate psychometrics.  Out of the 45 participants, 19 had two raters evaluate their OSCE performances simultaneously.  Our checklist and global rating score totals correlated well with the modified GRSETSOR scores (r=0.71, r² 0.50, p < 0.0001, figure 1).  ICC measured 0.835 (95% confidence interval 0.58-0.94) on evaluations with two raters.

Conclusion: We developed a tool that assesses the performance of shoulder injections with strong supporting validity evidence.  Our use of a multi-disciplinary team and modified Delphi method enhances content validity, while strong correlation of total scores with the modified GRSETSOR and high inter-rater reliability indicate relational and internal structure validity, respectively.  This tool can be used to assess simulated performance of shoulder injections; further study is needed to measure its utility in clinical settings.

Reference:
Doyle JD, Webber EM, Sidhu RS. A universal global rating scale for the evaluation of technical skills in the operating room. Am J Surg. 2007;193(5):551‐555. doi:10.1016/j.amjsurg.2007.02.003

Figure 1: Correlation of validated Global Rating Scale for the Evaluation of Technical Skills in the Operating Room scores with total scores from our developed Checklist and Global Rating Score for Shoulder Injections. Pearson’s correlation coefficient (r), R-squared (r2), and p value were calculated and demonstrate positive correlation between the two measures.

« Back to ACR Convergence 2020

ACR Meeting Abstracts - https://acrabstracts.org/abstract/development-and-validity-evidence-for-a-tool-to-assess-the-performance-of-shoulder-injections/