Background/Purpose: Over the last 20 years musculoskeletal ultrasound (MSUS) was established as a disease outcome measurement instrument. MSUS target domains, including consensual theoretical definitions of pathological processes, operational definitions of elementary lesions, acquisition standards and scoring systems developed as part of OMERACT methodology have been validated via a step-wise process consisting of Delphi surveys to produce theoretical and operational definitions, and intra- and interobserver reliability exercises based either on still images or videos, or on live acquisition using patients or healthy subjects to test such theoretical agreement. Our goal was to evaluate factors influencing Delphi- and reliability exercises in MSUS for developing recommendations to standardise consensus and validation studies aiming at producing new scoring systems.

Methods: A systematic search of the Central, Embase, and Medline databases was performed by two independent reviewers in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. Original articles published in English up to May 2024, reporting on Delphi consensus studies and intra- and interobserver reliability exercises performed on adult patients or healthy subjects according to the OMERACT Ultrasound Working Group (USWG) methodology were included. Attributes of the consensus studies and reliability exercises were extracted from the included studies using a predefined sheet.

Results: Out of 2744 articles, 25 were included in the final review. Of these, 15 reported on Delphi studies, while 25 reported on 33 reliability exercises (16 image-based and 17 patient-based). The majority of the Delphi studies included the target domains elementary lesions (87%) and scoring systems (73%) (Table 1). 86% consisted of up to 3 rounds, and 86% utilised ≥75% as a cut-off for reaching agreement among participants. The no. of participants ranged from 13 to 49, with response rates between 68-100%. With respect to reliability exercises, the no. of images used ranged from 23 to 152, the no. of patients from 6 to 13 and the no. of expert participants from 12-35 (image-based) and 9-19 (patient-based) (Table 2). All but one exercise featured two rounds, and all of them assessed both intra- and interobserver reliability. Exercises used various forms of kappa statistics, while a single exercise utilised an intraclass correlation coefficient to measure reliability. All but a single (97%) of reliability studies evaluated either elementary lesions and/or scoring systems. While 59% of patient-based reliability exercises featured a pre-training exercise, this was the case only in 6% of image-based exercises.

Conclusion: The included Delphi consensus and reliability exercise studies focused on either elementary lesions or scoring systems. Variation was observed in the no. of included experts, images/patients utilized, and the integration of pre-training exercises. These results represent the first step for producing recommendations to harmonise consensus and reliability studies in MSUS, to aid OMERACT groups and other researchers in developing/validating new definitions/scoring systems for imaging outcome measures.

« Back to ACR Convergence 2024

ACR Meeting Abstracts - https://acrabstracts.org/abstract/harmonising-consensus-and-reliability-studies-in-musculoskeletal-ultrasound/