Validation of Online Calibration Modules for the Spondyloarthritis Research Consortium of Canada MRI Scores Based on Real-Time Experiential Learning

Matthew Maksymowych¹, Hanne Boutrup², Jonathan Cheah³, Riccardo Guglielmi⁴, Eric Heffernan⁵, Jacob Jaremko¹, Mats-Peter Johansson⁶, Simon Krabbe⁷, Georg Kroeber⁸, Fardina Malik⁹, Susanne Juhl Pedersen⁷, Sander Shafer⁶, Ulrich Weber¹⁰, Pam Weiss¹¹, Brian Trinh¹², Joel Paschke¹³ and Robert G. Lambert¹⁴, ¹Radiology, University of Alberta, Edmonton, AB, Canada, ²University of Copenhagen, Copenhagen, Denmark, ³Medicine, Hospital for Special Surgery, New York, NY, ⁴Radiology, Ente Ospedaliero Cantonale, Lugano, Switzerland, ⁵Radiology, St. Vincent's University Hospital, Dublin, Ireland, ⁶Rheumatology, Gentofte Hospital, Copenhagen, Denmark, ⁷Center for Rheumatology and Spine Diseases, Copenhagen Center for Arthritis Research, Rigshospitalet, Copenhagen, Denmark, ⁸University of Southern Denmark, Odense, Denmark, ⁹Medicine- Rheumatology, Weill Cornell Medical College, New York, NY, ¹⁰Department of Research, King Christian 10th Hospital for Rheumatic Diseases, Graasten, Denmark, ¹¹Pediatrics, Children's Hospital Philadelphia, Philadelphia, PA, ¹²CARE Arthritis, EDMONTON, AB, Canada, ¹³CARE Arthritis, Edmonton, AB, Canada, ¹⁴University of Alberta, Edmonton, AB, Canada

Meeting: 2017 ACR/ARHP Annual Meeting

Date of first publication: September 18, 2017

Keywords: MRI, Validity and spondylarthropathy

Session Information

Date: Sunday, November 5, 2017

Title: Spondyloarthropathies and Psoriatic Arthritis – Clinical Aspects and Treatment Poster I

Session Type: ACR Poster Session A

Session Time: 9:00AM-11:00AM

Background/Purpose: The appropriate use of imaging-based scoring instruments is usually an ad hoc process based on passive learning from published manuscripts. Moreover, most instruments lack knowledge transfer tools that would facilitate attainment of pre-specified performance targets for reader reliability prior to use in clinical trials and research. We aimed to develop and validate online calibration modules for the SPARCC MRI Scores based on consensus scores from these instrument developers, experiential game psychology, and real-time iterative feedback built into a scoring schematic directly on the DICOM image.

Methods: The SPARCC BME inflammation calibration module is comprised of 50 DICOM cases, each with scans from baseline and 12 weeks after the start of tumor necrosis factor inhibitor therapy. The SPARCC structural score is comprised of 40 cases, each with scans from baseline and after 2 years follow-up. Scans are scored in pairs for status and change scores blinded-to-time-point. Continuous visual real-time feedback regarding concordance/discordance of scoring per SIJ quadrant with expert readers is provided by a color-coding scheme. Reliability is additionally assessed by real-time intra-class correlation coefficient with the first ICC data being provided after 20 cases per SIJ coronal slice (calibration-per-slice), and further ICC data after the entire case has been scored (calibration-per-case). Accreditation for SPARCC BME is achieved with a status score ICC of > 0.8 and a change score ICC of > 0.7, and for SPARCC structural, status score ICC of >0.7 is required for fat and ankylosis, >0.5 for erosion and backfill, and >0.5 for change score in all domains. 14 readers scored the SPARCC BME module and 11 the SPARCC structural module, only 2 having prior experience with these scores, and 9 being rheumatology fellows.

Results: For the SPARCC BME score and the calibration-per-slice method, all 14 readers completed the scoring and attained the required level of proficiency. For the SPARCC structural scores and the calibration-per-slice method, all 11 readers completed scoring, proficiency for status was attained by 10 for each lesion domain, and proficiency for change was attained by 11, 5, 6, 8 for fat, erosion, backfill, ankylosis, respectively. All readers rated the modules as easy and intuitive with average time for reading each case for SPARCC BME and Structural being 9 min and 12 min, respectively.

Conclusion: Experiential online learning is an effective and feasible calibration tool in the scoring of MRI scans.

SPARCC score	Calibration per SIJ Slice			Calibration per Case
SPARCC score	Baseline	12 weeks	Change	Baseline	12 weeks	Change
SPARCC BME	0.97(0.96-0.97)	0.94(0.91-0.96)	0.94(0.92-0.95)	0.77(0.71-0.84)	0.86(0.81-0.91)	0.69(0.59-0.78)
SPARCC Fat	0.84(0.73-0.94)	0.89(0.83-0.96)	0.81(0.74-0.88)	0.85(0.78-0.93)	0.86(0.79-0.92)	0.90(0.87-0.94)
SPARCC Erosion	0.83(0.74-0.91)	0.81(0.73-0.89)	0.46(0.28-0.63)	0.59(0.43-0.75)	0.46(0.26-0.65)	0.39(0.25-0.53)
SPARCC Backfill	0.58(0.38-0.78)	0.79(0.70-0.88)	0.49(0.31-0.67)	0.67(0.49-0.86)	0.61(0.43-0.78)	0.56(0.38-0.74)
SPARCC Ankylosis	0.92(0.88-0.96)	0.96(0.92-1.00)	0.71(0.56-0.86)	0.95(0.87-1.00)	0.92(0.85-0.98)	0.54(0.31-0.77)

Disclosure: M. Maksymowych, None; H. Boutrup, None; J. Cheah, None; R. Guglielmi, None; E. Heffernan, None; J. Jaremko, None; M. P. Johansson, None; S. Krabbe, None; G. Kroeber, None; F. Malik, None; S. J. Pedersen, None; S. Shafer, None; U. Weber, None; P. Weiss, None; B. Trinh, None; J. Paschke, None; R. G. Lambert, None.

To cite this abstract in AMA style:

Maksymowych M, Boutrup H, Cheah J, Guglielmi R, Heffernan E, Jaremko J, Johansson MP, Krabbe S, Kroeber G, Malik F, Pedersen SJ, Shafer S, Weber U, Weiss P, Trinh B, Paschke J, Lambert RG. Validation of Online Calibration Modules for the Spondyloarthritis Research Consortium of Canada MRI Scores Based on Real-Time Experiential Learning [abstract]. Arthritis Rheumatol. 2017; 69 (suppl 10). https://acrabstracts.org/abstract/validation-of-online-calibration-modules-for-the-spondyloarthritis-research-consortium-of-canada-mri-scores-based-on-real-time-experiential-learning/. Accessed .

« Back to 2017 ACR/ARHP Annual Meeting

ACR Meeting Abstracts - https://acrabstracts.org/abstract/validation-of-online-calibration-modules-for-the-spondyloarthritis-research-consortium-of-canada-mri-scores-based-on-real-time-experiential-learning/