Background/Purpose: Rheumatoid arthritis (RA) is a common chronic autoimmune disease characterized by inflammation of the synovium, leading to joint space narrowing and osseous erosions. The most widely used method for quantifying joint damage is the visual inspection of radiographic images by highly trained readers. This subjective, time-consuming, non-scalable method is an impediment to research on factors associated with RA joint damage and lack of quantitative information on progression of damage may delay appropriate treatment decisions by clinicians. To develop a rapid, automated scoring method and perform a rigorous and unbiased evaluation of method performance, we designed the RA2-DREAM Challenge, a crowdsourced competition to engage the international community of problem solvers (http://www.synapse.org/ra2).

Methods: We provided Challenge participants with 674 sets of radiographic images (hands/wrists and feet) and expert-curated modified Sharp/van der Heijde (SvH) scores on 562 unique patients from two completed NIH supported clinical studies, CLEAR and TETRAD. The data for participants were divided into three subsets: training (367 sets of images and corresponding scores), leaderboard (119 sets of images, no scores), and test datasets (188 sets of images, no scores). Challenge participants submitted containerized methods to automatically score overall RA-related damage (Sub-challenge 1), joint space narrowing (Sub-challenge 2), and erosions (Sub-challenge 3). Participants were allowed multiple submissions during the training phase, up to three submissions during the leaderboard phase, and one final submission. Submitted models were evaluated using a model-to-data framework against the manually curated scores using the weighted root mean square error (RMSE). To determine the overall performance, we ranked each team according to their RMSE, determined the robustness of the ranking by bootstrapping the predictions, and evaluated using the Bayes factor.

Results: We received 189 valid submissions from 26 teams in this open science competition. From the 16 final submissions, four teams were named top performers across the three Sub-challenges. Winning algorithms used various approaches, including joint segmentation annotation, deep learning neural networks (e.g. YOLO, RCNN, ResNet, XGBoost), regression modeling, and ensembled methods. In addition, the submitted models were validated using a set of 50 additional patient images from an independent study, the TEAR Trial. These validation analyses revealed results similar to those of the original Challenge. All submitted models were containerized to aid in the goal of producing a rigorous and reproducible scoring methodology.

Conclusion: After further validation, algorithms from this Challenge may provide feasible, accurate, and quick methods to quantify joint damage in RA. These and other artificial intelligence approaches may be applied to radiographic images from electronic health records to quantitate radiographic damage in RA, providing many datasets for research on factors influencing damage. Automated scoring of damage in RA may also help rheumatology health professionals to make prompt treatment decisions to improve patient outcomes.

« Back to ACR Convergence 2021

ACR Meeting Abstracts - https://acrabstracts.org/abstract/a-crowdsourcing-approach-to-develop-machine-learning-models-to-quantify-radiographic-joint-damage-in-rheumatoid-arthritis/