Background/Purpose: Radiographs are used to detect and monitor joint damage due to rheumatoid arthritis (RA).Scoring methods such as the Sharp van der Heijde tool (SVH) quantify joint space narrowing (JSN) and erosions. Obtaining SVH scores is time-consuming and requires expertise not always available. We aimed to develop and validate a deep learning system for automated detection and prediction of SVH scores in hand radiographs of patients with RA.

Methods: We used a convolutional neural network (CNN) based algorithm (You Only Look Once (YOLO)v5l6) trained on an object detection model (COCO-Common Objects in Context) to detect joints in 240 training and 89 test pediatric hand radiographs from the Radiologic Society of North America database. Radiographs were annotated by boxing and labeling the joints of interest: proximal interphalangeal, metacarpophalangeal, wrist, distal radius, distal ulna. Images were augmented using mix-up (mixing features of different images into one), mosaic (combining 4 training images into 1 image), rotation, translation, scaling, and shearing. The joint detection model was validated with 54 clinician-annotated radiographs from 4 adult RA patients followed for 9-13 years (10-12 images per patient) (joint detection gold standard). We applied a supervised vision transformer model (VTM) to predict each joint’s SVH erosion and JSN score. The VTM was validated using 2249 hand radiographs with clinician-assigned SVH scores from 381 RA patients from the Canadian Early Arthritis Cohort (SVH score gold standard). We applied techniques used to train highly class-imbalanced datasets including weighted sampling, stratified data-split, data augmentation and transfer learning (TL). As the joint detection model was trained to detect the whole wrist and we had clinician-assigned SVH scores for individual wrist joints, we trained a separate multi-task model to predict wrist joint scores from whole wrist images. The performance of the VTM to predict joint scores was compared to CNN-based EfficientNetV2 and MobileNetV3. Model accuracy for joint detection is reported as the F1-score (reflecting model precision and model recall) and mean absolute precision (mAP) for a range of Intersection-over-Union (IoU) measures reflecting the overlap between clinician and model assigned bounding boxes of detected joints. Accuracy for SVH score prediction is reported as root mean squared error (RMSE) and balanced accuracy.

Results: The joint detection model accurately identified target joints (pediatric data FI-score = 0.991, map0.1 = 0.993 with an IoU threshold of 0.1 ; adult data F1-score = 0.812, map0.1 = 0.871 (n=54) (Table 1). Applying TL improved the joint detection model’s mean precision by 0.05 over the COCO model. The VTM predicted JSN and erosion SVH scores with high accuracy (RMSE JSN 0.91, erosion 0.93). The multi-task models predicted SVH erosion and JSN scores of wrist joints with moderately high accuracy (0.6-0.91). EfficientNetV3 performed better for wrist joints (VTM vs EfficientNetV3 average difference 0.10) (Figure 1).

Conclusion: Automated deep learning systems accurately identify and predict joint damage in hand radiographs from patients with rheumatoid arthritis and may aid in monitoring joint damage.

« Back to ACR Convergence 2023

ACR Meeting Abstracts - https://acrabstracts.org/abstract/artificial-intelligence-models-for-computer-assisted-joint-detection-and-sharp-van-der-heijde-score-prediction-in-hand-radiographs-from-patients-with-rheumatoid-arthritis/