Background/Purpose:  Artificial intelligence (AI) techniques including deep learning have been rapidly evolving and shown good performance in many fields over recent years. However, there are few cases that apply these techniques to medical field, since it is difficult to collect dataset for training. In our research, to develop an AI-aided automatic radiographic scoring system for quantitating degree of bone destruction, we aimed to develop a learning-based model to automatically detect hand joint ankyloses in radiographic images.

Methods:  Our method consists three parts: (i) collecting dataset with an annotation tool, (ii) predicting hand joint ankyloses using deep learning, and (iii) visualizing where the model focuses in image. First, to collect and annotate dataset for deep learning efficiently, we developed the annotation tool (Fig. 1). A total of 216 radiographic images of hands were randomly obtained from patients who had visited our division at Keio University Hospital in 2015. Forty-three images were identified as having ankyloses in hand joints including wrist joints, MP joints and PIP/IP joints in agreement with a well-trained rheumatologist and radiologist. In learning phase, images were randomly divided into five sets (fold No.1 to 5) for 5-fold cross validation. We utilized ResNet[1], a deep learning model for classification of images, to predict hand joint ankyloses. In addition, we visualized where the model focuses in image by using Grad-CAM[2]. The ROC analysis was performed to evaluate performance of the model.

Results:  As a performance of detecting hand joint ankyloses, our model presented a precision value of 0.78, recall value of 0.86, and F-measure value of 0.82. Mean area under the ROC curve was 0.90 (Fig. 2). When visualizing the result of the model by Grad-CAM, there were some cases where ankylosis in hand joints in the image contributed to the inference results.

Conclusion:  A model based on deep learning to automatically detect hand joint ankyloses in radiographic images was developed with relatively small samples, which suggests that the predictive performance may increase by collecting more training dataset. Our developed model needs to be validated in an independent set of images and tested against other experienced doctors. In the next step of development, we are elaborating a plan for a deep learning-based scoring system to evaluate erosion and joint space narrowing.

References:

[1] HE, Kaiming, et al. Proceedings of the IEEE conference on computer vision and pattern recognition. 2016.

[2] R. R. Selvaraju, et al. 2017 IEEE International Conference on Computer Vision (ICCV). 2018.

Fig. 1

Fig. 2