Background/Purpose: Effective diagnosis and management of knee osteoarthritis (KOA) increasingly rely on integrating diverse data sources, including imaging and clinical information. The practice of combining data from several modalities to extract more comprehensive and complementary information for machine learning models that perform better than those that only use one data modality is known as data fusion. Data fusion in machine learning is expected to enhance predictive power for decision-making in health, leading to more reliable outcomes in poor validity conditions. This study aimed to evaluate the use of multimodal deep learning models assisted by transfer learning (TL) to enhance the prediction of KOA progression. The Development and application of a two-step transfer learning strategy utilizing the OAI and MOST datasets were also evaluated

Methods: From the Osteoarthritis Initiative (OAI) and the Multicenter Osteoarthritis Study (MOST) cohorts, the MRI and X-ray images of 973 and 741 knees were selected, respectively as shown in Figure 1. Two primary encoders were used as the backbone for the fusion of features extracted from MRI and X-ray images: Resnet 34 and DenseNet 201. TL strategies included one-step transfer using ImageNet pretrained weights and two-step transfer initially pretraining on one KOA cohort and then fine-tuning and validating on an external KOA cohort.

Results: Results demonstrated that ImageNet-based TL significantly improved the predictive performance of X-ray models, with AUC scores exceeding 0.7 in both the OAI and MOST cohorts (Tables 1 and 2). For MRI-based models, the lack of large-scale pretrained weights limited the performance benefit of TL. Multimodal configurations offered no clear advantage over unimodal or dual-modality models (Table 2). Without TL, DenseNet almost outperformed ResNet (Table 1). However, with TL, neither encoder demonstrated a clear advantage. The two-step TL did not show a significant benefit over one-step TL, highlighting the need for pretrained weights derived from larger KOA datasets.

Conclusion: These findings underscore the potential of transfer learning in predicting KOA progression and help fill an important gap in the focus on progression prediction

Figure 1: The CONSORT diagram of the subject selection process showing the number of patients (n) and number of knees (k) at each selection stage.

Table 1. AUC results for testing different fusion combinations without transfer learning in a) OAI and b) MOST cohorts. Results presented as mean AUC (± standard deviation)

Table 2. AUC results for testing different fusion combinations with transfer learning in a) OAI and b) MOST cohorts. Results presented as mean AUC (± standard deviation)

« Back to ACR Convergence 2025

ACR Meeting Abstracts - https://acrabstracts.org/abstract/enhanced-prediction-of-knee-osteoarthritis-progression-through-deep-learning-based-multimodal-data-fusion/