Background/Purpose: Since January 2021, providers in our practice have used the Encounter Appropriateness Score for You (EASY) to document their perception of the appropriateness of telehealth or in-person care after every outpatient rheumatology encounter. In this project, we report the development of a predictive model that uses EASY scores and other variables to identify future encounters that could be done by telehealth. We then demonstrate the utility of this model through a pilot implementation.

Methods: The EASY score asks providers to rate each in-person and telehealth encounter as follows: “Which of the following encounter types would have been the most appropriate for today’s visit? (irrespective of the pandemic).” 1 = Either in-person or telehealth acceptable, 2 = in-person preferred, 3 = telehealth preferred. The predictive model was developed using a 70/30 partition of training/test sets of outpatient rheumatology encounters in our practice with EASY scores from 1/1/21 – 12/31/21. The model used logistic regression to predict the outcome of a future encounter’s EASY score with “in-person preferred as the control class (0) and either in-person or telehealth acceptable and telehealth preferred as the class of interest (1). A validation set of encounters from January 2022 assessed model performance in comparison to the test set. To pilot the model, we presented 3 providers with future visits from 3/1/22 – 5/31/22 that the model identified as candidates for a switch in encounter type, along with an equal number of randomly selected fake encounter switch predictions. The providers could accept or reject the proposed change in encounter type, and they were blinded as to whether the proposed changes were true or fake. Cohen’s kappa was used to measure concordance between the true versus fake predictions and providers’ acceptance of those predictions.

Results: During model development, the training and test sets included 10,551 and 4,541 encounters, respectively. The model AUC was 0.831. We set the model’s operational point at a positive predictive value (PPV) of 0.80 for correctly identifying visits appropriate for telehealth with a sensitivity of 0.384; this was better than existing practice, which had a PPV of 0.766 at the same sensitivity (Figure 1). The validation set of 1,244 encounters showed similar performance. Significant contributors to the model included patient age, diagnoses, medications, RAPID3 scores, and provider preference for telehealth (Table 1). The pilot implementation included 114 encounters. Providers accepted 37/56 (66%) of the true predicted changes compared to 11/58 (19%) of the fake predictions; standard Cohen’s kappa = 0.4722, p < 0.001 (Figure 2). Of the patients who were offered a change, 17/22 (77%) agreed.

Conclusion: By measuring provider perceptions of telehealth appropriateness in every visit, we are able to predict ahead of time visits that could be done by telehealth. When providers and patients were offered these changes, the majority agreed. Thus, this predictive model has the potential to expand access to telehealth care to patients for whom it is appropriate.

« Back to ACR Convergence 2022

ACR Meeting Abstracts - https://acrabstracts.org/abstract/development-and-pilot-implementation-of-a-predictive-model-to-identify-visits-appropriate-for-telehealth/