Background/Purpose: Physician payments in the U.S. are shifting from fee-for-service to a value-based system, which will require quality measures to be extracted automatically from the electronic health record (EHR). However, the feasibility and accuracy of such “e-specification” of quality measures has not been widely reported. In this study, we compared the accuracy of automated electronic vs. manual chart review abstraction for a sample patient safety quality measure.

Methods:  We developed electronic specifications for the following measure – “If a patient receives rituximab, then s/he should be screened for prior Hepatitis B infection with Hepatitis B surface antigen and Hepatitis B core antibody tests prior to rituximab initiation, because Hepatitis B reactivation is a potentially fatal (and preventable) adverse event in this population.”  We used EHR data derived from a tertiary care university center that has almost one million outpatient visits per year. Patients were included if they had at least 2 encounters during the 12 months before and at least one encounter 30 days after their first rituximab dose during the study period (June 2012-January 2016).  We queried EHR structured data to define a denominator population of rituximab users and to classify the receipt of adequate Hepatitis B screening at any time prior to starting rituximab treatment.  Two physicians performed chart reviews, including reviewing all clinical notes and scanned documents, to validate these data.

Results:  The EHR query of structured data identified 1,167 users of rituximab in the denominator for this measure. Physician chart review found that this was an overestimate (321 patients did not receive the medication or had a historical reference to the medication). The automated query identified 618 (out of 1,167) patients in the numerator for this measure. Physician chart review found that this was an underestimate of actual testing, as 11% of the 926 patients in the denominator had additional lab results described in clinical narratives. The use of chart review in addition to electronic queries of EHR data increased the performance on this patient safety measure from 53% to 61%, representing a 15% increase.

Conclusion:  Relying solely on the extraction of structured data from the EHR resulted in an overestimate of the denominator population and an underestimate of the numerator of Hepatitis B screening for this patient safety quality measure. Such misclassifications may threaten the validity of e-measurement and potentially undermine physicians’ incentives to participate in a value-based payment system. As the number of e-Measures expands, accurate reporting will need to rely on additional strategies, including natural language processing for extraction of unstructured data from clinical notes, as well as changes to provider workflows to collect critical data elements in structured fields.