Background/Purpose: Cognitive impairment (CI) is common in systemic lupus erythematosus (SLE) patients, however there is no standard screening tool available. The American College of Rheumatology Neuropsychological Battery (ACR-NB) is the gold standard diagnostic tool but requires trained personnel and 1-hour to complete. The Automated Neuropsychological Assessment Metrics (ANAM) is a promising screening tool as it is a shorter, self-administered battery. We previously developed a composite index for interpreting ANAM, and in this study we aimed to provide further evidence on the construct validity of the ANAM as a screening tool for detecting CI in SLE compared to ACR-NB. Objectives: 1) to determine the best subtests and score types of ANAM to predict CI using Classification and Regression Tree (CART) analysis, 2) provide a new approach for interpreting ANAM results, and 3) compare the performance of ANAM composite index to the best CART model.

Methods: 300 consecutive consenting adult SLE patients attending a single center between 2016 and 2019 completed the ANAM and ACR-NB on the same day. Patients were classified CI or non-CI based on ACR-NB. CI was defined by a z-score of ≤-1.5 in ≥ 2 domains; non-CI if no domain had a z-score of ≤-1.5. ANAM has 15 subtests with each test comprising different score types (percentage of correct responses [PCT], mean reaction time [MR], throughput [TP], and coefficient of variation [CV]). We built 6 models including all subtests, but each model included a different set of score types. Each model was run through CART analyses with k-fold cross validation. The data was split into k-subsets (k=10). Each subset randomly selected 90% of patients as training data and 10% as testing data, repeated 3 times. A decision tree was generated for each model separating patients into CI or non-CI based on ANAM subtests, score types, and thresholds that best predict CI. We hypothesized ANAM would identify CI with good Area Under the Receiver Operating Characteristics (ROC) curves (AUC = 0.71-0.80) and a sensitivity ≥ 80%. The performance of the best CART model was compared to ANAM composite index using ROC analysis and kappa statistic. Composite Index: 3.88-0.05*Code Substitution Delayed_PCT – 8.4*Spatial Processing_CV + 2.44*Code Substitution Learning_MR + 9.87*Tapping Left_MR.

Results: 145 patients were defined as CI and 90 as non-CI based on ACR-NB. The best model included 6 ANAM subtests and PCT, MR, CV score types (Figure 1), with 89% sensitivity, 69% specificity, AUC of 81% (95% confidence interval: 75-86%), 81% accuracy, 82% positive predictive value, and 79% negative predictive value compared to ACR-NB. The most important subtests, score types, and thresholds were displayed in a decision tree (Figure 1). Concordance between the best CART model and ANAM composite index was 74%, κ = 0.48 (95% confidence interval: 0.37-0.59) (Figure 2).

Conclusion: Our findings extend the evidence on the construct validity of the ANAM as a screening tool for CI in SLE patients. Good concordance was found between the ANAM composite index and best CART model. Six most important subtests and scores for detecting CI from the ANAM were identified. A decision tree was generated to simplify the interpretation of ANAM and enhance clinical utility.

CI: cognitive impairment; PCT: percentage correct responses; MR: mean reaction time; CV: coefficient of variation of mean reaction time; TP: throughput For most of the ANAM tests, there are 4 score types provided. PCT represents accuracy. MR represents the average reaction time. TP is a measure of cognitive efficiency and is derived as the number of correct responses per minute. CV is an index of the patient’s consistency of response speed within a given timed subtest. It is a derived score (standard deviation of MR divided by MR). Better cognitive performance is indicated by higher PCT and TP scores, and lower MR and CV scores. Code substitution delayed measures delayed memory and requires the participant to compare a symbol-digit pair with previously defined and memorized symbol-digit pairs. The participant presses designated buttons to indicate whether the pair was correct or incorrect based on their memory. Tapping left hand and tapping right hand measure motor skill and reaction time. The participant is required to press the spacebar with the designated hand as many times as possible in a single trial. Simple reaction time repeated measures reaction time and requires the participant to respond as quickly as possible to a target stimulus. The “repeated” session is administered later in the battery and is also designed to measure fatigue. Logical relations measures abstract reasoning and verbal fluency whereby the participant evaluates the truth of statements and describes the order of two symbols displayed on the screen. Spatial processing measures visual spatial skills whereby the participant compares two histograms displayed side-by-side in different orientations.

CART: Classification and Regression Tree; AUC: Area under the curve; PCT: Percentage of correct responses; MR: Mean reaction time; CV: Coefficient of variation The ROC curve of the best model (PCT, MR, CV) from CART analysis reported an AUC of 81% compared to ACR-NB. The composite index formula (which was derived using logistic regression from our previous study) based on the model with PCT, MR, CV was applied to the current dataset and reported an AUC of 76% compared to ACR-NB.

« Back to ACR Convergence 2020

ACR Meeting Abstracts - https://acrabstracts.org/abstract/using-classification-and-regression-tree-analysis-to-assess-the-construct-validity-of-the-automated-neuropsychological-assessment-metrics-in-the-assessment-of-cognitive-impairment-in-sle-compared-to-t/