Background/Purpose: Non-random sampling of baseline variables in randomized controlled trials (RCTs) may be due unintentional errors, stratified randomization strategies, or data fabrication. Prior studies have suggested that non-random sampling may occur in up to 15% of RCTs in anesthesiology and have led to high profile retractions of fabricated data. No similar studies have been conducted in the field of rheumatology.

Methods: A PRISMA compliant systematic review was conducted to identify rheumatology RCTs published between 2010 and 2022 and met the following criteria: >20 patients per arm, first publication of trial data, ≥5 extractable variables, 2-5 trial arms, and studied a rheumatic disease. Baseline demographic and clinical variables from randomized arms were extracted. Monte Carlo simulations (50,000 per variable) were performed to calculate p values for individual baseline variables, which were then used to calculate an overall trial p-value using the Fischer-Stouffer method. The distribution of participant summary p-values, which should be uniform under conditions of true random sampling, was graphed. RCTs with trial p-values ≤ 0.01 or ≥ 0.99 underwent further scrutiny using a modified version of the Research Integrity Assessment (RIA) Checklist. Integrity concerns for individual metrics (No, Some, Significant Concerns) and overall (Low, Moderate, High Concern) were assessed.

Results: We identified 11,435 RCTs, 2,313 of which underwent full text review. Of these, 957 met inclusion criteria, from which 19,799 baseline variables were extracted. A uniform distribution of trial p-values would have produced approximately 19 studies with p values < 0.01 or >0.99; our search identified 73 such studies (Figure 1). Using the RIA checklist, 64 (87.7%) studies were found to be lacking across at least one metric of study integrity. The most common items among trials with high integrity concerns included trials not being prospectively registered (21, 28.8%) and reporting ethics approval number (20, 27.4%). None of the studies were retracted or had letters of concern written. Based on the RIA findings, 28 (38.4%) trials were rated as having a high level of data integrity concerns, 3 (4.1%) had moderate data integrity concerns, and 42 (57.5%) had low data integrity concerns.

Conclusion: Non-random sampling has occurred in nearly 1 out of every 100 rheumatology randomized controlled trials. The majority of studies were assessed as being of low risk of threats to their integrity, but 27 out of 957 studies (2.8%) had moderate to high concern of problematic conduct. Understanding common themes within these studies will assist in identifying areas that require further scrutiny in evidence synthesis and implementation.

« Back to ACR Convergence 2023

ACR Meeting Abstracts - https://acrabstracts.org/abstract/non-random-sampling-in-rheumatology-randomized-controlled-trials-evidence-of-concerning-trial-conduct/