Background/Purpose: The ACR practice guidelines establish U.S. and international treatment recommendations. We sought to characterize the quality of the guidelines using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument with the aim of identifying potential areas for improvement.

Methods: Using the AGREE II online tool, four independent reviewers appraised the ACR practice guidelines available at the ACR website and their immediate previous version. AGREE II consists of 23 questions that examine six quality domains: Scope and Purpose, Stakeholder Involvement, Rigor of the Methodology, Clarity of Presentation, Applicability and Editorial Independence. To avoid sequencing effects, the order of appraisal was randomly assigned across reviewers. The quality scores of each domain were calculated by summing up the individual item scores in a domain and then dividing that sum by the maximum possible score for that domain. We summarized the domain scores for each guideline and examined trends over time.

Results: The 9 guidelines (Glucocorticoid Induced Osteoporosis (GIOP); JIA; Gout; LN; OA; SpA; PMR; RA) listed at the ACR website as of April 2016 and 4 previous versions (SLE 1999, OA 2000, GIOP 2000 and RA 2008) were appraised using the AGREE II instrument. All guidelines were published between 1999 and 2015. The Table shows the percentages of the total possible score for each guideline. The minimum and maximum for each domain were 53-99 for Scope and Purpose; 47-99 for Stakeholder Involvement; 31-96 for Rigor of the Methodology; 50-99 for Clarity of Presentation; 26-78 for Applicability; 25-85 for Editorial Independence and 42-96 overall. The Figure displays the trend in score changes over time. The average quality of guidelines improved over each time period, especially in the Scope and Purpose, Rigor of the Methodology, Stakeholder Involvement and Clarity of Presentation. In contrast Applicability and Editorial Independence domains improved but not to the levels of the other domains. For the 4 guidelines with previous versions, the mean (SD) absolute improvements for each domain were 18 (±11) for Scope and Purpose; 13 (±8) for Stakeholder Involvement; 38 (±22) for Rigor of the Methodology ; 25 (±15) for Clarity of Presentation; 22 (±12) for Applicability; 24 (±17) for Editorial Independence and 31 (±5) overall.

Conclusion: Based on the AGREE II instrument, the quality of ACR guidelines has improved over the past 16 years. There remains potential for improvement in the applicability and editorial independence domains.