Background/Purpose: Health professions education (HPE) is an evolving discipline that frequently utilizes surveys to collect data about abstract concepts. A study of the methodologic rigor of surveys published in leading HPE journals raised concerns about their quality (Artino, 2018). Surveys that violate best practices in design can result in measurement errors and problems with reliability and validity.

A growing community of HPE researchers is recognized within ACR, and HPE scholarship is increasingly published in rheumatology journals. The prevalence and quality of the surveys used have not been examined. This gap is important to education scholars who conduct research relevant to the future workforce, sponsors, and funding agencies that support this work, and reviewers who appraise the scientific merit of a report.

This study aimed to determine the numbers of HPE studies that used surveys published in leading rheumatology journals, assess their quality using a previously described rubric, and examine the frequency of response rate reporting.

Methods: Two authors (MO, MJB) searched articles published in leading rheumatology journals over a 5-year period from 2019 through 2023 for those that used at least one survey tool and included health professionals as respondents. Titles and abstracts were independently screened for eligibility, and inclusion decisions were made in consensus between MO and MJB. Quality was assessed using a coding rubric which evaluated the reliability and validity of the reported evidence and examined the methodologic rigor of specific survey items.

All authors independently coded four papers and then met to compare ratings and resolve differences; three authors (MO, AMB, MJB) divided the remaining publications so that each article was independently coded twice.

Two authors (MO, MJB) also reviewed ACR abstracts presented over the same period to identify those that covered an HPE topic and utilized surveys.

Results: Twenty articles met inclusion criteria. Eleven (55%) included a complete copy of the fully formatted survey for review. Six (30%) were adapted from previously published surveys.  Response rates were reported in 13 (65%) of studies, ranging from 5% to 100%. Reliability and validity evidence were reported for 11 (55%) and 1 (5%) of the surveys, respectively (Table 1). Nine of the 11 (81%) surveys with a fully formatted copy violated at least one best practice for design, with the most common violations being agreement-response and multi-barreled items (Table 2).

Among 141 presentations at the ACR annual meetings education poster and concurrent abstract sessions from 2019 to 2023, 84 (60%) included surveys (Table 3). The prevalence of survey use in these abstracts was stable, ranging from 55-63%.

Conclusion: Surveys are prominent in HPE scholarship presented at ACR education abstract sessions. The majority of HPE research manuscripts that include surveys published in leading rheumatology journals failed to report validity and reliability evidence, and many violated established best practices in survey design. Improving methodologic rigor in designing, using, and reporting surveys in rheumatology education research will strengthen the data we collect for the questions we seek to answer.

« Back to ACR Convergence 2024

ACR Meeting Abstracts - https://acrabstracts.org/abstract/examining-the-quality-of-surveys-used-in-rheumatology-health-professions-education-research/