Background/Purpose: IMACS and PRINTO have developed preliminary core set activity measures and definitions of improvement (DOIs) for adult or juvenile dermatomyositis (DM) and polymyositis (PM). Our aim was to evaluate the inter-rater reliability and consensus among myositis experts rating patient profiles on clinical improvement, given core set measures and their change.

Methods: Fifteen adult and 15 juvenile myositis profiles using IMACS core-set measures (baseline and follow up) were evaluated by 30 adult and 29 pediatric myositis experts. PRINTO profiles on the same 15 juvenile DM patients were also evaluated. by 29 pediatric experts. Experts rated improvement on each profile as a) Likert scale: no improvement or worsening, minimal, moderate and major improvement, b) on the degree of improvement (0-10 scale). Percent agreement, kappa, Cronbach’s α, analysis of variance, and intra-class correlation (ICC) were used to determine overall inter-rater reliability, as well as that of different groups of myositis experts (adult (N=30) vs. pediatric specialists (N=29), more vs. less experienced, rheumatologist (N=50) vs. non-rheumatologist (N=9), North American (N=30) vs. other location (N=29)).

Results: Consensus (≥70%) on minimal clinically significant improvement was reached on all profiles. Eleven of 15 adult DM/PM profiles, 12/15 IMACS and 12/15 PRINTO juvenile DM profiles were rated as at least minimally improved. For all profiles, different group of raters reached the same consensus on at least minimal improvement. There was substantial agreement among all individual raters (kappa 0.64) and very high agreement between different group of raters for minimal and major clinical improvement (Table 1). The agreement between PRINTO and IMACS profiles on the same patients was poor (Table1), primarily because there was a difference in the consensus as to whether the patient improved or not in 4 juvenile profiles. Results were not explained by geographical differences between North American and European evaluators or by a better familiarity of PRINTO members with PRINTO measures. ICC was high for all raters (0.82) as well as different group of raters (Table1). Cronbach’s α for all profiles was very high (≥0.98) for improvement on the Likert scale and in the degree of improvement (0-10 scale). There was no difference in degrees of improvement rated by different group of myositis experts. However, IMACS profiles had a different degree of improvement as compared to the PRINTO profiles from the same pediatric patients.

Conclusion: Inter-rater reliability among myositis experts was excellent in assessing minimal and major improvement in patient profiles. The agreement between ratings in IMACS and PRINTO profiles was poor. These results will aide in the development of new DOIs for minimal and major clinical response for myositis.

Table 1: Inter-rater reliability among various groups myositis experts on clinical improvement

Groups	Minimal Improvement		Major Improvement		Improvement on Likert scale	Degree of improvement
	% agreement	Kappa	% agreement	Kappa	ICC	ICC
Adult (n=30) vs. Pediatric (n=29)	93.3%	0.83	93.3%	0.83	0.82	0.81
North American(n=30) vs. Other location (n=29)	100%	1.0	93.3%	0.84	0.82	0.81
More vs. less experienced	93.3%	0.83	93.3%	0.84	0.82	0.82
Rheumatologist (n=50) vs. Non-rheumatologist (n=9)	90%	0.74	96.6%	0.93	0.82	0.81
IMACS (n=15) vs. PRINTO (n=15) pediatric profiles	73.3%	0.16	60%	0.25	0.80	0.79

---

« Back to 2013 ACR/ARHP Annual Meeting

ACR Meeting Abstracts - https://acrabstracts.org/abstract/good-inter-rater-reliability-of-myositis-experts-in-assessing-clinical-improvement/