Abstract

Methods

Statistical Analysis

The primary outcome measure was the weighted kappa coefficient, reflecting agreement over the entire range of the mRS above chance among the rater pairs. Following standard convention kappa scores of 0.0–0.2 would be considered poor, 0.21–0.40 fair, 0.41–0.6 moderate, 0.61–0.8 good, and 0.81–1.0 excellent. In addition, to permit comparison with the range of reliability measures reported in prior studies of the mRS, we also calculated the unweighted kappa over the entire range of the mRS considering all ratings, the kappa for dichotomizations of the mRS, and the crude rate of agreement of raters (percent agreement), unadjusted for chance concurrence. These reliability scores were calculated for the paired ratings obtained among all 50 consecutively enrolled patients and among all survivors from this group at day 90.

Results

Among the 50 stroke patients, average age was 71.5 (range 43–93) 48% were female, and final diagnosis was ischemic stroke in 66%, hemorrhagic stroke in 26%, and transient ischemic attack in 8%. Neurologic deficit at the time of enrollment in the trial in the prehospital setting was a median Los Angeles Motor Scale score of 4 (range 1–5), while the first NIH Stroke Scale obtained after hospital arrival (and after exposure to prehospital study drug) was median 10.5 (range 0–40).

At day 90, 43 patients were alive and 7 had died. Across all 50 patients, the median NIHSS was 3, IQR 0–10, range 0–42. In the 43 alive patients, the Mini Mental Status Examination score was median 29, IQR 23–30.

Frequencies of mRS scores among all 100 ratings were: mRS 0 – 17, mRS 1–17, mRS 2 – 19, mRS 3 – 5, mRS 4 – 7, mRS 5–21, mRS 6 – 14. The mRS median was 2.0 and mean 2.8. The cross-tabulation of pair ratings in shown in Table 1. Raters scores concurred fully in 47 of the 50 patients, and in the remaining 3 patients scores differed by 1 level. Consequently, among all enrolled patients, for assigning patients among all possible mRS scores, the percent agreement was 94%, the weighted kappa was 0.99 (95% CI 0.99–1.00), and the unweighted kappa was 0.93 (95% CI 0.85–1.00). Among the 43 surviving patients, the percent agreement was 93%, the weighted kappa was 0.99 (0.98–1.0), and the unweighted kappa was 0.91 (0.82–1.00).

The kappa scores for the 6 possible dichotomizations of the mRS are shown in Table 2, and ranged from 0.93 – 1.00.

Discussion

In this investigation, raters using the Rankin Focused Assessment achieved excellent inter-rater reliability in assigning final outcome modified Rankin Scale disability ratings to patients three months after an index stroke. The inter-rater reliability in assigning mRS grades achieved with use of the RFA was substantially better than in most prior studies. In a recent meta-analysis of 10 prior studies of the inter-rater reliability of the mRS, the combined achieved unweighted kappa was moderate at 0.46,¹⁰ compared with the unweighted kappa of 0.93 (95% CI 0.85–1.00) observed in this study.

When he initially presented the scale over 50 years ago, Rankin provided only brief, broad descriptions for the categories of the mRS without clear operational criteria distinguishing one level from the next.¹¹ Consequently, the original scale leaves substantial leeway open to raters to develop idiosyncratic criteria or to apply the scale in an impressionistic manner.⁷ Because of the only moderate reliability of unstructured methods for assigning mRS grades, more formalized approaches have been previously developed by other groups. However, these have achieved only inconsistent or modest improvements in reliability and have additional potential drawbacks. The RFA was designed to incorporate elements of, and lessons learned from, these prior algorithms.

The Structured Interview (SI) for the mRS,^{7, 9} developed by Wilson and colleagues, was a pioneering instrument that first introduced a systematic approach to assigning mRS levels. However, the SI is somewhat complex to implement, which has limited its deployment in actual clinical trials. In addition, the SI improves the reliability of the mRS only moderately (kappa in meta-analysis of 0.62).¹⁰ The training DVD digital system developed by Quinn and colleagues is modeled on the successful NIH Stoke Scale training and certification system and has been widely adopted in clinical trials.⁸ However, this system has also been found to only moderately improve mRS reliability. ¹⁰

The RFA differs from these and other prior instruments in several distinct ways. Like the SI, the RFA has raters elicit information regarding specific functional items in five sections: 1) constant care, 2) basic activities of daily living, 3) instrumental activities of daily living, 4) limitations in participation in usual social roles, and 5) presence of common stroke symptoms. However, the RFA encourages the rater to gather information on patient functional performance from all available sources, including patient self-report, caregiver observations, physical therapist notes, physician and nursing records, and the rater’s own examination and interaction with the patient. In contrast, the SI is written in a manner that encourages elicitation of information from a single informant, a potentially problematic approach as individuals often have incomplete or biased perceptions of performance. Patients with anosognosia may underestimate and patients with the catastrophic reaction may overestimate their deficits. Individual family members may only see patients in limited settings and not have a fully-rounded picture of performance.

The RFA rates the patient based on current actual capacity and performance. In contrast, the SI asks the rater to factor out prestroke disability when assign a rating, forcing the rater to speculate on what the patient’s capacity and performance would have been if they had no other complicating conditions. Rating only specifically stroke-related dysfunction has advantages and disadvantages. A theoretical advantage is that scores reflect a treatment’s effect on the target condition uncontaminated by pre-existing deficits. However, in practice, identifying what a patient’s functional status would have been had they not had any pre-existing conditions requires conjecture by raters likely to decrease inter-rater reliability. Also, it makes the assessment instrument more complex, requiring documentation of which deficits are due to prior disability and which due to stroke-only disability. Moreover, this approach differs from that taken for other standard outcome measures in stroke clinical trials. The Barthel Index, the NIH Stroke Scale, and mortality status are all scored based on all cause sources, not just those speculated to be due to stroke alone. Additionally, restricting disability rating to stroke-specific items is problematic in clinical trials since interventions can alter functional outcome through nonstroke mechanisms, e.g. an adverse effect producing disabling congestive heart failure.

Like the SI, the RFA provides a detailed algorithm for scoring that concretely operationalizes criteria for distinguishing one mRS level from another. This approach is appropriate for instruments designed to enhance diagnostic judgments. The digital training vignette system, in contrast, does not as clearly provide operationalized criteria for assigning rankings. The experiential training and certification process works well for promoting inter-rater reliability in performing tasks that are intrinsically highly operationalized, such as physical exam techniques (like the NIHSS), but may be less useful for tasks requiring rendering of complex diagnostic judgments that have not been clarified by formal performance algorithms. Also, in developing the RFA, several items from the SI that raters found ambiguous or difficult were reworded or eliminated. For example, the SI item asking if patients need assistance to look after household expenses was reframed in terms of patient capacity rather than actual recent activity, because many fully capable elders do not perform this activity even when completely healthy, relying upon their spouse.

The RFA is logistically much simpler to employ in multicenter clinical trials and in local quality improvement projects than performance of mRS ratings by a central core laboratory, either by direct teleconference interviews or by central rating of videotapes of locally performed interviews. Also remote adjudication panel rating approaches have not yet been prospectively validated.

An additional strength of the present study is that it was conducted using researchers working in an actual clinical trial interviewing real stroke survivors. The study accordingly has ecologic validity for this common application of the mRS. Also, the coordinator raters in the study had a wide range of prior trial experience, from limited to extensive. Accordingly, the findings are likely to be generalizable to a wide range of raters.

We analyzed RFA performance using several statistical indices, weighted kappa, unweighted kappa, and percent agreement. Each provides useful insight. The percent agreement measure perhaps most accords with lay and clinician understanding of inter-rater agreement, but does not take into account chance concurrences. The unweighted kappa reflects performance above chance, but penalizes near misses to the same degree as wide disagreements. This metricis perhaps most appropriate for dichotomized applications of the mRS. The weighted kappa penalizes near misses to a lesser degree than wide disagreements and is most relevant when the mRS is analyzed over several levels.^12–14

We included patients with fatal outcome in our main analysis. Since contemporary clinical trials routinely employ the 7 level mRS version that includes a fatal outcome level, it is important to include these patients to obtain inter-rater reliability estimates that accurately indicate how the RFA will perform in clinical trials. Some prior mRS reliability studies have not included these patients, which will tend to lower the estimate of inter-rater reliability, since agreement on the hard endpoint of death is straightforward. To permit comparison with these studies, we also reported data on RFA performance in stroke survivors only, and it showed excellent reliability even when confined to this group.

This study has limitations. The patient sample was moderate in size. Physician-investigators did not participate as raters, non-nurse coordinators participated only to a limited degree, and all assessors were from a single trial group and a single country. Further reliability testing of the RFA in more diverse assessor groups would be beneficial. The study did not compare in the same patients mRS scores obtained with the RFA and scores obtained with any of the current common methods of scoring. Although there is no single widely accepted scoring method that can serve as a pre-existing gold standard, such comparisons with past practices would be of interest.

The Rankin Focused Assessment is brief and practical for use in multicenter clinical trials and routine practice. In this study, the RFA yielded high inter-rater reliability in grading the final global disability of patients participating in a randomized acute stroke treatment clinical trial. Further testing and validation of the RFA in larger studies with a more diverse group of assessors is desirable.

Supplementary Material

Acknowledgments

Footnotes

References