Handling missing data in RCTs; a review of the top medical journals

Content assessment

All articles were assessed by one reviewer (MF) using a standardized form, and 15% of the studies were randomly selected to be independently assessed by a second and third reviewer. We calculated kappa statistics to evaluate inter-rater reliability for methods used to handle missing data in primary analysis, whether an intention-to-treat analysis was performed, and sensitivity analysis. All disagreements were resolved by consensus.

Comparison of reviews on missing data

We compared our findings with previous reviews of missing data and ITT in RCTs. For each review we reported, if possible, the number of trials: included in the review, with missing data, reporting sensitivity analysis, reporting ITT, and missing data approaches. We aimed to compare our results directly to Wood et al, [16] to assess whether changes have occurred since their 2004 paper on RCTs published in 2001. We therefore used similar definitions, inclusion criteria and collected data in a similar fashion. We used chi-square tests for comparisons. Other reviews were compared qualitatively.

Extent and handling of missing data

Seventy-three (95%) trials reported some missing outcome data. The median percentage of patients with a missing outcome was 9%, with a range of 0 – 70%. Sixty-six trials reported reasons why outcomes were missing with reasons ranging from simply stating that patients were lost to follow-up to very specific explanations. The majority of trials reported these details in their CONSORT flow diagram [19].

Complete case analysis was the most common method to handle missing data (33, 45%). Twenty (27%) trials used simple imputation methods. Three (5%) used linear interpolation, eight (11%) used worst-case imputation, nine (12%) used last observation carried forward. Six (8%) trials performed multiple imputation. Fifteen (19%) trials used model based methods: 11 (15%) used mixed models and three (4%) used un-weighted generalized estimating equations (Table 2). Thus, a MAR assumption for the primary analysis was made in 17 (23%) of the trials with missing data.

Of the trials reporting more than 10% missing data, 13 (36%) used complete case, 13 (36%) used simple imputation, 3 (8%) used multiple imputation, and 7 (19%) used model based methods. Of the trials reporting less than 10% missing data, 20 (54%) used complete case, 7 (19%) used simple imputation, 3 (8%) used multiple imputation, and 7 (19%) used model based methods.

Sixty-six (86%) trials presented a sample size calculation, with 38 of them accounting for missing data in the calculation by inflating the sample size by one minus the expected attrition rate. The mean absolute difference in the actual attrition rate and the expected was 8% with a range of 0.3–31%. Two trials accounting for missing data in the sample size calculation had unclear expected attrition rates. 72% estimated higher attrition rates than observed, while 28% estimated lower attrition rates than observed.

Attempts to avoid missing data before and during the trial were mentioned in 26 trials. The median percentage of missing data for those who mentioned attempts to avoid missing data was 12%, with a range of 2–56%. The median percentage of missing data for those who did not mention attempts to avoid missing data was 9%, with a range of 0.6 – 70%.

Seventy-one (97%) trials reported the number of patients with missing outcome by treatment arm. Nine trials reported comparisons of baseline characteristics between patients with observed and missing outcomes. Six of them reported a significant difference.

Sensitivity analysis for missing data

Twenty-seven trials (35%) reported performing a sensitivity analysis with respect to missing data (Table 3). Of these, ten (37%) trials used multiple imputation in the sensitivity analysis. Six (22%) performed a complete case analysis. Four (15%) trials carried out simple imputation: two performed worst-case imputation and one imputed with baseline value. One trial performed both complete case analysis and LOCF. One trial performed worst-case imputation, LOCF, and multiple imputation. Two trials carried out adjustments using auxiliary data. One trial used un-weighted GEE and two trials used mixed models. Two trials reported that they performed a sensitivity analysis, but methods were unclear.

In total, 11 made a MCAR assumption for the sensitivity analysis, and 15 made a MAR assumption. Only 10 weakened the missingness assumption of the primary analysis to perform their sensitivity analysis, by using a MCAR assumption for the primary followed by a MAR assumption in the sensitivity. No researchers used MNAR models. Of the 36 trials reporting more than 10% missing data, 16 (44%) performed a sensitivity analysis: 4 used complete case, 2 used simple imputation, 1 used simple and multiple imputation, 6 used multiple imputation, 2 made adjustments using auxiliary data, and 1 approach was unclear. Of the 41 trials reporting less than 10% missing data, 11 (27%) performed a sensitivity analysis: 2 used complete case, 1 used complete case and simple imputation, 1 used simple imputation, 3 used multiple imputation, 3 used model based methods, and 1 was unclear.

Intention-to-treat analysis

Fifty-two reported the use of ITT and 14 reported the use of modified ITT for their primary analysis. Definitions of ITT and modified ITT differed across trials. Of the articles that reported ITT or modified ITT analysis, 21 (40%) included all randomized subjects in the primary analysis. Of the articles that reported ITT or modified ITT analysis, 62 (94%) had missing data. Of the trials with missing data, 48 (66%) reported ITT analysis and 14 (19%) reported modified ITT analysis.

Comparison of reviews on missing data

Of the five reviews that we considered, including our own, missing data rates were fairly similar, ranging from 61–95%. Use of complete case analysis and simple imputation were consistent across reviews, ranging from 45–65% and 17–27%, respectively. Recently, there has been an increase in application of multiple imputation and model based methods for missing data in primary analyses. The number of papers reporting sensitivity analysis for missing data (1–37%) and ITT analysis (48–85%) have both increased across time. See Table 4.

In comparison to Wood et al.’s [22] review of RCTs from 2001, trials reporting missing data were similar: 89% versus 95% (95% confidence interval (CI) for difference -3% to 15%; P = 0.18). The percentage of papers with 10% or more missing data was also similar, at 51% versus 47% (95% CI for difference -12% to 20%; P = 0.63). Of those that reported missing data, trials that used complete case analysis decreased from 65% to 45% (95% CI for difference -36% to -3%; P = 0.02). The percent that performed simple imputation was similar, at 22% as compared to 27% (95% CI for difference -9% to 19%; P = 0.49). The use of multiple imputation or model based methods increased from 10% to 27% (95% CI for difference 5% to 30%; P = 0.01). Mentions of attempts to avoid missing data was similar, at 35% compared to 29% (95% CI for difference -46% to -58%; P = 0.83). Sensitivity analysis for missing data increased from 21% to 37% (95% CI for difference 1% to 31%; P = 0.04). Reports of ITT or modified ITT analysis increased from 41% to 85% (95% CI for difference 29% to 59%; P <0.0001).

Summary

Our review of 77 RCTs published in the top medical journals found that 95% of trials reported some missing outcome data, with a median of 9%, and up to 70%. Complete case analysis was the most common way of handling missing data in the primary analysis (45%), followed by simple imputation (27%), model based methods (mixed models and un-weighted generalized estimating equations) (19%) and multiple imputation (8%). Sensitivity analyses were performed in 35% of the trials, but most (63%) did not weaken the assumptions regarding missing data from their primary analysis. An ITT or modified ITT was reported in 85% of the trials. Most reports included a sample size calculation (86%), and 58% of these inflated the sample size to account for expected attrition. These calculations tended to be conservative, with 72% estimating higher dropout than observed with a difference of 8%, and ranging up to 30% higher.

Relation to other literature

The amount of missing data appears to have remained fairly constant over time, as does the proportion of trials that mentioned attempts to avoid missing data. While it is possible that those trials that did not report prevention attempts did, in fact, employ them, it may be that researchers need to give more consideration to missing data during trial design and conduct.

The use of methods with the strong assumption that data are missing completely at random (complete case analysis, simple imputation and un-weighted GEE) has remained popular: 85% in the current review as compared to 89% in the 2001 review. This is in direct contrast to recommendations put forth by leaders in the field, including the National Research Council’s Committee on National Statistics (CNSTAT) Panel on Handling Missing Data in Clinical Trials which recommends a primary analysis that assumes data are missing at random, followed by sensitivity analyses which weaken this assumption and allows for data not missing at random [7, 18]. White et al. [4] suggest a four part strategy: 1) attempt to follow up all subjects; 2) carry out a primary analysis of all observed data that are valid under a plausible assumption; 3) perform sensitivity analyses to explore the effect of departures from the primary assumption; and 4) account for all randomized participants in at least one of the analyses. This approach was utilized in the analysis of an alcohol screening and brief intervention study [24]. It was discouraging that sensitivity analyses which contradict the assumptions of the primary analyses remain so rare.

The CNSTAT report favored inverse probability weighted generalized estimating equation (GEE) methods and multiple imputation, in part because auxiliary data associated with missingness can be incorporated into the analysis [18]. However, none of the RCTs in our review used weighted GEEs, and only 8 used MI for the primary analysis. While the number of trials reporting sensitivity analyses appear to have increased over time, from 1% in 1997, 21% in 2001, and up to 37% in our review, none of the studies reported using MNAR models, or appeared to follow the Panel’s guidelines for sensitivity analyses. Perhaps the reluctance to use more sophisticated approaches is due to a lack of knowledge or experience on the parts of applied researchers and/or biostatisticians. Perhaps it is due to the time lag between reports of methods and software to implement them.

Reports of ITT or modified ITT analysis have increased substantially over time, from 48% in 1997, 41% in 2001, 71% in 2007, to 85% in 2013. However, we found, as others have [1, 2], that these terms are used inconsistently: only 40% actually included all randomized participants in the primary analysis. White et al. [20] call for at least one analysis (primary or sensitivity) to include all participants. This lack of consistent definition (even amongst methodologists [25]) and clarity regarding who was included in the analysis has led the CONSORT statement’s authors to remove the ITT request in their 2010 update (over the original 2001 statement) [19].

Many trials had repeated measurements, (79%) but only 14 of these used all the measurements in the primary analysis, often resulting in a strong MCAR assumption. Using all outcome data, even if the primary interest is in a specific time point, can reduce some or all of the bias due to data which are missing non-randomly. For example, Bell and Fairclough use several methods to analyze quality of life measured at four time points in an RCT with substantial missing data. A t-test comparing the two arms at the 4^th timepoint found a treatment effect of -0.4; a contrast from a mixed model estimated the effect to be -8.0 [5]. Simulation studies have demonstrated the bias that can occur when a MCAR assumption is made for data which are MAR [17, 26]. Approaches which use all repeated measures data and are valid for MAR data include multiple imputation, mixed models, inverse probability weighted GEEs, and Bayesian analysis [5–7].

Strengths and limitations

A strength of our review is the inclusion of other reviews to assess possible time trends. In particular, we followed Wood and colleagues [22] methods and definitions in order to make direct comparisons between 2001 and 2013. A limitation is the difficulty in making comparisons with the other reviews, due to different inclusion criteria. For example, inclusion of survival outcomes may reduce missing data rates, as participants who drop out are often considered censored. Different definitions may also hinder comparisons. For example, the rate of missing data when measurements are taken repeatedly could be: 1) the number of patients with any missing primary outcome data divided by the total number of patients randomized; or 2) the number of missing assessments divided by the total number of assessments. We chose the former, in line with Wood et al. [22] Another limitation is that we focused on the top 4 medical journals. It is likely that trial reports appearing in these journals have higher standards of conduct and reporting, so that this review may underestimate the extent of missing data and overestimate the use of sensitivity analyses.

Recommendations and conclusion

We have several recommendations. First, missing data should be considered at each stage of a trial: design, conduct, analysis, and reporting. Prevention is the best way to handle missing data, so more effort needs to be put into missing data at the design and conduct stage. The CNSTAT report [7], and it’s synopsis [18] discuss several approaches. Second, we recommend that in trials with repeated measurements, all data should be used in an analysis that makes a plausible assumption about missing data. Usually this will be a MAR assumption. Third, sensitivity analyses that weaken the assumptions about missing data should be carried out and reported. For example, if the primary analysis uses a MAR assumption, the sensitivity analysis should assume MNAR.

There appears to be a large gap in translation between statistical methods research and the use of these methods in applications, such as RCTs. For example, simple imputation remains popular, despite warnings from many statisticians against their use, particularly LOCF [5–7, 13, 14, 27–29]. This failure to translate persists, despite papers regarding missing data, sensitivity analyses, and strategies for intention to treat in the presence of missing data being published in high impact medical journals [4, 16–18]. More statisticians should attempt to make their work accessible to applied researchers, by publishing secondary papers in appropriate applied journals showing how to make their methods work in practice. Applied statisticians and researchers should read these papers to update their skillsets and use modern methods that increase statistical power and in some cases reduce bias. Editors and reviewers should demand that modern methods which use all the data are used, at least in the sensitivity analysis.