National health interview study (NHIS)

The NHIS is conducted continuously throughout the year using face-to-face interviews in respondents’ homes. Basic health information is collected for all family members, by proxy if necessary. Additional health and socio-demographic information, including health risk behaviors, is collected, by self-report, from one adult family member [2].

National health and nutrition examination survey (NHANES)

The NHANES collects information on adults and children and combines face-to-face interviews with physical examination measures. The NHANES has been conducted periodically since the early 1960s. In 1999 the NHANES became a continuous survey with data released every two years [7].

National survey on drug use and health (NSDUH)

The NSDUH is annually compiled from face-to-face interviews. It focuses primarily on substance abuse among respondents 12 years of age and older [1].

Current population survey (CPS)

The CPS is conducted by the Bureau of Labor Statistics and the Census Bureau [8]. Data are combined from telephone survey and other modes of collection. Data are published for respondents over 15 years of age.

Access to health care and general health measures

Several scholars investigated whether self-reported claims of health care coverage can be substantiated (Table2). Mucci [20] conducted a study asking BRFSS respondents who indicated that they had health insurance to retrieve their insurance cards and validate information provided during the course of the interview. Findings for respondents enrolled in health maintenance organizations (HMOs) showed very high levels of agreement (93% sensitivity). Other respondents were also provided reliable self-reports on health coverage (76% specificity). Respondents were more accurate when reporting that they were enrolled (93%) than reporting correctly on the type of plans (76%) in which they were enrolled. The authors also found higher levels of reliability of health care plan self-reports when respondents had been enrolled for longer periods of time. Clements [21] collected information regarding members in HMO plans from BRFSS respondents as well as collecting respondents’ HMO plan names. Respondents were asked whether they belonged to an HMO. This information was compared with lists of commercial HMOs, by using HMO plan names as reported by respondents. Self-reported membership in HMOs taken from the BRFSS was found to be to be a reliable measure (k = 0.87). These authors also conducted test/retest studies on this question with 78% of respondents reporting the same health care plan information during follow-up interviews.

Fahimi [13] compared confidence intervals for prevalence estimates of no health insurance coverage from BRFSS (18.4 to 19.1) and NHIS (18.7 to 20.0). There were no significant differences in the estimates produced by the two surveys, both of which rely on self-report, albeit using different modes of data collection. Differences were found between BRFSS and NHIS on whether respondents had not sought medical care due to costs (confidence intervals of 14.9 to 15.4 and 7.4 to 8.2, respectively). The BRFSS asked respondents to categorize their general health as “excellent,” “very good,” “good,” “fair” or “poor.” Salomon [22] found that BRFSS and NHIS showed significant differences in the confidence intervals for prevalence of reported “fair” or “poor” self-assessments of health. Time trends of self-reported overall health were also compared in four national surveys: the BRFSS, NHANES, CPS and NHIS. BRFSS respondents reported poorer overall health than other surveys. These scholars suggest that these differences may be attributable to under coverage of cell phone populations, which tend to be younger. However all surveys reviewed by these authors, including the BRFSS, consistently showed that the proportion of respondents indicating that their general health is “excellent” was declining.

Immunization and prevention/ screening measures

Several measures of immunization, preventive screening and testing are collected by the BRFSS (Table3). Shenson [23] conducted reliability and validity testing of self-reported pneumococcal vaccination data from the BRFSS using test/retest methods. Seventy three percent of respondents provided identical information on vaccination when retested, two years after the initial data collection time period. Validity measures were conducted by comparing data provided by a small subset of BRFSS respondents with Medicare claims or medical records. Self-reports had a sensitivity of 0.75 and specificity of 0.83, within substantial levels of kappa. Bradbury [24] conducted test/retest comparisons of responses to questions regarding colorectal cancer testing for BRFSS respondents in Massachusetts. Overall, their research suggested that reliability was moderate to substantial (k = 0.55 to 0.69). Questions regarding having had tests were more reliable than questions regarding timing of such tests.

Cronin [25] tested validity of self-reports of mammography screening by comparing rates from BRFSS data with rates calculated by the NHIS and compared those rates to mammography registry data. They found that BRFSS estimates were similar to those reported by the NHIS. Both methods of self-reporting (BRFSS and NHIS) produced lower prevalence rates than registry rates. Fahimi [13] compared rates of vaccination for flu and pneumonia from the BRFSS and the NHIS. BRFSS respondents were more likely to report having had pneumonia and/or annual influenza vaccines than were respondents on the NHIS. Overall, reliability and validity studies on immunization, preventive screening and testing published since 2004 showed consistency across national surveys. Test/retest reliability indicated that similar answers were provided for some measures, even when two years elapsed between administrations of the survey. Respondents were better able to accurately recall that they had a preventative test than they were able to recall the dates of testing or screening.

Physical activity measures

Questions on the BRFSS related to physical activity produced data that allow researchers to classify respondents into levels of recommended and vigorous physical activity, from inactive to vigorously active. Eight studies were identified from the literature which presented findings of reliability and/or validity of BRFSS physical activity measures (Table4). Yore [26] conducted research including test/retest of the physical activity questions over a nine month period. Participants also were asked to maintain a log of physical activity and wore accelerometers to assess validity of self-reported responses. Log responses were more highly correlated with telephone self-reports than were measures taken from the accelerometer as the standard (k = 0.4 to 0.52 and 0.17 to 0.22, respectively). Reliability of data was also higher for those respondents who were in the vigorous activity category or when assessing responses related to strengthening. Validity was assessed by comparing telephone interview responses to log entries and accelerometer readings. Validity, using the log as a benchmark, ranged from k = 0.40 to 0.52, while validity estimates with the accelerometer as a base were lower at k = 0.17 to 0.22. The authors concluded that the validity and reliability of the BRFSS can be used to classify persons into groups of levels of activity. A second research effort by Yore and other colleagues [19] indicated that BRFSS occupational physical activity measures were highly replicated, especially when time between repeated measures was short. Overall, these publications supported the findings of the BRFSS at the moderate level of the kappa statistic. Substantial (k > 0.6) agreement was found when assessing reliability among persons who were categorized at the vigorous level of physical activity.

Everson and McGinn [10] reported reliability findings for test/retest physical activity responses by race and gender. They found some variability among race/gender groups. Their overall Inter-Correlation Coefficient (ICC) placing respondents into groups of vigorously active, moderately active or inactive activity ranged from 0.32 to 0.85. They also examined reliability for occupational and leisure measures of the BRFSS with ICCs ranging from 0.36 to 0.82. Sedentary indicators ranged from 0.32 to 0.83. Overall their study found fair to substantial reliability for measures tested. Brown [27] provided similar findings for test-retest methods, reporting “fair” to “substantial” ICC agreement, when measured using Landis and Koch’s [16] categories for kappa interpretation. Brown’s research also included percentage of respondents who were assigned to the same groups of levels of activity, based on responses to repeated BRFSS measures. Overall 77.6% of respondents were assigned to the same groups/levels of physical activity across repeated administrations of the survey. Hutto [28] found that overall, vigorous activity and walking were consistently reported even when question order was changed. However there were differences noted, especially for moderate physical activity when walking questions were posed prior to other activity questions. The authors recommended posing walking questions after moderate and vigorous physical activity questions in order to avoid bias in self-reporting. Pettee [18] found that a question from the BRFSS on television viewing held up well when a test/retest reliability study was conducted. ICCs for a one week retest were 0.55 and were at 0.42 for a three week retest schedule.

The BRFSS may also be compared with other surveys, interviews and physical measures taken of the same or similar populations. Carlson [29] conducted a review of prevalence estimates and trends of measures of physical activity across three surveys: the BRFSS, the NHIS and NHANES. As was noted earlier, the NHIS was conducted in face-to-face format, and NHANES combined face-to-face interviews supplemented with physical measures. The surveys also differed in the number and detail of physical activities responses collected. Levels of reported physical activity were higher for the BRFSS, than for the other two surveys. For example, the percentage of persons estimated to be “active” was 30.2 for the NHIS, 33.5 for the NHANES and 48.3 for the BRFSS. The three surveys were in agreement when trends were assessed, with higher levels of activity being associated with younger and among white, non-Hispanic respondents. These differences may be caused by the fact that the BRFSS included more measures of physical activity than the other surveys.

Reis [30] tested a self-assessment of physical activity related to work, the occupational physical activity questionnaire (OPAQ), which they correlated with the single occupational question from the BRFSS. Research participants provided information through self-reports and physical measures. Information was also collected through accelerometers worn by study participants. Reis found substantial agreement (k = 0.71) between the aggregated measures of occupational physical activity on the OPAQ and the BRFSS.

Overall, the identified studies of reliability and validity for physical activity measures supported findings of the BRFSS. The reliability of indicators was supported using test/retest methods and time trend methods. Reliability measures for physical activity questions were found to be in the fair to substantial ranges of the statistic k. Findings indicate that the most reliable estimates were achieved for persons who exercise regularly. Validity was assessed by comparison with other surveys, although some of the comparison surveys used different data collection methods. Some research also compared BRFSS physical activity measures and responses to physical measures such as accelerometers. Variation of prevalence estimates was found in some instances, but trends were similar when comparing among survey results over time.

It is not surprising to find that differences in reporting physical activity change over time. Respondents who were contacted for test/retest studies may have, in fact, changed their levels of activity in the interim between testing. Therefore, higher levels of reliability of measures in shorter term retests are reasonable.

Chronic conditions and mental health measures

The BRFSS collected data on a number of chronic conditions, including diabetes, asthma, arthritis, and cardiovascular diseases. Fahimi [13] compared prevalence levels of diabetes and asthma among BRFSS, NHIS and NHANES. NHIS and BRFSS estimates were similar, with NHANES estimates showing significant differences (Table5). When asked whether they had been told that they have diabetes, respondents to the BRFSS and NHIS had similar prevalence estimates (confidence intervals of 7.9 to 8.1 and 7.8 to 8.5, respectively). NHANES estimates on this question ranged from 5.1 to 7.4. A similar question on asthma diagnosis resulted in more variance between BRFSS and NHIS (confidence intervals of BRFSS 13.1 to13.6; NHIS 9.5 to 10.3).

Bombard [31] conducted a validity and reliability study of BRFSS arthritis questions among seniors. Telephone responses were compared to written medical history and physical examination information for a select group of study participants. Agreement between the modes of self-reports was high (k = 0.68) and sensitivity and specificity of the questions as compared to the physical measures was 70.8 and 70.3%, respectively. Sacks [32] also conducted validity tests of arthritis questions using physical measures. Persons who had upcoming appointments were asked BRFSS questions prior to physical examinations. Self-assessments were found to be more accurate among older respondents. Sensitivity and specificity were at 77.4 and 58.8%, respectively for persons aged 45 to 64 and sensitivity and specificity at 83.6 and 70.6% for participants over 64 years of age. Cossman [33] used data on prescription drugs for treatment of cardiovascular disease as a proxy for prevalence. They then compared prevalence rates using these data on a substate/ county level to prevalence rates produced by the BRFSS. Correlation coefficients ranged from 0.43 to 0.66 (moderate to strong) for the area within twenty-four states where BRFSS modules on cardiovascular disease were administered.

The BRFSS included a number of quality of life and related mental health. Andresen [34] conducted test/retest responses questions among Missouri respondents (Table6). They found moderate to excellent reliability across quality of life measures, with only slight variation in categorical (when compared to continuous variables) measures and among older respondents. Self-reported overall health measures reliability was substantial (k = 0.75) as were measures of poor physical health days (k = 0.71), poor mental health days (k = 0.67), limited activity days (k = 0.57), healthy days (k = 0.75), frequent mental distress (k = 0.58) and frequent physical distress (k = 0.64). Variation was also greater when time between test/retest measures was longer. Kapp [35] conducted a similar study within the same state (Missouri). The authors compared item reliability for all respondents and for cancer survivors. Kappa statistics for all measures and groups tested were within the moderate to substantial range (k = 0.43 to 0.80) and found the measures to be appropriate quality of life indicators among cancer survivors.

Behavioral health risks/status

Three components of behavioral health and status (overweight and obesity, tobacco use and alcohol use) are examined in this section. A comprehensive study of multiple indicators from BRFSS, NHANES and NHIS was conducted by Fahimi [13]. These authors found that the BRFSS prevalence measures of obesity were statistically similar to those of NHIS (Table7). Observed differences between BRFSS and self-reports from the NHANES were small. As with the two previously cited studies, height was less likely to be biased than was weight.

Prevalence estimates reported by Ezzati [36] found that while bias in self-reported height and weight estimates were found, especially among women, these biases could be corrected through the use of weighting. These scholars used NHANES to determine benchmarks for regions and states then adjusted BRFSS data accordingly. They concluded that telephone survey respondents provided data that underreported Body Mass Index (BMI) but that BMI data were useful when appropriately weighted. Since telephone survey data are less expensive to collect, the authors found this method to be acceptable to ascertain national and sub-national prevalence estimates of obesity. Yun [37] found similar results, noting that self-reported biases were not consistent across demographic groups, and that appropriate weighting is necessary to correct for demographic factors such as gender and educational attainment. Their findings indicated that prevalence of obesity and overweight was underreported by between 9.5 and 5.7 percentage points. Underestimation was particularly noted among 20–39 year old females.

Although tobacco use is widely noted to be related to health status, there are relatively few comparative studies published since 2004 concerning reliability of tobacco use prevalence measures across national surveys. This may be due to the fact that question format differs on these studies, making them somewhat difficult to compare. The BRFSS, NHIS and NHANES all measured tobacco use in some way. Klein [38] included a fourth survey, the Harris poll online (HPOL) a non-random web-based sample survey of over 100,000 respondents, to review differences among national survey estimates of tobacco use prevalence (Table8). After weighting, they found that BRFSS (using national median of the state BRFSS surveys) and NHIS estimates were statistically similar, and that NHANES estimates were slightly higher. The HPOL results, taken from nonrandom samples, differed slightly from NHANES, BRFSS and NHIS findings. The authors concluded that self-reports varied by methodology and question format, but that measures from all surveys produced utile information for researchers.

Fahimi [13] compared national survey data from 2006 for NHANES, NHIS and BRFSS. They calculated confidence intervals for current smoking and found no statistical differences between confidence intervals of prevalence estimates from BRFSS (20.4 to 21.0) and NHIS (20.3 to 21.6), but higher levels of prevalence reported by NHANES (21.4 to 25.9). Fahimi’s research included a number of comparisons of BRFSS and NHIS prevalence estimates related to alcohol consumption. These questions were not asked of NHANES participants. NHIS and BRFSS surveys included questions on drinking which differed in format. The NHIS question provided information on self-reported consumption of 5 or more drinks in one day, while the BRFSS self-reports use total number of drinks in a single occasion to determine binge drinking. Although NHIS question did not measure binge drinking, estimates were statistically similar for binge drinking and average number of drinks, despite question wording differences. BRFSS respondents were classified into groups of persons who drink five or more drinks on “one occasion,” while NHIS respondents were grouped into categories which include five or more drinks “in one day”. Miller [39] compared state-level prevalence estimates from the BRFSS and NSDUH. The NSDUH differed in method from the BRFSS in that it was conducted as a face-to-face interview in respondents’ homes. These researchers combined data from 1999 and 2001 from the BRFSS. Eight states with large enough samples of NSDUH data were used to review state-level prevalence rates. Despite some variance in prevalence for individual states and some demographic categories, characteristics of binge drinkers between the two surveys were similar.