Brief Overview of Sample Design, Nonresponse Bias Assessment, and Analytic Guidelines for NHANES August 2021-August 2023

First published: September 20, 2024

Summary

This brief overview accompanies the NHANES August 2021 – August 2023 public use file release to inform data users of changes for this cycle. A full report on the sample design, nonresponse bias assessment, survey weighting method and analytic guidelines is forthcoming. The Plan and Operations of the National Health and Nutrition Examination Survey, August 2021-August 2023 covers changes in field operations for this cycle.

The NHANES August 2021 – August 2023 files are the first NHANES data release that includes data collected since the COVID-19 pandemic began. The pandemic impacted NHANES in important ways, including prematurely ending the 2019-2020 collection cycle in March 2020 (the 2019-March 2020 data were combined with 2017-2018 data and released as NHANES 2017 – March 2020 Pre-pandemic data). NHANES data collection resumed in August 2021 using a new sample design to reduce contact between interviewers and survey participants during the continuing COVID-19 pandemic. The declining trend in NHANES response rates, which continued for the August 2021 – August 2023 cycle, reflects trends seen across many federal surveys.^1,2 Given low response rates and sample design changes described below, further investigation of weighting methods for August 2021-August 2023 data was undertaken to help evaluate the quality of estimates. The sample design changes, nonresponse bias analysis, weighting methods and quality evaluations, new phlebotomy weights, and analytic guidelines are briefly presented below.

Sample Design and Data Collection Changes

A four-year sample was initially designed for 2019-2022, but data collection stopped in March 2020 due to the COVID-19 pandemic. When SARS-CoV-2 infections had subsided enough to return to data collection, the decision was made not to collect data from the remainder of the 2019-2022 sample, but to draw a new sample that would limit the contact between NHANES staff and participants. NHANES returned to data collection in August 2021 with a new two-year sample design similar to past designs that drew a multi-year, stratified, clustered four-stage sample of the U.S. civilian noninstitutionalized population. As with past two-year data release cycles, this cycle included 30 primary sampling units (PSUs) made up of counties from which area segments and then dwelling units were sampled. Participants were selected from within households in dwelling units. Sample selection within dwelling units for the August 2021-August 2023 cycle differed from previous cycles in two ways:

• No oversampling by race and Hispanic origin, and income: To minimize in-person interactions during the COVID-19 pandemic, NHANES reduced the number of households contacted for eligibility screening by eliminating oversampling by race and Hispanic origin and income from the sample design before beginning data collection. In prior cycles, to accomplish oversampling, dwelling units were assigned to specific demographic characteristics prior to interviewers knocking on the door. This resulted in “walk-aways” when there were no residents in the dwelling unit who fell into the pre-assigned demographic category. For example, a dwelling unit assigned to sample a Black woman would become a walk-away if no Black women lived there. In August 2021 – August 2023, dwelling units were no longer preassigned and at least one resident in every sampled dwelling unit was selected as eligible to participate if someone age 18 or older lived there and the household was not completely occupied by active-duty members of the military. This means that fewer households needed to be visited for screening. As a result of no oversampling by race and Hispanic origin and income, the number of participants for some demographic subgroups is noticeably lower or higher than in previous cycles although the total number of participants is roughly the same. Users should expect reduced statistical precision (e.g., larger standard errors and wider confidence intervals)) for groups with fewer participants. Point estimates may be affected as well (see Analytic Guidelines below).
• Person-level oversampling based on age group was changed. NHANES has always oversampled by age, but this was the first cycle where all the people encountered in a household who were ages 0–19 years and 60 years and older were selected. The number of selected adults ages 20–59 years depended on the number in the household. One was selected at random in households with one to three adults in this age group and two were selected at random in households with four or more adults. The potential increase in clustering by household was investigated and the mean number of participants from each household was similar to previous cycles.
• Multi-mode screening. The largest change in contact and recruitment methods for the August 2021 – August 2023 data collection was the multimode (self- and interviewer-administered) household screening approach. This was the first time NHANES used a self-administered screener, which was implemented to reduce in-person contact with households and potential for spreading SARS-CoV-2. Households were first contacted by mail with an invitation to participate using a web-based questionnaire. Across four mailings, households were provided with a link to the online screener, a paper version of the screener, and a phone number they could call to screen with a NHANES interviewer. Households that did not complete their screener in one of those modes were attempted in-person.³
• Reduced in person contact. NHANES also made other survey design changes to reduce contact between interviewers and participants that did not impact sampling or weighting approaches. For example, the first dietary recall interview mode was changed from in-person during the Mobile Examination Center (MEC) examination to a telephone interview following the examination. Analyses to assess possible impact of the mode change are forthcoming.

Response Rates and Nonresponse Bias Assessment

The unweighted interview and examination response rates for the August 2021 – August 2023 cycle were 34.6% and 25.7% respectively.ⁱ These are markedly lower than those for 2017 – March 2020 which were 51.0% and 46.9% respectively. Although low response rates do not necessarily indicate high nonresponse bias,⁴ they are a common survey quality metric because they reflect increased difficulty of the overall survey effort and increased risk for bias. To assess if bias was associated with higher nonresponse⁵, a thorough nonresponse bias investigation was conducted that included:

• Assessing differences in response rates and representativeness of the target population among respondent subgroups,
• Comparing characteristics of respondents with those from external data sources and benchmarking health estimates with other national survey estimates, and
• Evaluating alternative postsurvey weighting adjustments.

A similar investigation for the 2017 – 2018 cycle found both nonresponse bias and sampling variation (i.e., the sample of counties visited by the survey was an outlier on some characteristics)⁶. Additional weighting adjustment methods were then adopted for that cycle and used for subsequent cycles, including August 2021 – August 2023. The August 2021 — August 2023 investigation found no evidence that the randomly sampled counties did not adequately represent the U.S. civilian, noninstitutionalized population. Despite the continued decline in response rate and alteration of sampling methods at the household level to accommodate safety requirements during the COVID-19 pandemic, no major sources of bias were found that were not adequately addressed with weighting methods.

Weighting Methods and Quality Evaluations

As with past cycles, NHANES survey weights adjust for the probability of selection, nonresponse bias, and coverage bias at each stage of data collection: screener, health interview, and examination participation at the MEC. Screener nonresponse adjustment involved grouping dwelling units that were similar based on a) interviewer observations made prior to any in-person contact (e.g., building type) and b) area demographic information (e.g., median age of the Census tract in which the dwelling unit is located). Interview nonresponse adjustment included the same variables plus data collected in the screener (e.g., age of the screener respondent). Examination nonresponse adjustment then added data collected in the interview (e.g., whether the respondent had used any prescription medication in the past 30 days). The final raking calibration variables included race/ethnicity, sex, age, education, Census tract-level income, and urbanicity of the sampled county. Past estimation reports contain detailed summaries of the NHANES weighting process and can be found on the NHANES Survey Methods and Analytic Guidelines webpage.

Survey weight evaluations included analyzing their properties. To ensure that the weights would not add excessive variance to estimates, NCHS evaluated their distributional characteristics and impact on effective sample size (i.e., compared to the nominal sample size, the effective sample size is usually lower after accounting for the complex design with survey weights). Evaluations also included comparing selected weighted demographic and health estimates from NHANES August 2021 – August 2023 to benchmark surveys such as the American Community Survey and the National Health Interview Survey. Finally, plausibility of trends using past NHANES and August 2021 – August 2023 estimates were investigated.

To assess if additional weighting methods could further address nonresponse and coverage error, four alternative sets of weights were created and evaluated. To create the alternative additional four sets of weights, newly identified variables from the nonresponse bias assessment, such as interviewer observations and health conditions reported during screening, were either included in nonresponse adjustments or in calibration steps. The weight evaluations described above were repeated for all four sets of weights. None of the alternative weight versions improved representativeness based on comparisons to demographics and health insurance coverage estimates from the American Community Survey. However, some of the alternative versions had increased variation of weight values, which lowered precision of some estimates and decreased the effective sample size. Therefore, the August 2021 – August 2023 data have been released with the initial set of survey weights, which had the lowest impact on variance (i.e., they produced estimates with higher precision and narrower confidence intervals).

New Phlebotomy Weights

Analysis of nonresponse patterns for the phlebotomy component in the MEC examination revealed differences by age group and race/ethnicity, among other characteristics. Most notably, young children had a lower response rate to the blood draw. Therefore, an additional phlebotomy weight has been included in this data release to address possible nonresponse bias. This weight should be used for analyses that use variables derived from blood analytes and is included in all relevant data files.

Analytic Guidelines

The following analytic guidelines summarize how the changes in sample design and data collection for the NHANES August 2021 – August 2023 cycle may impact data analysis.

• Certain estimates may have greater variability and lower precision.
  The total sample size for the August 2021-August 2023 is similar to those for previous 2-year cycles. However, because there was no oversampling by race and Hispanic origin and income, sample sizes for demographic subgroups differ. In particular, the sample size for non-Hispanic White persons is higher and non-Hispanic Black persons is lower compared to previous cycles that included oversampling. Additionally, sample sizes are higher for persons ages 60 years and older and lower for younger age groups. Estimates for subgroups may have lower precision and as a result, statistical power to detect differences between groups may be lower. In addition, estimates for age and race/Hispanic origin subgroups, particularly children and adolescent age groups, may show unexpected patterns. As with any estimates based on survey samples, analysts are urged to consider plausibility, context, and trends when assessing findings.
• Cross-sectional analyses (i.e., concatenating August 2021 – August 2023 data with earlier cycles to produce estimates covering the combined time period)
  Analysts should use caution when combining August 2021-August 2023 with earlier cycles. The August 2021 – August 2023 cycle represents a two-year data collection period. However, there is a 1.5-year gap between the end of data collection for the 2017 – March 2020 Pre-pandemic cycle and start of the August 2021 – August 2023 cycle. Cross sectional analyses combining August 2021 – August 2023 with earlier cycles assume that the unobserved data between the two cycles (i.e., April 2020 – July 2021) do not differ significantly from observed data. That assumption may not be reasonable for some health behaviors and outcomes given disruptions that occurred in healthcare delivery, employment, and education during the COVID-19 pandemic before availability of vaccines. During August 2021 – August 2023, pandemic-era restrictions were gradually relaxed and immunity to COVID became more widespread.⁷ Analysts should consult other data sources that cover the early pandemic period prior to the August 2021-August 2023 data collection to test this assumption whenever possible.
• Trend analyses (i.e., using the August 2021 – August 2023 cycle as a separate time point trended with previous cycles)
  As with previous cycles (except 2017 – March 2020), the August 2021 – August 2023 represents a 2-year period. The midpoint is August 2022 (or 2022.7). This midpoint can be used with midpoints of previous cycles in trend analysis. For example, the interval midpoints in an analysis with 2013 – 2014, 2015 – 2016, 2017 – March 2020 and August 2021 – August 2023 cycles would be 2014, 2016, 2018.6 and 2022.7.
  
  As with cross-sectional analysis, analysts are encouraged to consider whether data for the time period not covered by data collection (April 2020 to July 2021) is likely to differ from the observed trend. For example, short-term effects that occurred and disappeared during the 16-month pause in data collection would not be measured. Analysts should consider other data sources or empirical research to inform interpretation of trend analyses.
• Directly comparing August 2021 – August 2023 estimates to earlier NHANES estimates (i.e., without estimating a trend)
  Direct statistical comparison may be conducted using standard analytic techniques and guidance for analyses using sample weights and design variables to account for the unequal probability of selection, nonresponse, and the clustered design.
• With the addition of phlebotomy weights, analysts will need to consider which survey weights are most appropriate to use. The NHANES Tutorial reviews how to select the appropriate survey weights for NHANES analyses using the principle that the weights for the smallest subpopulation should be used. For example, if an analysis included both interview and examination variables, the examination weights should be used because they account for nonresponse to the examination among interviewed people. For this cycle, when a blood analyte variable is included, it is best to use the phlebotomy weights which account for nonresponse to the blood draw in the MEC. However, the pattern of nonresponse for phlebotomy may differ from survey components that follow the examination, notably, the dietary interviews. For analyses including variables from both the phlebotomy component and the dietary interview(s), it may be preferable to use the dietary weights to account for the day of the week the interview was conducted. Because the dietary weights don’t account for missing blood analyte values, analysts should still assess how missing responses for blood analytes values and other variables in their analysis may introduce bias.
• The NHANES Tutorial is a resource for analysts using data from NHANES cycles from 1999-2000 through August 2021-August 2023. A wide range of topics is covered by tutorial modules that provide general guidance and sample analytic software code to calculate nationally representative health estimates.

References

1. Douglas Williams, Jeffrey Gonzalez. The Dueling Roles of Public Trust and Survey Participation. Presented at: May 17, 2024; Atlanta, GA. Accessed June 13, 2024. https://aapor.confex.com/aapor/2024/meetingapp.cgi/Paper/2566
2. Williams D, Brick JM. Trends in U.S. Face-To-Face Household Survey Nonresponse and Level of Effort. J Surv Stat Methodol. 2018;6(2):186-211. doi:10.1093/jssam/smx019
3. Terry A, Chiappa M, McAllister J, Woodwell D, Graber J. Plan and Operations of the National Health and Nutrition Examination Survey, August 2021–August 2023. National Center for Health Statistics (U.S.); 2024. doi:10.15620/cdc/151927. https://www.cdc.gov/nchs/data/series/sr_01/sr01-066.pdf
4. Groves RM, Peytcheva E. The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. Public Opin Q. 2008;72(2):167-189. doi:10.1093/poq/nfn011
5. Office of Management and Budget (OMB). Standards and Guidelines for Statistical Surveys. https://nces.ed.gov/fcsm/pdf/OMB_Standards_Guidelines_Statistical_Surveys.pdf
6. Fakhouri THI, Martin CB, Chen TC, et al. An Investigation of Nonresponse Bias and Survey Location Variability in the 2017-2018 National Health and Nutrition Examination Survey. Vital Health Stat 2. 2020;(185):1-36.https://www.cdc.gov/nchs/data/series/sr_02/sr02-190.pdf
7. U.S. Centers for Disease Control and Prevention. CDC Museum COVID-19 Timeline. Centers for Disease Control and Prevention. March 15, 2023. Accessed June 18, 2024. https://www.cdc.gov/museum/timeline/covid19.html

ⁱ The response rates reported here are unconditional response rates that incorporate dwelling unit response to the screener (screener response rate 65.6%), and response to the interview or MEC. The steep reduction in screener response rate from the 2017 – March 2020 cycle (88.7%) is due partly to the COVID 19 pandemic, partly to the nation-wide decline in response rates (see Williams, 2024; Williams and Brick, 2018), and partly to the new requirement that at least one person was sampled at all occupied dwelling units. Detailed participation counts and response rates for the August 2021 – August 2023 and past cycles are available online.