NHANES survey design affects variance estimates
As stated in the module on sampling in NHANES, the NHANES has a complex, multistage, probability cluster design. Typically, individuals within a cluster (i.e., county, school, city, census block) are more similar to one another than those in other clusters and this homogeneity of individuals within a given cluster is measured by the intracluster correlation. When designing a survey with a complex sample, you ideally want to decrease the amount of correlation between sample persons within clusters. To achieve this, you want to sample fewer people within each cluster but sample more clusters. However, NHANES can only sample 30 Primary Sampling Units (PSUs) within a 2-year survey cycle because of operational limitations such as the cost of moving the survey MECs and geographic distances between PSUs. The sample size in each PSU is roughly equal and it is intended to yield about 5,000 examined persons per year.
For a complex sample survey such as NHANES, variance estimates computed using standard statistical software packages that assume simple random sampling are generally too low (i.e., significance levels are overstated) and biased because they do not account for the differential weighting and the correlation among sample persons within a cluster. There is a loss of precision and a reduction in the effective sample size because individuals are chosen within clusters instead of being sampled randomly throughout the population.
WARNING
Standard statistical software packages that assume simple random sampling calculate variance estimates that are generally too low and biased because they do not account for differential weighting and the correlation among sample persons within a cluster.
The impact of the complex sample design upon variance estimates is measured by the design effect (DEFF). It is defined as the ratio of the variance of a statistic which accounts for the complex sample design to the variance of the same statistic based on a hypothetical simple random sample of the same size.
If the DEFF is 1, the variance for the estimate under the cluster sampling is the same as the variance under simple random sampling. The DEFFs for NHANES are typically greater than 1.
When the DEFF is greater than 1, the effective sample size is less than the number of sample persons but greater than the number of clusters. The effective sample size is calculated by dividing the sample size in a subgroup by the DEFF. The design effect is an attribute of a statistic calculated on a particular variable, rather than for the overall NHANES survey cycle. DEFF can be very different for different variables due to differences in variation by geography, by household intra class correlation, and by demographic heterogeneity. Design effects for a variable can also be different for different demographic subgroups (i.e. race and Hispanic origin or age groups.)
IMPORTANT NOTE
Statistical software that accounts for the sampling design effect must be used to calculate an asymptotically unbiased estimate of the variance and should be used for all statistical tests and the construction of confidence limits. These procedures require information on the first stage of the sample design (identification of the PSU and stratum) for each sample person.
Reference
Park, I and Lee, H (2004) "Design Effects for the weighted mean and total estimators under complex survey sampling." Survey methodology 30:183-193.