Evaluating Survey Questions: An Inventory of Methods

Introduction

This document was developed by the Federal Committee on Statistical Methodology (FCSM) Subcommittee on Question Evaluation Methodology to provide an inventory of methods used by federal statistical agencies to evaluate survey questions. Since question development methods play a crucial role in assuring the quality of survey questions and the data they produce, this document also includes a section on developing survey questions which outlines important steps in creating the survey questions that will subsequently be tested and evaluated. The question development section is followed by a discussion of question evaluation, a description of data collection methods, and a list of the methods federal agencies use to evaluate survey questions, along with a definition, common uses, strengths, and limitations for each evaluation method.

Question Development and Evaluation Methods (QDEM)

The many techniques employed to assure that survey questions evoke the intended response and to evaluate question quality come under the broad umbrella label of “question development and evaluation methods” (QDEM). This includes assuring: (1) that the survey questions ask what the researcher intended, (2) that they are consistently understood as intended, (3) that respondents are willing and able to consistently answer the questions as intended, and (4) that providing the full response is not overly burdensome (Groves et al., 2009). This process involves both development and evaluation of the questions (Figure 1). While conceptually distinct, the development and evaluation processes become intertwined in practice.

Question development involves identifying and defining the concepts to be measured, and drafting the questions to measure these concepts. After questions have been developed, they should be assessed using one or more of the evaluation methods defined in this inventory. The process should be iterative, where questions evaluation (testing) informs revisions (development) to the questions, followed by evaluation of the revised version(s), with further revisions if necessary. Through this process, revisions constitute a later stage of question development.

Graphical representation of the relationship between survey question development and evaluation methods. The focus of this document is depicted by the blue circle (survey question evaluation methods), which includes methods that are used both in question development and evaluation. The overlap depicted is intended to be illustrative and not to convey the actual size of the overlap.