JavaScript is not enabled in your browser. JavaScript must be enabled to ensure full functionality in Q-Bank. Contact Q-Bank if you need further assisatance. Skip to main content

Question Evaluation

Question evaluation is the process of testing and analyzing how survey questions perform. With an understanding of how survey questions perform, survey managers and data users have full confidence in the validity of the data as well as an opportunity to revise the questions to better fit their study. Some of the strengths of question evaluation are listed below.

Question evaluation can determine if a question...

Captures the intended concept

Ideally, questions should capture constructs that they were created to capture. However, it is important to know if questions are not capturing the intended constructs. Understanding the constructs that the question is capturing allows survey managers to modify the question, and allows data users to understand the potential range of constructs that may have been captured in the data. Both of these measures increase the validity of the data.

Is interpreted consistently across respondent groups

Respondents' interpretations of a question may differ depending on their social experiences and demographic factors. For example, respondents with different education levels might interpret or comprehend question terminology differently.

Poses problems for respondents when attempting to answer

There are many possible reasons why respondents might have difficulty answering a question. Question evaluation can help to shed light on this process.

Why conduct question evaluation?

Question evaluation studies conducted prior to fielding the survey can identify and fix problems to reduce error. It can also ensure comparability across demographic and cultural groups, and between translations. After a question has been fielded, question evaluation studies can provide documentation for data users. Question evaluation reports can explain what construct was captured by a question, and support the data user in their analyses of survey data.

In addition, Question Evaluation is required for all new surveys or questions. The Office of Management and Budget stipulates in Section 1.4 of The Standards and Guidelines for Statistical Surveys:

"Agencies must ensure that all components of a survey function as intended when implemented in the full-scale survey and that measurement error is controlled by conducting a pretest of the survey components or by having successfully fielded the survey components on a previous occasion."
OMB Standards and Guidelines for Statistical Surveys (PDF)

Additional Information

View Q-Bank reports to see examples documenting the methodology and findings from question evaluation studies. Visit the QUEST website to see presentations related to Question Evaluation from survey methodologists and researchers across the world. Visit the Q-Bank Bibliography to find additional resources on aspects of question evaluation.