Analysis of Cognitive Interviews: Overview

As is the case for analyses of qualitative data, the general process for analyzing cognitive interview data involves synthesis and reduction—beginning with a large amount of textual data and ending with conclusions that are meaningful and serve the ultimate purpose of the study. For example, Miles and Huberman (1994) describe qualitative analysis as an interactive process of “data reduction (extracting its essence), data display (organizing its meaning) and drawing conclusions (explaining the findings)” (cited in Suter, 2012). For analysis of cognitive interviews, reduction and synthesis can be conceptualized within five incremental steps—conducting interviews, producing summaries, comparing across respondents, comparing across subgroups of respondents, and reaching conclusions. With each incremental step, a data reduction product is created. A description of each of these steps and the resulting reduction product is presented below:

1. Conducting interviews to produce interview text: collecting narratives from respondents that reveal how each respondent made sense of and went about answering a survey question,
2. Synthesizing interview text into summaries to produce detailed summaries: detailing how and why each respondent interpreted the question as well as how they formulated their answers, including events or experiences considered as well as any difficulties answering the question,
3. Comparing summaries across respondents to produce thematic schema: identifying and mapping common themes that detail phenomena captured and the process of formulating a response,
4. Comparing identified themes across subgroups to produce an advanced schema: identifying ways in which different types of respondents may process questions differently depending on their differing experiences and socio-cultural backgrounds,
5. Making conclusions to produce final study conclusions: determining and explaining the performance of a question as it functions within the context of respondents’ various experiences and socio-cultural locations.

Although these steps are described separately and in a linear fashion, in practice they are iterative; varying levels of analysis typically occur throughout the qualitative research process.

As each step is completed, data are reduced such that meaningful content is systematically extracted to produce a summary that details a question’s performance. In detailing a question’s performance, it is possible to understand the ways in which a question is interpreted by various groups of respondents, the processes that respondents utilize to formulate a response as well as any difficulties that respondents might experience when attempting to answer the question. It is the ultimate goal of a cognitive interviewing study to produce this conceptual understanding, and it is through data reduction that this type of understanding is possible. Figure A illustrates each step in the analytic process and the synthesized product for each step. The downward pyramid represents the reduction of data as the analyst moves from the raw data of individual interviews to thematic schema generated by the comparisons of interviews.

The different levels of analysis described above represent not only data reduction, but also movement toward larger conceptual themes; analysis is the simultaneous process of data reduction and knowledge building. Figure A along with Figure B illustrates the dual nature of this process. The pyramid in Figure A points downward because it represents the reduction of data; the pyramid in Figure B points upward because, as the analyst progresses through each step, explanation regarding a question’s performance grows.

While the two processes of data reduction and knowledge production may be heuristically separated, in reality the processes occur simultaneously. In reducing the cognitive interview data the analyst produces a more comprehensive understanding of a question’s performance; as analysis is performed, understanding of the question response process becomes more complex and complete. In the beginning it is only possible to understand how each individual respondent makes sense of and answers the survey question. By the end, individual interpretations are understood as well as how those interpretations relate across groups and within the overall context of the question’s performance.

It is important to recognize that, with each step of analysis, a series of decisions must be made by the analyst. For example, in writing summaries, the analyst must decide what information is important enough to be included and which information they feel can be omitted from the summary. In developing a thematic schema, the analyst determines whether an interpretive pattern has emerged across respondents. These decisions are significant as they directly bear upon the ultimate findings of the cognitive interviewing study. For this reason, it is important to consider the ways in which such decision making can be made transparent so that results can be replicated or, at least, theoretically understood. An audit trail, consisting of the analytic products generated from each level of analysis, reveals the types of decisions made by the analyst in order to conduct the study.