Background & History

Attachment 1: Background and History of Cognitive Interviewing

In 1983-1984, the Committee on National Statistics conducted a seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The participants in the CASM I seminar were cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological proposals that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the NCHS National Health Interview Survey (NHIS) as a test case.

Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to im­prove health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology, and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for Feder­al statistical agencies to conduct qualitative research on the cogni­tive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques - now commonly referred to as cognitive interviewing - proved effective for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that failed to measure what was intended, and that cognitive interviews were effective for identifying these kinds of measurement errors. At that point, a Questionnaire Design Research Laboratory (QDRL) was created at NCHS to provide such testing for surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology (OMB No. 0920-0222).

Since the inception of the QDRL, several other Federal agencies, including the Census Bureau and Bureau of Labor Statistics (BLS), have established cognitive laboratories or otherwise developed capacity for the conduct of cognitive interviews. These interviewing activities are currently conducted under Generic OMB Clearance at NCHS (OMB No. 0920-0222), at the Census Bureau (OMB No. 0607-0725), and at the Bureau of Labor Statistics (OMB No. 1220-0141) as cognitive interviewing techniques have been performed almost continuously for the evaluation of numerous survey questionnaires. In fact, one of the major conclusions of a second CASM seminar (CASM II, held in 1997) was that cognitive testing of survey questionnaires has become a standard practice in the Federal government, as well as in private and academic survey research organizations.

Generally, testing staff are not the original authors of the survey questionnaires and do not make decisions about the overall content and survey objectives. Rather, they are methodological specialists – either Agency staff members or contracted specialists -- who submit questionnaires to intensive evaluations designed to improve these measures. This work has proven to be effective for enhancing the quality of Federal survey data for over twenty years.

In some cases NCI has pre-existing relationships with agencies that already conduct pretesting under such a Generic clearance. In particular, testing of the periodic Cancer Supplement to the National Health Interview Survey has been conducted at the NCHS QDRL (OMB No. 0920-0222); and pretesting of the NCI-sponsored Tobacco Use Supplement to the Current Population Survey (TUS-CPS) has been conducted by the Census Bureau (OMB No. 0607-0725). Increasingly, however, NCI has developed one-time and periodic surveys – such as the Health Information National Trends Survey (HINTS: OMB No. 0925-0538) -- that do not involve collaboration by other Federal agencies, or that would benefit from NCI-sponsored pretesting. For these surveys, it is advantageous to the government if development were to follow a pretesting sequence equivalent to that used at NCHS, the Census Bureau, or BLS.

The term Intensive Pretesting will be used as a general term to refer to cognitive interviews, focus groups, and usability testing; these are distinguished from observational, field-based Pilot Household Interviews. The use of the Pilot Household Interview, subsequent to cognitive testing, was introduced by NCHS researchers in the 1990s (NCHS Cognitive Methods Staff Working Papers #3; #15; Hyattsville, MD), and have been supported first under OMB No. 0920-0222, and then under the first cycle of NCI Generic Pretesting Clearance OMB No. 0925-0589.

2