Supporting Statement for Request for Clearance:
QUESTIONNAIRE DESIGN RESEARCH LABORATORY
2007-2009
OMB No. 0920-0222
Contact Information:
Karen R. Whitaker, B.S.
Laboratory Manager, Questionnaire Design Research Laboratory
Office of Research and Methodology
National Center for Health Statistics/CDC
3311 Toledo Road, Room 3101
Hyattsville, MD 20782
301-458-4569
301-458-4031 (fax)
kwhitaker@cdc.gov
October 4, 2006
SUMMARY
An OMB clearance extension with revision is requested for “NCHS Questionnaire Design Research Laboratory (OMB No. 0920-0222).” This project encompasses general questionnaire development and pre-testing activities to be carried out in 2007-2009 in the Office of Research and Methodology, National Center for Health Statistics (NCHS), Centers for Disease Control and Prevention (CDC). The activities are to be conducted by the staff of the Questionnaire Design Research Laboratory (QDRL) and involve the development of health-related survey questionnaires, using a methodology which has been employed effectively since 1985. This request is almost identical to the previously-approved clearance. At the request of OMB, slight modifications have been made to A. 9. - the description of payments. Slight modifications to A.10. reflect changes in the informed consent documents due to technology changes in the method of recording some interviews.
Five types of activities are carried out:
1) Survey questionnaire development and testing based on cognitive interviewing methodology. Draft questionnaires to be used in CDC, other federal agencies, or other academic or professional institutions are developed and tested through rounds of cognitive interviews in laboratory or field settings with specifically-recruited volunteer participants. Most commonly these are one-on-one interviews, although group interviews (focus groups) are also used occasionally. Results of cognitive interviews are used to make questionnaire design decisions that minimize survey response error.
2) Research on the cognitive aspects of survey methodology. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing individual interviews or focus groups. In all cases, the purpose of the research is not necessarily to test particular questions, but to enhance our understanding of the psychology of response, to develop better standards for questionnaire design, or to improve data collection procedures.
3) Research on computer-user interface design for computer-assisted instruments (including Web-based surveys), also known as usability testing. Laboratory or field participants help us assess ease of use (e.g. Computer Assisted Personal Interviewing (CAPI), Computer Assisted Self-Interviewing (CASI) instruments), comprehension, and quality of on-line help.
4) Pilot Household interviews: A limited number of pilot interviews (either personal or telephone) are conducted with household participants, using professional field interviewers. Sources of response error are identified through observation by methodologists, and techniques such as the coding of the interviewer-participant interaction.
5) Studies of the optimal design and presentation of statistical graphical and textual material. This could include evaluations of the effectiveness of data presentation on the Web (including Web-based surveys), informational brochures, poster presentations, and statistical publications.
Clearance is requested for a period of 3 years.
OMB Approval of Individual Collections
CDC will continue to submit individual collections under this generic clearance to OMB. OMB will provide feedback on the individual collections within ten working days, whenever possible, as is currently the case.
SUPPORTING STATEMENT FOR OMB CLEARANCE
NCHS Questionnaire Development and Pre-testing Research
A. JUSTIFICATION
A.1 Circumstances Making the Collection of Information Necessary
In 1983/1984, the Committee on National Statistics conducted a two-part seminar on the Cognitive Aspects of Survey Methodology (CASM) under a grant from the National Science Foundation (NSF). The participants in the CASM seminar (CASM I) were survey researchers and cognitive psychologists from academic institutions and survey researchers from the National Center for Health Statistics (NCHS) and the Bureau of the Census. The seminar examined a number of cognitive-related methodological studies that might lead to improvements in the questionnaires and interviewing procedures employed in scientific surveys in general, and in the National Health Interview Survey (NHIS) as a test case.
Following this seminar, the NSF provided funding to NCHS to investigate how relevant knowledge and techniques in cognitive science could be applied to improve health surveys. The project, begun in 1984, was called Laboratory-Based Studies of the Cognitive Aspects of Survey Methodology (CASM), and used cognitive psychological methods to study the survey interviewing process. In its final report, NCHS concluded that it is feasible and efficient for a Federal statistical agency to conduct laboratory research on the cognitive aspects of survey questionnaires. Subsequently, NCHS applied the cognitive research techniques being tested under the grant to develop the 1987 NHIS supplement, a comprehensive set of questions on knowledge, attitudes, and practices regarding cancer risk factors. Cognitive research techniques (also known as cognitive interviewing) proved invaluable for identifying conceptual problems with draft questions. The NCHS project staff concluded from this experience that past questionnaire design procedures were often unable to identify questions that were failing to measure what was intended, but that interviews in the laboratory were effective for identifying these kinds of measurement errors.
The Questionnaire Design Research Laboratory (QDRL) was created at NCHS to provide such testing for NCHS surveys on a regular basis, as well as to continue more general research on the survey response process, questionnaire design, and pretesting methodology. Cognitive interviewing in the QDRL is now performed almost continuously for the evaluation of numerous survey questionnaires. In fact, one of the major conclusions of a second NCHS/NSF-sponsored CASM seminar (CASM II, held in 1997) was that cognitive testing of survey questionnaires has become a standard practice in the Federal government, as well as in private and academic survey research organizations. Generally the QDRL staff are not the original authors of the survey questionnaires and do not make decisions about the overall content and survey objectives; rather, they are methodological specialists who subject questionnaires to intensive evaluations designed to improve these measures. This work has proven to be effective for enhancing the quality of NCHS survey data for nearly 20 years.
The cognitive interviewing techniques used to date at NCHS have focused mainly on concurrent and retrospective protocol analysis. In concurrent protocol analysis, a volunteer participant is asked to think aloud as he/she answers the questions, and the interviewer probes the participant for additional information. The interviews are generally semi-structured; the interviewer uses draft survey questions as a guide, but probes as needed to determine the participant's interpretation of the questions and the recall processes used to arrive at his/her answers. This method uncovers ambiguities in question wording, participant strategies for dealing with vague questions, or questions that ask for information that is not readily available.
In retrospective protocol analysis, the interviews are more structured than in concurrent protocol analysis; the interviewer first administers the entire draft questionnaire, and then reviews the questions and responses with the participant, probing for reactions to the questions. While less information is gained about the recall techniques used by participants, there is also less deviation from the natural flow of an interview. Interviews are usually audio or video recorded, so that the interviewer can concentrate on probing the responses and can analyze the content later.
Occasionally, “focus groups” (or group interviews of 5-10 individuals) are used to discuss general concepts that survey questions will focus on. Individual interviews are generally preferable to focus groups for evaluating specific questions because respondents usually respond to surveys individually, and the group dynamic can have a strong influence on interpretations and responses. However, focus groups can sometimes help questionnaire designers to understand the circumstances of various groups of people, and this information can be used to craft questions that better match respondent experiences.
Additional issues arise in computer-assisted survey instruments. Issues include the human-interface design, ease of use, comprehension, privacy, quality of on-line help and efficiency of screen organization. Optimal designs may be dependent on culture and education. Some of our research is designed to identify problems arising from the design of computer-based questionnaires.
Cognitive interviewing methodology identifies problems that are missed by traditional field pretests. Field interviewers may not be sufficiently trained to identify questionnaire problems, and such tests are often conducted too late to allow for substantial revisions to be made. Nevertheless, field pretests are a vital complement to cognitive interviews because they allow the questions to be administered in their actual environment under actual interviewing conditions. For that reason, we sometimes conduct small-scale pilot household interviewing at various points in questionnaire development. Also, as time and resources allow, we sometimes record the behaviors of both interviewers and survey participants in such interviews to allow for systematic analysis. These activities were used successfully to develop the questionnaires used in previous NHIS Supplements. The QDRL therefore plans to apply these techniques in development of the NHIS revised Periodic and Topical Modules (formerly referred to as Supplements) and of modules from other surveys.
Generally, pilot interviews for face-to-face surveys are conducted in the participant's household, and pilot interviews for telephone surveys are conducted over the telephone. Professional field interviewers (Census Bureau Field Representatives or other interviewers who are contracted for the tested survey) conduct these interviews. A subset of these interviews is observed by a survey professional (a QDRL or other NCHS staff member, or member of a Federal agency sponsoring the questionnaire). As the interviewer conducts the pilot household interview, the observer compiles notes regarding respondent misunderstandings or difficulty answering, or questions that interviewers have difficulty administering, which help to identify potential question revisions. Subject matter staff are debriefed on these findings and the results of the pilot testing will be used to modify the questionnaire for the later field pretest.
This practice allows testing of types of individuals who do not ordinarily volunteer for cognitive interviews in the laboratory, and who may be more typical of the usual survey participant; it also provides information collected under realistic field conditions, and collected early enough to be useful for questionnaire design decisions. This testing will be referred to in this document as Pilot Household Interviewing.
Full-scale field pretests are generally conducted after questionnaire development work (cognitive interviews and/or pilot household interviews) is completed. Such pretests are a vital component of preparations for a major survey collection and serve to evaluate questionnaire flow, length, logical progression, and so on. These are major undertakings beyond the usual scope of QDRL work (although QDRL staff may observe and participate in such pretests). It is important to note that cognitive interviews and pilot household interviews do not replace full-scale field pretests, but complement them with a greater focus on questionnaire design issues.
In addition to the applied questionnaire development activities described above, QDRL staff design and conduct research studies on the cognitive aspects of survey methodology more generally. Such research could take the form of experiments embedded within fielded surveys (generally referred to as “split-ballot” experiments), experiments conducted in the laboratory, or exploratory studies employing individual interviews or focus groups. The purpose of the research is to enhance our understanding of the psychology of response, to develop better standards for questionnaire design, or to improve data collection procedures. Ultimately these studies produce generalizable knowledge that improves the quality of data collection instruments more generally.
Data collection for this project is authorized under 42 USC 242k (Section 306 of the Public Health Service Act). A copy of the legislation is provided in Attachment A. CDC is requesting terms of clearance identical to previous submissions. CDC will submit individual collections under this generic three year clearance to OMB. OMB will continue to provide feedback on the individual collections within 10 working days of the submission. Remuneration of respondents will be evaluated on a case by case basis.
A.2. Purpose and Use of Information Collection
The purpose and use of collecting this information fall into five categories:
2.1 Development and testing of specific survey questionnaires
2.2 Research on the cognitive aspects of survey methodology
2.3 Research on human-computer interfaces/usability
2.4 Pilot household interviewing
2.5 Studies of the optimal design and presentation of statistical graphical and textual material.
A.2.1 Development and testing of specific survey questionnaires:
This data collection primarily uses cognitive interviewing methodology to identify and correct questionnaire flaws, e.g., questions which are vague or ambiguous, cannot be answered readily or accurately by the participant, or otherwise contribute to the non-sampling errors of the survey. Attachment B contains an outline of the contributions of the cognitive interviewing methodology to the questionnaire development process, the methods used at various stages of the process, and the strengths and limitations of this methodology. The methods used will vary depending on the stage of development of the various data collection instruments to be studied. When questions have been used successfully in earlier surveys, testing will evaluate whether the questions function appropriately in the new context. In cases where there is evidence that previously developed questions were not entirely reliable or valid, more extensive evaluation will be conducted. The most extensive questionnaire development activities will be applied to untested draft questions and undeveloped lists of data objectives.
a) National Health Interview Survey (NHIS) (OMB # 0920-0214): The NHIS collects annual data on health status and limitations, use of health care, AIDS testing, family resources, health insurance, access to care, injury, health behaviors and functioning. Personal interviews are conducted in approximately 43,000 households including about 106,000 persons. The QDRL conducted cognitive testing of various modules between 2004 and 2006, including health insurance, disability, immunization, cancer screening questions, complementary and alternative medicine and mental health. It is anticipated that the QDRL will conduct testing for numerous modules during 2007-2009.
In addition, other parts of the Department of Health and Human Services (DHHS), such as the Assistant Secretary for Planning and Evaluation (ASPE) occasionally request that NCHS include new policy relevant questions on the NHIS. Examples included citizenship status, health insurance coverage, and the receipt of government services by low-income individuals and families. The QDRL assisted in the development and pretesting of the survey questions. Assignments like this may occur during the 2007-2009 period and will be handled in a similar way. Further, in cases in which it may be difficult to identify and recruit the appropriate participants (for example, persons who are undocumented aliens), contractors who have expertise in the use of cognitive techniques with difficult-to-locate populations will be enlisted to conduct the research with oversight by QDRL staff.
b) State and Local Area Integrated Telephone Survey (SLAITS) (OMB # 0920-0406): In response to the need for the development of a national capacity to generate high quality State and local level data for tracking and monitoring current and emerging health and welfare policy-related issues, the State and Local Area Integrated Telephone Survey (SLAITS) was initiated. SLAITS was designed to provide quick turnaround data on a variety of broad health and welfare related issues and includes questions on health insurance coverage, access to care, perceived health status, utilization of services, and measurement of child well-being. SLAITS uses the same Random-Digit-Dial (RDD) telephone design approach and sampling frame used in the National Immunization Survey (NIS).
The QDRL conducted cognitive testing for a Children’s Health module in 2000, and Asthma modules in 2000 and 2001, and childhood immunization in 2006. The QDRL may be asked to conduct cognitive testing of various questionnaires and modules during 2007-2009.
c) Pregnancy Risk Assessment Monitoring System (PRAMS): PRAMS is a surveillance project of the Centers for Disease Control and Prevention (CDC) and state health departments. PRAMS collects state-specific, population-based data on maternal attitudes and experiences prior to, during, and immediately following pregnancy.
The QDRL conducted cognitive testing on PRAMS questionnaires in 1999, 2001, and 2003. We anticipate testing new questions, as well as questions that will be proposed as expansions and refinements to those already found in PRAMS questionnaires during 2007-2009.
d) National Health and Nutrition Examination Survey (NHANES) (OMB # 0920-0237):
NHANES collects annual data about the health and diet of people in the United States. The survey consists of two parts: an in-home interview and a health examination. The in-home interview asks questions about health status, disease history, and diet. The health examination consists of tests based on age and gender and is performed in a Mobile Examination Center.
The QDRL conducted cognitive testing of a brochure designed to be used by the field interviewers to convert survey refusals (2000), and cognitive testing of various modules for the “in-home interview,” including sexual orientation (2000), physical activity and pain (2001), and a follow-up telephone survey of male participants with positive prostate specific antigen (PSA) in 2002. Testing and evaluation of other modules may be conducted during 2007-2009.
e) National Immunization Survey (NIS) (Clearance exempt): The NIS is a list-assisted random-digit-dialing telephone survey followed by a mailed survey to children’s immunization providers that began data collection in April 1994 to monitor childhood immunization coverage.
The target population for the NIS is children between the ages of 19 and 35 months living in the United States at the time of the interview. Data from the NIS are used to produce timely estimates of vaccination coverage rates for all childhood vaccinations recommended by the Advisory Committee on Immunization Practices (ACIP). Estimates are produced for the nation and for each of 78 Immunization Action Plan (IAP) areas, consisting of the 50 states, the District of Columbia, and 27 large urban areas.
The QDRL may be asked to conduct cognitive testing of various modules during 2007-2009.
f) National Survey of Family Growth (NSFG) (OMB # 0920-0314): The National Survey of Family Growth (NSFG) is a multipurpose survey based on personal interviews with a national sample of men and women 15-44 years of age in the civilian non-institutionalized population of the United States. Its main purpose is to provide reliable national data on marriage, divorce, contraception, infertility, and the health of adults and infants in the United States. QDRL conducted cognitive testing on NSFG modules in 1994, 1998, 1999. The QDRL may be asked to conduct cognitive testing of future cycles of the survey in 2007-2009.
g) National Health Care Survey (NHCS) (various clearances): The National Health Care Survey (NHCS) embraces a family of health care provider surveys, obtaining information about the facilities that supply health care, the services rendered, and the characteristics of the patients served. Each survey is based on a multistage sampling design that includes health care facilities or providers and patient records. Data that are collected directly from the establishments and/or their records, rather than from the patients, identify health care events--such as hospitalizations, surgeries, and long-term stays--and offer the most accurate and detailed data on diagnosis and treatment, as well as on the characteristics of the institutions. These data are used by policymakers, planners, researchers, and others in the health community to monitor changes in the use of health care resources, to monitor specific diseases, and to examine the impact of new medical technologies, to mention a few. The QDRL may be asked to conduct cognitive testing as these questionnaires are updated and supplemented with additional questions during 2007-2009.
h) Other questionnaire testing and development: In addition to the specific questionnaire testing and development activities listed above, we anticipate that QDRL staff will be asked over the next three years to test questionnaires developed by NCHS, other components of CDC, other Federal agencies, and possibly academic and professional institutions that collect data relevant to public health. It is appropriate that the QDRL perform these activities, as it is currently the sole Federal facility performing cognitive interviewing in order to develop DHHS survey questionnaires, and is frequently the only one available for development of questionnaires originating outside of DHHS. However, because the requests may arrive with little advance notice, we cannot presently specify the nature of these questionnaires. Such a general plan was obtained in the previous clearance (No. 0920-0222), and the QDRL was thus able to conduct quick response testing of several questionnaires, including an American Community Adult Tobacco survey for the Office of Smoking and Health/CDC; multiple HIV Testing Behavior Questionnaires for the National Center for HIV, STD, and TB Prevention, HIV Incidence and Case Surveillance Branch/CDC; Adult Core Respiratory Disease Questionnaire for the National Institute of Occupational Safety and Health (NIOSH)/CDC; Cancer Screening Questions for the National Cancer Institute (NCI); Disaster Mental Health Questions for the National Institute of Mental Health (NIMH); Health Insurance Questions for the Current Population Survey (CPS); and the Abbreviated Blood Donor Screening Questionnaire redesigned by an interagency task force. No procedural problems were encountered as a result of such testing. In fact, the flexibility associated with cognitive interviewing allowed for survey pre-testing in a timely manner that minimized participant burden.
The interviews for questionnaire development activities (a) through (h) above will usually be conducted in QDRL facilities using cognitive interviewing procedures described in Attachment B. If we are unable to obtain adequate numbers of individuals from particular population subgroups (e.g., elderly, or those who have specific health problems), we will attempt to make arrangements with organizations such as centers for the elderly, or service organizations for persons with specific health conditions, to interview participants at outside locations.
Usually, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), or web-based. For a telephone interview, we will either make arrangements to call the participant at home, or to conduct the interview in our laboratory, but calling the participant from another laboratory room with face-to-face debriefing following.
It is possible that NCHS QDRL may collaborate with other agencies through interagency agreements which will include confidentiality provisions to test and develop survey questionnaires. Consent forms will be modified to reflect their participation.
A.2.2 Purpose and Use of Research on the Cognitive Aspects of Survey Methodology
The second major purpose of data collection is to conduct research on the cognitive aspects of survey methodology:
a) Research on the affects of alternative questionnaire designs: Many questionnaire design recommendations are based on cognitive testing; others are based upon past experience or general principles of questionnaire design. In any case, it is often advantageous to quantify how these design decisions affect data collection in the field. For example, we may develop theories that certain changes to the structure of a question will make it easier to understand and more efficient to administer. One way to explore this possibility is to conduct field experiments where an original and alternative version of a question are each administered to half of a sample. In addition to comparing response distributions of the two versions, interviews can be tape recorded and coded so that a variety of interviewer and respondent behaviors can be compared. Such experiments may focus on grammatical structure of questions, number of questions used to measure a particular concept, context of the question, and similar design decisions. Experiments may be embedded into field surveys, conducted in the laboratory, or some combination. Experiments may also be conducted by comparing survey data to other data sources such as external records or detailed respondent diaries. Some of this work may be performed through contracts.
b) Research on appropriateness of response scales: An important determinant of survey data quality is that questions include appropriate response scales. In particular, response scales must have clear meanings to participants, and must allow them to adequately express their experiences. An emerging body of research suggests that seemingly trivial variations in response scales (e.g., using a scale from 1 to 10 as opposed to a scale from -5 to +5) can significantly affect response distributions. Preliminary research has also been conducted on the meanings of vague quantifiers (such as often, sometimes, and rarely) and the benefits of certain scales over others (e.g., seven-point scales over feeling thermometers). QDRL staff will be engaged in additional research along these lines, possibly including cognitive laboratory testing of alternative response scales, as well as split-ballot experimentation.
c) Research on cognitive aspects of nonresponse: Nonresponse creates numerous analytic difficulties on major surveys. Minimizing this problem requires a greater understanding of the cognitive processes that lead participants to decide not to answer surveys or particular survey questions. QDRL staff plan to conduct cognitive interviews using a variety of types of survey questions (behavioral and attitudinal) in order to explore these decision processes further. Survey nonresponse will be explored through examination of reasons that nonresponders provide for their unwillingness or inability to complete surveys. It is also possible that data will be collected through experimental questionnaires administered outside of the laboratory that explore the effect of various design decisions on item nonresponse. Contracts may be used for some components of this data collection and analysis.
d) Research on Perceptions of Quality of Life: QDRL staff will examine survey participants’ perceptions of their self-assessed quality of life, and the basis for their responses to questions which purport to measure quality of life, especially from a health perspective. Such questions are increasingly important to both NCHS and CDC surveys as quality of life, rather then, simply length, becomes a key measure. Questions from the Behavior Risk Factor Surveillance System (BRFSS) Quality of Life Module will be subject to ongoing evaluation by QDRL staff. In particular, cognitive testing will be conducted to determine whether modifications to question wording, response category ordering, and question re-ordering are likely to fundamentally affect the patterns of responses obtained. We also anticipate that several experts in the field of survey methodology and health assessment may be enlisted, under contracts, to assist in this research by, for example, conducting independent cognitive research, and comparing those results with those obtained in the NCHS Laboratory.
e) Respondent Perceptions of Confidentiality and Survey Participation: To encourage participation, NCHS surveys such as the NHIS and NHANES depend on advance letters, promising confidentiality and explaining uses of the data collected (a copy of the 2005 NHIS advance letter is included as Attachment C). However, it is not known how well these statements are generally understood, and believed, by survey participants. Therefore, QDRL staff proposes to conduct cognitive interviews of laboratory participants in order to examine their comprehension of such statements. The results will be used to propose modifications to procedures used to communicate key issues related to informed consent, and to explain the need and purpose for survey data in a way intended to increase survey participation.
f) General Methodological Research: QDRL staff constantly evaluate and refine the cognitive interviewing methods used at NCHS, especially in order to respond to changes such as the wide-spread introduction of CAPI (Computer Assisted Personal Interviewing) as a data-collection mechanism. Further, QDRL staff regularly conduct applied research on questionnaire design issues, such as the optimal wording for measures of complex concepts related to health status, utilization, and behavior. In 2007-2009 QDRL staff plan to continue research on methods evaluation and general questionnaire design research. We envision that over the next three years, the QDRL will work collaboratively with survey researchers from Universities and other Federal agencies to define and examine several research areas, including, but not limited to: 1) differences between face-to-face and telephone cognitive interviewing, 2) effectiveness of different approaches to cognitive interviewing, such as concurrent and retrospective probing, 3) reactions of both survey participants and survey interviewers to the use of Computer Assisted Personal Interviewing (CAPI), and 4) social, cultural and linguistic factors in the question response process. Procedures for each of these studies will be similar to those applied in the usual testing of survey questions. For example, questionnaires that are of current interest (such as NHIS Modules) may be evaluated using several of the techniques described above. Or, different versions of a survey question will be developed, and the variants then administered to separate groups of participants in order to study the cognitive processes that account for the differences in responses obtained across different versions.
These studies will be conducted either by QDRL staff, DHHS staff, or NCHS contractors who are trained in cognitive interviewing techniques. The results of these studies will be applied to our specific questionnaire development activities in order to improve the methods that we use to conduct questionnaire testing, and to guide questionnaire design in general.
A.2.3 Research on human-computer interfaces/usability
The third major purpose of this data collection is to conduct research on computer-user interface designs for computer-assisted instruments, which is often referred to as “usability testing.” This research examines how survey questions, instructions, and supplemental information are presented on computer instruments (e.g., CAPI or Computer Assisted Self-Interviewing (CASI) instruments), and investigates how the presentation affects the ability of users to effectively utilize these instruments. Authors of computer-assisted instruments make numerous design decisions: how to position the survey question on a computer screen; how to display interviewer instructions that are not to be read to respondents; the maximum amount of information that can be effectively presented on one screen; how supplemental information such as “help screens” should be accessed; whether to use different colors for different types of information presented on the screen; and so on. Research has shown that these decisions can have a significant effect on the time required to administer survey questions, the accuracy of question-reading, the accuracy of data entry, and the full exploitation of resources available to help the user complete his or her task.
Usability testing has many obvious similarities to questionnaire-based cognitive research (described in Section A.2.1), since it focuses on the ability of individuals to understand and process information in order to accurately complete survey data collection. It is also somewhat different, in that the typical user can be an interviewer (in the case of CAPI instruments) as well as a respondent (in the case of CASI instruments). It also focuses more heavily on matters of formatting and presentation of information than traditional cognitive testing does.
A.2.4 Purpose and Use of Pilot Household Interviewing:
The fourth major purpose of data collection is to apply unobtrusive field-based questionnaire evaluation techniques, especially with respect to future NHIS Topical and Periodic Modules and modules from other surveys. The different questionnaires may be pilot-tested either individually or in groups, depending on developmental status of the instruments, the appropriateness of combining them, and their overall length. It is envisioned that for any single pilot test, five professional field interviewers will conduct a total of approximately 100 household interviews. There are two components to the proposed form of testing: a) a limited number of interviews on a draft version of the questionnaires that are conducted using household participants. In order to maximize realism these interviews will be conducted by NCHS and other staff trained in observational techniques, and b) inclusion in the questionnaires of two different versions of particular questions, in order to determine which version functions better in the field environment.
The major activities outlined above have well-demonstrated practical utility. As a result of laboratory testing, questionnaires may produce substantially less response error than would occur in the absence of this testing. Thus, users of NCHS data, in both Federal agencies and in the general health research community, will be less likely to be misled by erroneous statistical results. This assertion is supported by nineteen years of experience in using these techniques, and has been supported by findings presented at many statistical and research related conferences, and published in scientific journals such as Applied Cognitive Psychology. The practical utility of Pilot Household Interviewing has also been supported in findings reported at an annual meeting of the American Statistical Association. Further evaluation of the efficacy of this method will be ongoing.
A.2.5 Studies of the optimal design and presentation of graphical and textual material.
The final major purpose is related to the growth of the Internet for collecting data (including Web-based surveys), and in disseminating health information. NCHS is the Federal government’s principal health statistics agency, and is responsible for collecting and disseminating many reports and volumes of data annually. During the last few years, the techniques developed for determining whether participants understand survey questions have been applied with great utility to studying whether statistical publications and Web releases are optimally clear. Another recent project involved the development and testing of a brochure designed by staff of the National Health and Nutrition Examination Survey (NHANES) to convert refusals to acceptance. We anticipate that there will be more work of this type during 2007-2009.
A.3. Use of Improved Information Technology and Burden Reduction
Testing may be conducted using most recent modes of survey data collection, including CAPI/CASI, touch-tone data entry (TDE), or other modes applied to national surveys. Of course, direct interaction between research participants and interviewers remains a vital part of the testing process even when advanced data collection technology is used.
A.4. Efforts to Identify Duplication and Use of Similar Information
The QDRL at NCHS is the only government facility that currently conducts testing and development of NCHS or other CDC questionnaires, and is frequently the only one available to test questionnaires from other agencies. Similar facilities at the Bureau of the Census and the Bureau of Labor Statistics bear the responsibility for testing survey questionnaires associated with their own agencies. The demand for QDRL activities exceeds available resources.
It is also possible that the NCHS QDRL may collaborate through interagency agreements which will include confidentiality provisions to test and develop survey questionnaires. Consent forms will be modified to reflect their participation. This work, however, is not duplicative, but cooperative in nature, and should result in a higher quality final product. Researchers in the NCHS Questionnaire Design Research Laboratory also maintain very close contact with other experts in the field of questionnaire development in the academic survey community, in the health sciences field, at the Bureau of the Census, the Bureau of Labor Statistics, General Accounting Office, the National Science Foundation, National Cancer Institute, and the Energy Information Administration. From these contacts, it is clear that no other projects that duplicate the current proposal are now underway.
A.5. Impact on Small Businesses and Other Small Entities
In the past, representatives of small businesses have been interviewed as part of testing of establishment surveys, such as the National Employer Health Insurance Survey (NEHIS) (OMB# 0920-0341). If such requests are made, these businesses will be approached in the same manner as the individuals we normally recruit; we will ask the organization to identify the appropriate staff members with whom to conduct the cognitive interviews.
A.6. Consequences of Collecting the Information Less Frequently
The project usually involves one-time data collection activities. There are no legal obstacles to reducing the burden.
A.7. Special Circumstances Relating to Guidelines of 5 CFR 1320.5
There are no special circumstances.
A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agencies
A.8.1 A Federal Register notice for this collection was published on May 1, 2006 (Vol. 71, No. 83, p. 25590-25591). No comments have been received. The text of the notice is contained in Attachment D.
A.8.2.1 Other agencies: some of the topics selected for the NHIS are anticipated
to be requested by other agencies. These agencies have been involved in
development of survey objectives and the draft questionnaires.
A.8.2.2 Other individuals: Researchers who have special interest and expertise in the research areas explored will be contacted as necessary.
Consultants outside of CDC:
Nancy A. Mathiowetz
University of Wisconsin-Milwaukee
Bolton Hall #750
Milwaukee, WI 53201
(414) 229-2216
nancym2@uwm.edu
Alisu Schua-Glusberg
Research Support Services
906 Ridge Avenue
Evanston, IL 60602
(847) 864-5677
alisu@email.com
National Health Interview Survey questionnaires
(general consultation)
Floyd J. Fowler, Jr., Ph.D.
Center for Survey Research, University of Massachusetts
100 Arlington Street
Boston, Massachusetts 02116
(617) 956-1150
Floyd.fowler@umb.edu
James Scanlon
Deputy Assistant Secretary
Office of the Assistant Secretary for Planning and Evaluation
Hubert H. Humphrey Building
200 Independence Ave., S.W., Room 440D
Washington, D.C. 20201
(202) 690-7100
Jim.Scanlon@HHS.GOV
NHIS Periodic Modules
Ronald Manderscheid, Ph.D.
Chief, Survey and Analysis Branch
Division of State and Community Systems
Center for Mental Health Services
Room 15C-04
5600 Fishers Lane
Rockville, MD 20857
(301) 443-3343
rmanders@samhsa.gov
Gordon Willis, Ph.D.
National Cancer Institute
6130 Executive Boulevard
MSC 7344 EPN 4005
Rockville, MD 20892
(301) 594-6652
willisg@mail.nih.gov
General Methodological Research /Pilot Household Interviewing Project
Floyd J. Fowler, Jr., Ph.D.
Center for Survey Research, University of Massachusetts
100 Arlington Street
Boston, Massachusetts 02116
(617) 956-1150
Consultants within CDC:
NHIS Modules (general consultation)/Participant Perceptions of Confidentiality Project
Jane Gentleman, Ph.D.
Director, Division of Health Interview Statistics
Room 2218
National Center for Health Statistics
3311 Toledo Rd.
Hyattsville, MD 20782
(301) 458-4001
JGentleman@cdc.gov
CDC Behavioral Risk Factors Surveillance System (BRFSS) / Quality of Life Question Project
Paul Siegel, M.D.
Office of Surveillance and Analysis
National Center for Chronic Disease Prevention and Health Promotion
Mail stop K30
Centers for Disease Control and Prevention
Atlanta, GA 30333
(404) 488-5296
PZSiegel@cdc.gov
b) There are no unresolved problems.
c) Consultation with representatives of those from whom data will be collected will take place in the form of interviews with volunteers to determine the feasibility of collecting the needed data, the most promising approach for data collection, and general attitudes about the participants which might influence data collection.
A.9. Explanation of Any Payment or Gift to Respondents
Cognitive interview participants generally receive remuneration, for several reasons:
Eligibility criteria for participants are usually specific. Some of these criteria are determined by the subject matter of the survey (e.g., questions may be relevant only to people with certain health conditions). The more specific the subject matter, the more difficult it is to recruit eligible participants; payments help to attract them.
Cognitive interviews require an unusual level of mental effort, as participants are asked to explain their mental processes as they hear the question, discuss its meaning and point out any ambiguities, and evaluate the acceptability of response options that are provided.
They are usually asked to travel to the laboratory testing site, which involves transportation and parking expenses. (Many participants incur additional expenses due to leaving their jobs during business hours, making arrangements for child care, etc.).
For a standard cognitive interviewing project, in which one-hour interviews are conducted at NCHS and eligibility requirements are of average complexity, participants will be paid $40.00. The payment may be reduced to an amount no lower than $30.00 if the interview is of shorter duration, or does not require the participant to travel to NCHS. Higher payments may be requested on a case-by-case basis for particularly difficult recruitments. For example, in a 1995 study, the QDRL was unable to find auto mechanics and truck drivers willing to be interviewed for less than $75. On rare occasions, a lower payment is proposed.
It is important to offer payments sufficient to attract the full range of needed participant types for cognitive interviewing projects. Inadequate participant recruitment limits the effectiveness of the questionnaire evaluation. In addition, we face competition from other laboratories (public and private) in a highly saturated research area. Sometimes our advertisements run next to ads offering participants substantially higher payments for the same commitment. Requests and justification for remuneration will be included in each individual collection submission.
For activities that are meant to resemble the usual household interview (for example, our Pilot Household Interviewing activities), participants will not receive remuneration.
A.10. Assurances of Confidentiality Provided to Respondents
An assurance of confidentiality is provided to all respondents according to section 308(d) of the Public Health Service Act (42 USC 242m) which states:
"No information, if an establishment or person supplying the information or described in it is identifiable, obtained in the course of activities undertaken or supported under section...306 (NCHS legislation),...may be used for any purpose other than the purpose for which it was supplied unless such establishment or person has consented (as determined under regulations of the Secretary) to its use for such other purpose and (1) in the case of information obtained in the course of health statistical or epidemiological activities under section...306, such information may not be published or released in other form if the particular establishment or person supplying the information or described in it is identifiable unless such establishment or person has consented (as determined under regulations of the Secretary) to its publication or release in other form,..."
In addition, legislation covering confidentiality is provided according to section 513 of the Confidential Information Protection and Statistical Efficiency Act (PL 107-347) which states:
“Whoever, being an officer, employee, or agent of an agency acquiring information for exclusively statistical purposes, having taken and subscribed the oath of office, or having sworn to observe the limitations imposed by section 512, comes into possession of such information by reason of his or her being an officer, employee, or agent and, knowing that the disclosure of the specific information is prohibited under the provisions of this title, willfully discloses the information in any manner to a person or agency not entitled to receive it, shall be guilty of a class E felony and imprisoned for not more than 5 years, or fined not more than $250,000, or both.”
Informed consent: QDRL participants
QDRL participants are usually recruited by expressing their personal willingness to participate. They read or hear about the study through media advertisements, flyers, and word-of-mouth, and either call the laboratory answering machine number or contact a person coordinating the recruitment. Thus, participation is strictly voluntary and participants are not chosen randomly.
QDRL participants read and sign Attachment E, the Assurance of Confidentiality and Informed Consent Form (written at an 8th grade reading level), prior to the start of the interview. There are five templates in the attachment to cover various consent situations. The form states that participation is voluntary, they are free to terminate the interview at any time, and if they do so, they will still receive remuneration. The consent form describes the purpose of interview taping, specifies that the tapes may be played for other staff working closely on that project, that voice and face identifiers will remain on the tapes, and that they may be recognized by a staff member viewing or listening to the tapes. Participants are given a copy of the consent form, which contains contact information for the QDRL Laboratory Manager, the NCHS Research Ethics Review Board (ERB), and the NCHS Confidentiality Officer.
In addition to consenting for the interview to be taped at the beginning of the interview, a participant may be asked by the interviewer to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings Form [at the close of the interview]. The purpose of this form is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research finding at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. Participants are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If participants grant Special Consent, recordings are kept for as long as there is a justifiable use for the recordings as determined by the NCHS ERB.
At the close of the interview, the participant may also be asked by the interviewer to sign Attachment I, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL participants’ privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interview with minors will never be shown to others not included in the study staff. Participants are given a copy of the form which contains information about how to contact the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
Informed consent: Pilot Household Interviews
For pilot interviews of household and telephone participants, standard operating procedures regarding informed consent specific to the survey being tested will be slightly modified for QDRL testing to reflect “participation in the testing of survey questions,” rather than “participation in the actual survey.”
Confidentiality of responses and safeguarding of materials: QDRL participants
All participants receive Attachment E, the Assurance of Confidentiality and Informed Consent Form, which describes the procedures by which confidentiality of records identifying individuals is maintained.
Laboratory Manager Responsibilities
The Laboratory Manager is responsible for safeguarding schedules, consent documents, audiotapes and videotapes, questionnaires, the QDRL computer database, and cash payments to participants. The outside doors to the Interview Rooms and the Observation Room lock from the outside. A closet in the Interview Room also locks and is used primarily to store equipment and money for participant payments.
Documents: In order to protect the confidentiality of QDRL participants, the NCHS Confidentiality Office has requested that the Laboratory Manager store QDRL project-related documents in three separate locked filing cabinets.
Cabinet # 1: Names, telephone numbers, and other information transcribed from the answering machine onto a list of potential participants; signed Assurance of Confidentiality and Informed Consent Form; signed Special Consent for Expanded Use of Video and Audio Recordings; signed Special Consent to Send Video and Audio Recordings to Off-site Researchers.
Cabinet #2: Individual project file folders containing: OMB documentation; NCHS ERB documentation; demographics of participants; recruitment methods; questionnaires; and final reports.
Cabinet #3: Copy of the memorandum to the NCHS Budget Office summarizing the project payments by date, name and amount, and copies of the SF-1164, Participant Payment Forms.
The original and one copy of the SF-1164, Participant Payment Form, as well as a memorandum summarizing the project, payments by date, name and amount, are given to the NCHS budget analyst assigned to ORM. The analyst has been instructed to safeguard these documents and to store them in a locked cabinet or drawer. It is our understanding that the NCHS budget analyst sends one copy of the SF-1164 Participant Payment Form and memorandum to CDC/FMO with safeguarding instructions.
Waiver of SF-1164 and signed informed consent
On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, address, telephone number, and social security number) which the QDRL routinely collects in order to 1) pay respondents through the SF-1164 form and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the participant at risk of potential harm resulting from breach of confidentiality. In these cases, the QDRL has made arrangements with the CDC Budget Office not to use the SF-1164. In its place, a project specific tabular form is used to collect participant #, date, time, and location. Two interviewers sign-off certifying that payment in the amount of $XX has been made to the participant.
In addition, the QDRL requests a waiver of signed informed consent from the NCHS ERB.
Computer information: A custom-designed QDRL database contains personal information on participants including name, address, phone numbers, date-of-birth, marital status, employment, and household income. Other information, such as schedules, is automated in a word processing file. The computer is in the Laboratory Manager’s locked office and is password protected. Back-up files are locked in a cabinet.
Safeguarding of audio and video recordings at NCHS: The Laboratory Manager labels each audio and video recording by participant identifier number, date, time, and project title. No other identifying information is labeled on the recording. Recordings are viewed/listened to in the Observation Room which is locked when not in use, or on QDRL Staff’s desk top computers which are hardwired to the secure QDRL Local Area Network (LAN). Only QDRL Staff holding proper passwords have access to interview recordings on the QDRL LAN. The QDRL LAN is located in the QDRL Control Room and locked when not in use. Because the server is not located on the NCHS LAN, it is inaccessible to others inside or outside NCHS. Recordings are stored in the Observation Room which is locked when not in use, or on the secure QDRL LAN.
Safeguarding of video recordings seen by staff other than NCHS: Depending on the project, sponsors and collaborators may be from CDC, and occasionally from other DHHS or outside Federal agencies. The Assurance of Confidentiality and Informed Consent Form is tailored to describe each project and will specify which agencies are collaborating in the research and which staff(s) may be viewing the recording. Any outside NCHS collaborator viewing the recording (whether onsite at NCHS or off-site at a collaborating agency) will be required to sign a Nondisclosure Affidavit. Occasionally, a collaborator will be unable to travel to NCHS to view the recording. For those recordings in which Special Consent to Send Video and Audio Recordings to Off-site Researchers was granted, a contract will be developed and signed by the Director of NCHS and the Director of the relevant organization. The contract will state how they will protect the participants’ privacy and the recording until its return to NCHS. The contract will be coordinated through the NCHS Confidentiality Officer who will oversee shipment and the return of the recording to NCHS.
Interviewer Responsibilities
Upon completion of an interview, the interviewer is responsible for the questionnaire, and any notes written on other pieces of paper. The interviewer is instructed to lock all materials in his/her work area until all analysis is completed.
Reports, publications, and presentations: No participant names or other identifying information is included in any reports, publications, or presentations of cognitive testing results.
Interviewing outside of the QDRL1: Sometimes interviewers must travel to establishments or individuals’ homes in order to conduct interviews when it is not feasible for participants to travel to the QDRL. It is the interviewer’s responsibility to take necessary steps to ensure confidentiality and safeguarding of materials. Interviews at establishments should be conducted in private rooms with a closed door. If no private room is available, the participant can select a private area and the interviewer will judge whether the area is private enough to ensure confidentiality. If the interviewer assesses that the area is not private and/or soundproof enough, and no alternative area can be provided, the interview is canceled. For those surveys conducted in the participant’s home, the interviewer requests in advance that the participant arrange for privacy. However, interview location within the home is the choice of the participant. In all cases, extreme care is taken with audio and video recordings and any materials that contain personal identifiers such as the SF-1164 Participant Payment Form, the Assurance of Confidentiality and Informed Consent Form, Special Consent for Expanded Use of Video and Audio Recordings, or the Special Consent to Send Video and Audio Recordings to Off-site Researchers. Materials are then transported to the QDRL, where standard procedures are followed.
Focus groups: In focus group settings, participants are together and obviously can hear each other’s comments, statements, and questions. Participants are told in their initial telephone screening interview that they will be participating in a discussion group with other volunteers. Before the group discussion begins, participants sign the Assurance of Confidentiality and Informed Consent Form which is tailored to specify that they will be participating in a focus group. The Assurance of Confidentiality also states that they will be asked to pick a name and put it on a name tag, and that they do not have to use their real name. It is the responsibility of the interviewer (usually referred to as a moderator when conducting a focus group) to instruct the group that the information discussed will be held confidential by NCHS staff and should be treated confidentially by all participants. Participants are strongly urged to respect the privacy of the other participants and not to discuss with others what was discussed by the group.
Pilot household interviews: For each Pilot Household Interview, approximately five professional field interviewers having experience administering the particular survey to be tested will be assigned on a temporary (one week) basis to NCHS for purposes of carrying out these interviews. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. However, these procedures will be slightly modified for QDRL testing to reflect “participation in the testing of survey questions,” rather than “participation in the actual survey.”
Contractor Responsibilities
On the rare occasion when contractors are used to collect data as part of QDRL projects, they are contractually bound by NCHS confidentiality provisions, and must submit documentation concerning their safeguarding practices to NCHS prior to data collection. This is standard NCHS practice and does not reflect a special QDRL procedure. The contractor employee will view the NCHS Confidentiality video and sign the NCHS non-disclosure statement before starting work on the project.
The NCHS Privacy Act Coordinator has reviewed this request and has determined that the Privacy Act is applicable. The System of Records name is Health and Demographic Surveys Conducted in Probability Samples of the U.S. Population.
The Privacy Act System of Records number for the QDRL is 09-20-0164.
A.11. Justification for Sensitive Questions
Most of the questionnaires currently proposed for study generally do not contain questions that are highly sensitive in nature. There are some exceptions, such as the National Survey of Family Growth, NHIS questions on income and HIV, and NHANES questions on sexual orientation. Again, one purpose of cognitive and other pre-testing of such questions is to determine means for fashioning these questions in such a way that sensitivity is minimized, and responses are valid. Attachment J contains NCHS ERB approval for research involving human participants. An updated approval will be provided once it is received.
A.12. Estimates of Annualized Burden hours and costs:
A. An average of 850 individuals participate in QDRL activities in a given year and the average annual participant burden is estimated to be 600 hours. Estimates of participant burden for each of the questionnaire development studies, over the course of data collection, are provided below. Estimates are based mainly on the practice of conducting one-hour interviews with participants. The estimates cover the time that each participant will spend communicating with the Laboratory Manager (see Attachment K), in answering screener questions and survey questions and, in some cases, being debriefed about the decision and recall strategies they used. For our General Methodological Research studies, questionnaire administration is anticipated to frequently require less than an hour of a participant’s time (for example, a fifteen-minute interview may be conducted), and in rare cases, the burden may be more than one hour (although not more than 2 hours). Because the hours per response are expected to vary, we will select the final sample size for each project in such a way that the total burden hours do not exceed the estimate listed above. For focus groups, the usual amount of time is 90 minutes (1.5 hours) with instructions and ancillary paperwork processes taking an additional 15-25 minutes.
For interviews in the laboratory, time required to travel to the lab is not covered, because distances and modes of transportation are unknown. No retrieval of information by participants is anticipated; although it is possible that validation of data at some point may require participants to check records, probably those kept at home. In that case, the study will be designed so that the response time includes record retrieval. All estimates are based on NCHS' experience with 1988 through 2006 questionnaire development activities.
Average Annual Burden
Projects |
Number of Participants |
Number of Responses/ Participant |
Average hours per response |
Response burden |
QDRL Interviews |
|
|
|
|
1) NCHS Surveys |
120 |
1 |
1.25 |
150 |
2) Other questionnaire testing |
120 |
1 |
1.25 |
150 |
|
500 |
1 |
18/60 |
150 |
|
60 |
1 |
1.25 |
75 |
Focus Groups (5 groups of 10) |
50 |
1 |
1.5 |
75 |
Total |
|
|
|
600 |
B. Annualized costs to respondents.
No costs are anticipated. Payments to participants are designed to compensate them for their effort and any out-of-pocket costs.
A.13. Estimates of Other Total Annual Cost Burden to Respondents and Recordkeepers
None.
A.14. Annualized Costs to the Federal Government
The cost to the government consists mainly of the salaries of the lab staff who will (1) assist the questionnaire designers in the design of appropriate laboratory instruments, (2) recruit, schedule, and assist in interviewing volunteer participants, and (3) assist in the analysis of the results and recommend changes in questionnaire wording.
Total annualized project costs are as follows:
NCHS costs for laboratory staff to plan, conduct, and analyze the outcomes of the questionnaire development activities:
Managerial 1.00 FTE $114,000
Professional 7.00 FTE $626,000
Support 1.00 FTE $55,000
Payment of QDRL participants 460 @ $40 $18,400
Payment, under contracts, for assistance with
methodological research $125,000
Local travel: (see note below under
travel costs) $3,000
Materials for conducting household
interviews: $400
transcription costs, and hardware and software upgrades $50,000
Flyers $150
Advertisements $10,500
TOTAL $1,002,450.00
Travel costs: Most data will be collected in NCHS office space. However, it will be more efficient in certain instances to hold interviews with individuals at other locations, which will involve some travel costs. Further, household interviews will require limited numbers of in-person interviews in participant households. Household interviews will be done locally, in order to limit travel costs, unless there is a compelling reason to do otherwise (for example, if participants critical to the study can be interviewed only at a distant location).
A.15. Explanation for Program Changes or Adjustments
This project is included in the current inventory (No. 0920-0222) at 600 hours annually. There is no hour increase/change in burden.
A.16. Plans for Tabulation and Publication and Project Time Schedule
This clearance request is for questionnaire development activities to be conducted prior to field-testing, and for developmental work that will guide future questionnaire design. The majority of laboratory investigations will be analyzed qualitatively. The survey designers and lab staff serve as interviewers, and use detailed notes and transcriptions from the in-depth cognitive interviews to conduct analyses. The results of these investigations will be used primarily to develop reliable survey instruments and methods. For the Pilot Household Interviewing activities, qualitative and quantitative analysis will be performed on samples of observational data from household interviews in order to determine where additional problems occur. Because NCHS is using state-of-the-art questionnaire development techniques, methodological papers will be written which may include descriptions of response problems, recall strategies used, and quantitative analysis of frequency counts of several classes of problems that are uncovered through the lab interview and observation coding techniques.
A.17. Reason(s) Display of OMB Expiration Date is Inappropriate
Not applicable.
A.18. Exceptions to Certification for Paperwork Reduction Act Submissions
Not applicable.
B. STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
The purpose of cognitive interviewing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions.
Data collection procedures for cognitive interviewing are quite different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves such as: how participants interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
While survey research employs a deductive, quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, cognitive testing usually employs an inductive, qualitative methodology and generally relies upon a relatively small sample. Unlike survey research, the primary objective of cognitive testing is not to produce statistical data that can be generalized to an entire population. Rather, the objective of cognitive testing is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally do not achieve full inclusivity of all social and demographic groups. As a general rule, sample definitions are based upon the content of the survey, as well as the purpose and objectives of the particular study.
Whenever possible and appropriate, laboratory interviews may also rely on quantitative analysis, e.g., whether participants were assigned to random groups to receive different versions of questionnaires, or engage in well-established laboratory tests such as “card sorts.” Cognitive interviewing also does not preclude basic tabulations of responses, especially when samples are relatively large.
For the Pilot Household Interviewing study, participants will be sampled from household units in unused Census Bureau area segments in local communities, or, if conducted by telephone, will be selected through random-digit dialing.
B.2. Procedures for the Collection of Information
Recruitment. Participants will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, the advertisement or flyer may be developed to identify appropriate participants. For example, if the questionnaire to be tested includes a majority of questions about asthma-related health behaviors, then the recruitment may target asthma sufferers. A sample recruitment flyer and a sample newspaper advertisement are enclosed as Attachments L and M. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact who then either posts the flyer or distributes it to members of the organization.
For the pilot household interviewing studies, such as those done for NHIS, participants were sampled from household units in unused Census Bureau area segments in local communities. Census Bureau field interviewers followed their standard procedure by walking through neighborhoods, knocking on doors, and utilizing an advance letter (See Attachment C for copy of an Advance letter that is sent to actual survey respondents) to find persons willing to participate.
b) Screening and scheduling procedures. The first contact with potential laboratory research participants occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on an answering machine. The Laboratory Manager then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview/focus group takes place, video/audio taping procedures, and the remuneration to be offered. First, the Laboratory Manager determines through a brief series of questions (Attachment K) whether the volunteer possesses the desired research characteristics (e.g., we ask for gender and age to avoid interviewing 8 people with very similar demographic characteristics). If the person does possess the desired research characteristics and would like to participate, he/she is scheduled for an interview/focus group. Otherwise, the volunteer is asked whether he/she would be interested in participating in future laboratory interviews. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers and for those who would like to be contacted in the future. For those callers who are ineligible for the study and do not want to be contacted in the future, only demographic characteristics will be maintained for future analysis of successful recruitment efforts. Attachment N contains a sample script of the answering machine message. Attachment K contains a sample screening script.
For pilot interviews of household participants, the questionnaires to be tested are applicable to the general population, so no special population selection will be necessary. Any adult household participant is eligible for the interview.
QDRL Interview Methods.
i) One-on-one Interviews. If a laboratory interview is scheduled, the individual will usually travel to the QDRL. The QDRL is located on the 6th floor in the NCHS office building in Hyattsville, Maryland. The lab contains a waiting room, a control room, two private interview rooms, and a remote observation room for use by survey sponsors and other researchers to view an interview while in progress. On rare occasions, a participant may be unable to travel to the laboratory for the interview (for example, an individual may be housebound or have very limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for laboratory interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by the Laboratory Manager before the day of interview.
When the participant arrives at the QDRL, he/she is greeted by the Laboratory Manager and asked to read a brief description of the study, which includes assurances of confidentiality and the legislative authority for the research (Attachment E). The need for recording the interview (audio or video) is explained and the participant is asked to sign a consent form. The form is designed at an 8th grade reading level. In the rare instance that consent is not granted, the session is not recorded in audio or video, depending upon the individual’s concern area. If the participant grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped. The interviewer will get verbal consent from the participant to do so prior to turning off the machine. If the participant does not give verbal consent the interviewer will mark the tape for erasure, and the interviewing materials will be destroyed. If offsite, the interviewer will give the tape to the Laboratory Manager for erasure upon their return to NCHS. Otherwise the tape will be erased and the interviewing materials destroyed within 24-hours of the interview.
Attached to the Assurance of Confidentiality and Informed Consent Form is Attachment F, the Participant Data Collection sheet that participants are asked to complete. The purpose of this sheet is to collect recruitment and sociodemographic information on the participant. Participants are also asked to fill-out the SF-1164 Payment form.
On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, address, telephone number, and social security number) which the QDRL routinely collects in order to 1) pay respondents through the SF-1164 form and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the participant at risk of potential harm resulting from breach of confidentiality. In these cases, the QDRL has made arrangements with the CDC Budget Office not to use the SF-1164. In its place, a project specific tabular form is used to collect participant number, date, time, and location of the interview. Two interviewers sign-off certifying that payment in the advertised amount has been made to the participant.
In addition, the QDRL requests a waiver of signed informed consent from the NCHS ERB.
The interviewer then begins the interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used (see Attachment O). Interviewing procedures vary depending on the specific laboratory technique to be applied. The selection of the laboratory technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, participants are presented draft survey questions and asked to explain how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved. The interviewer may also ask the participant to think aloud while answering.
If possible, the cognitive interview will be conducted in the mode intended for the survey, either face-to-face, Computer Assisted Personal Interviewing (CAPI), telephone, Computer Assisted Telephone Interviewing (CATI), self-administered or web-based. For a telephone interview, the participant is called from one lab room to another and in-person debriefing follows.
In addition to consenting for the interview to be taped at the beginning of the interview, the participant may be asked at the end of the interview, and after receipt of remuneration, to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. If the participant does sign the special consent form, he/she will be given a copy which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
The participant may also be asked by the interviewer to sign Attachment I, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL participant’s privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. If the participant does sign the special consent to send video and audio recordings form, he/she will be given a copy which contains information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
ii) Focus Group Methods. Participants generally need to travel to the focus group location, which could be at NCHS, another Federal agency, or a room at another institution.
When participants arrive they are greeted by staff working on the project and directed to the focus group room where they are individually greeted by the Laboratory Manager/Staff person. Participants are given a packet containing the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E), the Participant Data Collection Sheet (Attachment F), the SF-1164 Payment form and instructed to fill them out. To maintain confidentiality, participants are seated at separate tables. Once the forms have been completed, they will be returned to the QDRL manager for completeness, verification and safekeeping.
Participants then each receive a separate packet containing a thank-you letter signed by the Director of NCHS, their remuneration, and a copy of the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E). Participants are then ushered into the focus group room and are seated around a table. In the rare instance that consent is not granted, participants will still receive remuneration.
A QDRL staff member or person working on the project, as outlined in the Assurance of Confidentiality and Informed Consent, will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell participants to pick a name to put on the name tag. Participants will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
In addition to consenting for the discussion to be taped at the beginning of the focus group, participants may be asked, at the close of the discussion, to sign Attachment H, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Participants of Discussion Groups. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. Participants will be given a copy of the form which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If any one participant from the focus group does not grant special consent, the tape will not be used in this way.
Once the focus group has concluded, the QDRL staff member or person working on the project will usher participants to the elevator and take them to the lobby exit.
Pilot Household Interview Methods.
For each of the tested questionnaires, between 50 and 100 household interviews will be conducted. These interviews will be conducted at such a time that the questionnaire to be tested has been developed in the laboratory, but has not yet been field pre-tested. Questionnaires will be combined for this testing when they are at similar levels of development, or where it is logically dictated that they should be combined, such as for the appropriate NHIS Core and Topical Modules. Questionnaires will be either draft paper versions, or draft CAPI instruments contained on laptop computers.
For each pilot test to be conducted, approximately five professional field interviewers having experience administering the particular survey to be tested will be assigned on a temporary (one week) basis to NCHS for purposes of carrying out these interviews. Households for these interviews will be selected randomly within a prescribed interviewing area (defined according to Census tract location), or through random-digit dialing; these methods have been found to be successful in the past. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. However, these procedures may be slightly modified for QDRL testing to reflect “participation in the testing of survey questions” rather than “participation in the actual survey.” Individual project protocols submitted for NCHS ERB approval will specify whether participation is for an actual survey or the testing of survey questions. If, at the time of pilot household testing, the survey’s informed consent and administration procedures are in the process of being amended, previous approved procedures will be modified for use. Individual 10-day notices submitted for OMB approval will outline the informed consent procedures and survey administration procedures specifically modified for QDRL testing. If time allows, prescripted follow-up probes may be asked to clarify participants’ responses. An NCHS staff member or a staff member of the agency sponsoring the questionnaire will observe as many of the interviews as possible. The results of pilot household testing will be used to modify the questionnaire for the later field pretest.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify participants. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory participant's time has been a proven incentive for volunteers to participate in the study.
After laboratory volunteers have been recruited, the probability of the participant failing to show is minimized by making reminder phone calls to volunteers.
B.4. Tests of Procedures or Methods to be Undertaken
This submission is a request for authorization to conduct tests of procedures and methodologies typical in cognitive testing research. The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of CDC and other Federal surveys cognitively tested by the Questionnaire Design Research Laboratory (QDRL) over the past 20 years. The procedures and methodologies used by the QDRL are consistent with other cognitive testing laboratories such as the Census Bureau, and the Bureau of Labor Statistics.
B.5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data
The person with overall responsibility for the statistical and technical aspects of the described activities is:
Lawrence H. Cox, Ph.D.
Associate Director, Office of Research and Methodology
Room 3211
National Center for Health Statistics
3311 Toledo Road
Hyattsville, Maryland
(301) 458-4631
LCox@cdc.gov
1Off-site interviews fall into two categories. First, it is not always feasible for individuals to travel to the QDRL, or it may be more efficient for interviewers to travel to a particular site. Second, we occasionally conduct establishment studies where a visit to the business location is pertinent to data collection.
| File Type | application/msword |
| File Title | Supporting Statement for Request for Clearance: |
| Author | Karen Whitaker |
| Last Modified By | mxm3 |
| File Modified | 2006-10-06 |
| File Created | 2006-10-06 |