Supporting Statement B for Request for Clearance:
QUESTIONNAIRE DESIGN RESEARCH LABORATORY
2010-2012
OMB No. 0920-0222
Contact Information:
Karen R. Whitaker, B.S.
Laboratory Manager, Questionnaire Design Research Laboratory
Office of Research and Methodology
National Center for Health Statistics/CDC
3311 Toledo Road, Room 3225
Hyattsville, MD 20782
301-458-4569
301-458-4031 (fax)
kwhitaker@cdc.gov
November 27, 2009
B. STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
The purpose of cognitive interviewing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions.
Data collection procedures for cognitive interviewing are different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves such as: how respondents interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
While survey research employs a deductive, quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, cognitive testing usually employs an inductive, qualitative methodology and generally relies upon a relatively small sample. Unlike survey research, the primary objective of cognitive testing is not to produce statistical data that can be generalized to an entire population. Rather, the objective of cognitive testing is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally do not achieve full inclusivity of all social and demographic groups. As a general rule, sample definitions are based upon the content of the survey, as well as the purpose and objectives of the particular study.
Whenever possible and appropriate, laboratory interviews may also rely on quantitative analysis, e.g., whether respondents were assigned to random groups to receive different versions of questionnaires, or engage in well-established laboratory tests such as “card sorts.” Cognitive interviewing also does not preclude basic tabulations of responses, especially when samples are relatively large.
For a pilot household interviewing study, such as the National Health Interview Survey (NHIS), respondents can be selected from the NHIS sample itself.
B.2. Procedures for the Collection of Information
Recruitment. Laboratory respondents will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, the advertisement or flyer may be developed to identify appropriate respondents. For example, if the questionnaire to be tested includes a majority of questions about asthma-related health behaviors, then the recruitment may target asthma sufferers. A sample recruitment flyer and a sample newspaper advertisement are enclosed as Attachments L and M. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact who then either posts the flyer or distributes it to members of the organization.
For pilot household interviewing studies, such as those done for NHIS, respondents can be sampled from household units in unused Census Bureau area segments in local communities. Census Bureau field interviewers follow their standard procedure by walking through neighborhoods, knocking on doors, and utilizing an advance letter (See Attachment C for copy of an Advance letter that is sent to actual survey respondents) to find persons willing to participate.
b) Screening and scheduling procedures. The first contact with potential laboratory research respondents occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on an answering machine. The QDRL Laboratory Manager/QDRL Staff person then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview/focus group takes place, video/audio taping procedures, and the remuneration to be offered. First, the QDRL Laboratory Manager/QDRL Staff person determines through a brief series of questions (Attachment K) whether the volunteer possesses the desired research characteristics (e.g., we ask for gender and age to avoid interviewing 8 people with very similar demographic characteristics). If the person does possess the desired research characteristics and would like to participate, he/she is scheduled for an interview/focus group. Otherwise, the volunteer is asked whether he/she would be interested in participating in future laboratory interviews. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers and for those who would like to be contacted in the future. For those callers who are ineligible for the study and do not want to be contacted in the future, only demographic characteristics will be maintained for future analysis of successful recruitment efforts. Attachment N contains a sample script of the answering machine message. Attachment K contains a sample screening script.
For pilot interviews of household respondents, the questionnaires to be tested are applicable to the general population, so no special population selection will be necessary. Any adult household respondent is eligible for the interview.
c) QDRL Interview Methods.
i) One-on-one Interviews. If a laboratory interview is scheduled, the individual will usually travel to the QDRL. The QDRL is located on the 6th floor in the NCHS office building in Hyattsville, Maryland. The lab contains a waiting room, a control room, two private interview rooms, and a remote observation room for use by survey sponsors and other researchers to view an interview while in progress. On rare occasions, a respondent may be unable to travel to the laboratory for the interview (for example, an individual may be housebound or have very limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the respondent, such as a senior center. To reduce the number of "no shows" for laboratory interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by the QDRL Laboratory Manager/QDRL Staff person before the day of interview.
When the respondent arrives at the QDRL, he/she is greeted by the QDRL Laboratory Manager/QDRL Staff person and asked to read a brief description of the study, which includes assurances of confidentiality and the legislative authority for the research (Attachment D). The need for recording the interview (audio or video) is explained and the respondent is asked to sign a consent form. The form is designed at an 8th grade reading level. In the rare instance that consent is not granted, the session is not recorded in audio or video, depending upon the individual’s concern area. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped. The interviewer will get verbal consent from the respondent to do so prior to turning off the machine. If the respondent does not give verbal consent the interviewer will mark the tape for erasure, and the interviewing materials will be destroyed. If offsite, the interviewer will give the tape to the QDRL Laboratory Manager/QDRL Staff person conducting recruitment for erasure upon their return to NCHS. Otherwise the tape will be erased and the interviewing materials destroyed within 24-hours of the interview.
Attached to the Assurance of Confidentiality and Informed Consent Form is Attachment E, the Respondent Data Collection sheet that respondents are asked to complete. The purpose of this sheet is to collect recruitment and sociodemographic information on the respondent. Respondents are also asked to fill-out the Cash Payment Receipt Form (Attachment F).
On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, and address) which the QDRL routinely collects in order to 1) pay respondents and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the respondent at risk of potential harm resulting from breach of confidentiality.
In addition, the QDRL requests a waiver of signed informed consent from the NCHS ERB for these instances.
The interviewer then begins the interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used (see Attachment O). Interviewing procedures vary depending on the specific laboratory technique to be applied. The selection of the laboratory technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, respondents are presented draft survey questions and asked to explain how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved. The interviewer may also ask the respondent to think aloud while answering.
If possible, the cognitive interview will be conducted in the mode intended for the survey, either face-to-face, Computer Assisted Personal Interviewing (CAPI), telephone, Computer Assisted Telephone Interviewing (CATI), self-administered or web-based. For a telephone interview, the respondent is called from one lab room to another and in-person debriefing follows.
In addition to consenting for the interview to be taped at the beginning of the interview, the respondent may be asked at the end of the interview, and after receipt of remuneration, to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. If the respondent does sign the special consent form, he/she will be given a copy which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
The respondent may also be asked by the interviewer to sign Attachment I, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL respondent’s privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. If the respondent does sign the special consent to send video and audio recordings form, he/she will be given a copy which contains information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
ii) Focus Group Methods. Respondents generally need to travel to the focus group location, which could be at NCHS, another Federal agency, or a room at another institution.
When respondents arrive they are greeted by staff working on the project and directed to the focus group room where they are individually greeted by the QDRL Laboratory Manager/QDRL Staff person. Respondents are given a packet containing the Assurance of Confidentiality and Informed Consent for focus groups (Attachment D), the Respondent Data Collection Sheet (Attachment E), the Cash Payment Receipt Form (Attachment F) and instructed to fill them out. To maintain confidentiality, respondents are seated at separate tables. Once the forms have been completed, they will be returned to the QDRL manager for completeness, verification and safekeeping.
Respondents then each receive a separate packet containing a thank-you letter signed by the Director of NCHS, their remuneration, and a copy of the Assurance of Confidentiality and Informed Consent for focus groups (Attachment D). Respondents are then ushered into the focus group room and are seated around a table. In the rare instance that consent is not granted, respondents will still receive remuneration.
A QDRL staff member or person working on the project, as outlined in the Assurance of Confidentiality and Informed Consent, will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell respondents to pick a name to put on the name tag. Respondents will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
In addition to consenting for the discussion to be taped at the beginning of the focus group, respondents may be asked, at the close of the discussion, to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Respondents of Discussion Groups. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. Respondents will be given a copy of the form which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If any one respondent from the focus group does not grant special consent, the tape will not be used in this way.
Once the focus group has concluded, the QDRL staff member or person working on the project will usher respondents to the elevator and take them to the lobby exit.
d) Pilot Household Interview Methods. For each of the tested questionnaires, between 50 and 200 household interviews will be conducted. These interviews will be conducted at such a time that the questionnaire to be tested has been developed in the laboratory, but has not yet been field pre-tested. Questionnaires will be combined for this testing when they are at similar levels of development, or where it is logically dictated that they should be combined, such as for the appropriate NHIS Core and Topical Modules. Questionnaires will be either draft paper versions, or draft CAPI instruments contained on laptop computers.
If the pilot test is to be conducted in households that are part of the NHIS sample, the test questions will be incorporated into the NHIS questionnaire and administration will follow NHIS protocols.
For each pilot test to be conducted, approximately five professional field interviewers having experience administering the particular survey to be tested will be assigned on a temporary (one week) basis to NCHS for purposes of carrying out these interviews. Households for these interviews will be selected randomly within a prescribed interviewing area (defined according to Census tract location), or through random-digit dialing; these methods have been found to be successful in the past. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. However, these procedures may be slightly modified for QDRL testing to reflect “participation in the testing of survey questions” rather than “participation in the actual survey.” Individual project protocols submitted for NCHS ERB approval will specify whether participation is for an actual survey or the testing of survey questions. If, at the time of pilot household testing, the survey’s informed consent and administration procedures are in the process of being amended, previous approved procedures will be modified for use. Individual 10-day notices submitted for OMB approval will outline the informed consent procedures and survey administration procedures specifically modified for QDRL testing. If time allows, prescripted follow-up probes may be asked to clarify respondents’ responses. An NCHS staff member or a staff member of the agency sponsoring the questionnaire will observe as many of the interviews as possible. The results of pilot household testing will be used to modify the questionnaire for the later field pretest.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential laboratory research respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify respondents. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory respondent's time has been a proven incentive for volunteers to participate in the study.
After laboratory volunteers have been recruited, the probability of the respondent failing to show is minimized by making reminder phone calls to volunteers.
B.4. Tests of Procedures or Methods to be Undertaken
This submission is a request for authorization to conduct tests of procedures and methodologies typical in cognitive testing research and to build on this research through targeted field tests. The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of CDC and other Federal surveys cognitively tested by the Questionnaire Design Research Laboratory (QDRL) over the past 24 years. The procedures and methodologies used by the QDRL are consistent with other cognitive testing laboratories such as the Census Bureau, and the Bureau of Labor Statistics.
B.5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data
The person with overall responsibility for the statistical and technical aspects of the described activities is:
Jennifer Madans, Ph.D.
Associate Director of Science
National Center for Health Statistics
3311 Toledo Road
Hyattsville, Maryland
(301) 458-4500
JMadans@cdc.gov
LIST OF ATTACHMENTS
ATTACHMENT A: Public Health Service Act
ATTACHMENT B: 60-day Federal Register Notice
ATTACHMENT C: Sample of a 2009 advance letter sent to NHIS respondents
ATTACHMENT D: Informed consent templates for QDRL interviews
Template 1: Adults
Template 2: Minors (and parents of minors)
Template 3: Focus Groups
Template 4: Offsite
Template 5: Offsite/Waiver of Cash Payment Receipt Form & signed informed consent
ATTACHMENT E: Respondent data collection sheet
ATTACHMENT F: Cash Payment Receipt form
ATTACHMENT G: Special Consent for Expanded Use of Video and Audio Recordings
for Individual Respondents of Discussion Groups
ATTACHMENT I: Special Consent to Send Video and Audio Recordings to Off-site Researchers
ATTACHMENT J: IRB Approval Documentation (NCHS Research Ethics Review Board)
ATTACHMENT K: Sample screening script
Template 1: Respondent recruited from newspaper advertisement
Template 2: Respondent recruited from QDRL Database
ATTACHMENT L: Sample recruitment flyer
ATTACHMENT M: Sample newspaper advertisement
ATTACHMENT N: Sample script of answering machine
ATTACHMENT O: Detailed explanation of cognitive interviewing procedure read by interviewer to respondent
| File Type | application/msword |
| File Title | Supporting Statement for Request for Clearance: |
| Author | Karen Whitaker |
| Last Modified By | Karen Roberta Whitaker |
| File Modified | 2009-12-01 |
| File Created | 2009-12-01 |