Letter to Dr. Schwab - justification

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

May 13, 2011

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Questionnaire Design Research Laboratory (QDRL) (OMB No. 0920-0222, exp. 03/31/2013) plans to conduct Spanish-language research 1) to conduct usability testing on the Audio Computer-Assisted Self Interview (A-CASI) software developed to administer a proposed question on sexual identity for the National Health Interview Survey (NHIS) and, 2) to cognitively test the proposed sexual identity measure, as well as other demographic questions that appear on the NHIS, including questions on race, ethnicity, marital status, employment status and family income. English testing of this research project was described in a letter to you dated April 15, 2011 and approved on May 6, 2011. Since that time, a new version of the sexual identity question has been developed, and due to HHS’s revised timeline for testing of the sexual identity question, the social security number question and the questions on alcohol that are in the English version have been dropped. The social security number question and the alcohol use questions will be submitted for Spanish testing under a separate cover at a later time. The English version of the sexual identity question for males and females is shown below:

[Male version question]

9. Do you think of yourself as:

Gay, homosexual or Same-Gender-Loving

Straight or Heterosexual, that is, not gay

Bisexual

Something Else (Go to 9a)

Don’t Know (Go to 9b)

9a. [If something else is selected] By something else, do you mean that…

You are not straight, but identify with another label such as queer, trisexual, omnisexual or pan-sexual

You are transgender or are transitioning genders

You have not or are in the process of figuring out your sexuality

You do not think of yourself as having a sexuality

You personally reject all labels of yourself

You made a mistake and did not mean to pick this answer

You mean something else (Go to 9c)

9b. [If don’t know is selected] By Don’t Know, do you mean that…

You don’t understand the words

You understand the words, but you have not or are in the process of figuring out your sexuality

You mean something else

9c. [If you mean something else is selected]

What do you mean by something else?

Please type in your answer __________________________________

[Female Version Question]

9. Do you think of yourself as:

Lesbian, Gay or Homosexual

Straight or Heterosexual, that is, not gay

Bisexual

Something Else (Go to 10a)

Don’t Know (Go to 10b)

10a. [If something else is selected] By something else, do you mean that…

You are not straight, but identify with another label such as queer, trisexual, omnisexual or pan-sexual

You are transgender or are transitioning genders

You have not or are in the process of figuring out your sexuality

You do not think of yourself as having a sexuality

You personally reject all labels of yourself

You made a mistake and did not mean to pick this answer

You mean something else (Go to 10c)

10b. [If don’t know is selected] By Don’t Know, do you mean that…

You don’t understand the words

You understand the words, but you have not or are in the process of figuring out your sexuality

You mean something else

10c. [If you mean something else is selected]

What do you mean by something else?

Please type in your answer __________________________________

We propose to start advertising for volunteer participants as soon as we receive clearance and to start testing as soon as possible after that.

Background Information about Cognitive Testing of Questionnaires

The methodological design of this proposed study is consistent with the design of typical cognitive testing research. As you know, the purpose of cognitive testing is to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions. The analysis will be qualitative.

Proposed project: National Health Interview Survey (NHIS) Spanish A-CASI

The QDRL has been tasked with developing a measure of sexual identity for NHIS and an A-CASI system on which the question will be administered. A primary difficulty in designing a sexual identity question for a national population survey is that the construct of sexual identity (as opposed to a more objective construct, such as behavior) is a complex concept; respondents’ conceptions of the construct itself can vary dramatically depending on demographic characteristics (e.g., race/ethnicity, gender, age), socio-cultural context and geographic location. QDRL has conducted multiple evaluations using cognitive interviewing methods on various sets of questions designed to capture information on sexual identity.

Analysis consistently shows that the questions produce measurement error due to problems with respondent comprehension of terminology as well as the use of labels that are not consistent with the way respondents identify themselves. Analyses of survey data from the National Health and Nutrition Examination Survey (NHANES) and the National Survey of Family Growth (NSFG) support these conclusions. Lack of understanding key terms and unsuitable use of labels results in high levels of responses in non-substantive categories such as 'something else', 'don't know', and refused. As a result, responses that are not analyzed have higher prevalence than the responses of interest. Additionally, depending on how questions are worded, estimates vary and do not always make theoretical sense. For example, estimates from NHANES show twice as many gay men than bisexual men, while NSFG estimates show an equal proportion. Also of concern is the fact that the percent of the population responding in the non-substantive categories varies across demographic and socioeconomic statuses, which further reduces the ability to interpret the data. It is believed that the newly revised A-CASI sexual identity question will resolve many of these interpretive problems.

The sexual identity question being tested in this project was developed from previous cognitive testing studies. The first draft of this question was tested in the NHBS English/Spanish cognitive interviewing project using an interviewer administered paper instrument and was described in a letter dated to you on November 22, 2010/January 12, 2011 and approved on December 21, 2010/February 7, 2011. A new version of the sexual identity question was developed based on findings from that study. In addition, we will also test other demographic questions that appear on the NHIS, including questions on race, ethnicity, marital status, employment status and family income. Please note, respondents are asked income question 11 in order to determine the size of the family living in their home. In order to administer the appropriate family income questions, we must know the family size.

The introduction to the usability test/cognitive interview and usability interview guide appears in Attachment 1. The Spanish cognitive testing interview guide and Spanish A-CASI instrument appears in Attachment 2. Testing procedures will conform to the cognitive interviewing techniques that have been described in our generic OMB clearance package.

We propose to recruit as many as many as 100 Spanish speaking adults (aged 18 years and older). We plan to recruit 50 respondents who identify as heterosexual with some demographic variety (particularly in terms of gender, education, race/ethnicity, and income), and 50 respondents who identify as gay, lesbian, bisexual or transgendered with some demographic variety (particularly in terms of gender, education, race/ethnicity, and income). We will begin testing in the QDRL and we may need to test the instrument in various locations across the U.S. in order to ensure adequate coverage of respondent demographics. Respondents will be recruited through a newspaper advertisement (Attachment 3a), flyers distributed at community centers (Attachment 3a and 3b), by contacting respondents from past QDRL projects listed in the QDRL Respondent Database who have expressed interest in participating in upcoming projects, and through word-of-mouth.

General Interview Procedures

Cognitive interviews will be conducted by QDRL staff members, and interviews will last for 90 minutes each. Respondents will be informed of audio taping procedures during the screening process, and in the process of reviewing the consent forms prior to the start of the interview. Only individuals who agree to be audio taped will be eligible to participate in the study.

Due to the sensitive nature of the questions, the NCHS QDRL will conduct the study anonymously. QDRL Staff will collect minimal personal identifiers. A first name and a contact telephone number for scheduling and reminder calls, if available, may be used—however, full name, address and home telephone number, will not be collected. The QDRL routinely collects this information in order to 1) pay respondents through the use of the approved cash payment receipt form, and to 2) acquire informed consent. It is the QDRL’s belief that the collection of these identifiers would put the respondent at risk of potential harm resulting from a breach of confidentiality.

Respondents who decide to participate will be asked to read the waived signed Informed Consent form which allows for the audio taping of the interview. The interviewer will witness the reading of the waived signed Informed Consent form. In addition, prior to the start of the interview the interviewer will turn on the tape recorder and respondents will be asked to verbally acknowledge that they have agreed to participate in the research study and that they have agreed to be audio taped. If the respondent decides to turn off the audio tape anytime during the interview, they will be asked for consent to retain the interviewing materials and the portion already taped. The interviewer will get verbal consent from the respondent to do so prior to turning off the tape.

Interviews will begin with an overview about the purpose of the study. After a brief training session on using the A-CASI system, respondents will be asked to answer the questions using a computer with headphones. When respondents are done answering the questionnaire, interviewers will print off their responses. Interviewers will then conduct a face-to-face debriefing interview to assess respondents’ confidence in using the A-CASI, comfort level, and any problems them might have encountered. For purposes of the usability test, we will also collect paradata using the BLAISE software. Paradata include recording of response timing, keystrokes, access of definitions, access of help menus, use of audio, respondent’s general navigation through the instrument and changes to answers. Following the debriefing, we will conduct a cognitive test of the questionnaire. The QDRL interviewer will read the respondents the questions they were administered during the ACASI and remind them of their answers. Interviewers will ask emergent probes to better understand the question-response process. Examples of the sorts of probes that may be asked at the interviewer’s discretion include:

• Why did you answer that way?

• How did you arrive at your answer?

• Can you tell me more about that?

• Can you clarify what you mean?

Interviewers may use some or all of these probes, depending upon the content of the interview and how much information the respondents reveal without being prompted. In some interviews, probes will be administered throughout the interview, and for others they will be administered after completing the questionnaire.

We propose paying participants $50, which is $10 over our standard payment. Given the length of the cognitive interview (90 minutes vs. our traditional 60 minute interview), and given the sensitive nature of the questions we hope the extra $10 above our $40 standard payment will encourage respondents to participate in the study. In total, for this project, the maximum respondent burden will be 150 hours of interviewing in addition to travel time.

After the interview, respondents will be given the thank-you letter signed by the Director of NCHS, a copy of the informed consent document, and $50.

Because the interviews contain sensitive information and the respondent’s voice will be on the tape, the respondent will not be asked for Special Consent for Expanded Use of Video and Audio Recordings which would allow QDRL staff to play the tape at conferences as part of a presentation, for students, or for other people who write survey questions. Audio tapes will not be shared.

Extreme care will be taken with all tapes and paperwork from the interviews conducted off-site. Tapes and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets.

An updated burden table for this project is shown below:

Projects	Number of Participants	Number of Responses/ Participant	Average hours per response	Response burden
QDRL Interviews
2) NCHS Surveys	100	1	1.5	150

Attachments (3)

cc:

M. Moien

D. Holcomb

DHHS RCO

5