Form Protocol

Pre-decisional and deliberative

Cognitive Interview Protocol - 2023 Race/Ethnicity Question Testing

RESPONDENT ID #: _________________________ INTERVIEW DATE: ____ / ____ / ___

INTERVIEWER’S NAME:_________________________________________________

START TIME: ______: ______ AM / PM

PARTICIPANT DEMOGRAPHICS Interviewer: Fill in this information before your interview, from the tracker data. You can reference this at the end of the interview, when you get to the optional demographic confirmation questions.
Age group
Race
Hispanic
Language
Gender/SO
Number in household
Education
Additional noteworthy information

Interviewer: at the start of the interview, complete the following:

1. Connect to Teams

2. Confirm if respondent has already signed consent

   1. If consent already signed, verbally confirm consent for participation, consent for recording, and inform of observers (if present)

   2. If consent not signed, review it with the participant, ask them to sign it, verbally confirm consent for participation, consent for recording, and consent for observers

3. Ask if respondent has any questions

4. Start recorder and state date, time, participant number, and reconfirm consent

SURVEY LINKS:

R/E Vertical Question Testing Instrument (Version A): https://research.rm.census.gov/jfe/form/SV_3P47n8UdBshu1g2

R/E Two Page Question Testing Instrument (Version B): https://research.rm.census.gov/jfe/form/SV_6gLOcYnRsA86LlA

Section 1: Introduction

READ OR PARAPHRASE THE INTRODUCTION TO THE RESPONDENT

Thank you for agreeing to participate in our research study. Today, we are working on improving questions that may be used across many federal surveys. In a few minutes, I will send you a link to a questionnaire and ask you to answer questions about yourself and others in your household. I will also ask that you share your screen so I can watch your progress. I would like you to answer the questions the same way you would if you were at home, with one major difference.

READ THE THINK-ALOUD EXAMPLE IN ITALICS BELOW VERBATIM

You will be asked to “think aloud” as you answer each question. This may feel a little unnatural, but it will help us understand how you think about and answer each question. Here’s an example:

Suppose this were a survey about your home and one of the questions asked, “How many windows are there in your home?” Someone thinking aloud as they respond might say, “What do you mean by windows – I have sliding doors out to the back yard, so I am not sure whether to count those as windows or not. There are two windows in the kitchen, one in the living room, and two in each bedroom. So perhaps we have 7 windows. But I am not sure whether to count the sliding doors. I think I won’t count them, so I’ll answer 7 windows.

In this example it’s clear that “windows” is not a term everyone understands in the same way. This process of thinking out loud lets me understand how you came up with your answer.

After you read each question, think through your response aloud and tell me your answer. After you answer all of the questions, I will ask you some follow-up questions to get your feedback. For example, I might ask you what a particular term or phrase means or ask you to put the question in your own words. There are no right or wrong answers to any of the questions that I might ask. I am not trying to test or quiz you about these terms. We just want to make sure that everyone is understanding these survey questions in the same way. We also want to know if the questions are worded clearly and, if they are not, what changes should be made. Your participation is important because it will help the Census Bureau improve these questions.

Before we begin, do you have any questions?

ANSWER QUESTIONS, AS NEEDED

INTERVIEWER INSTRUCTIONS:

1. Send participant link to respondent instrument in the chat (If chat is not working, email link to participant)

2. Have participant share their screen

Section 2: Survey Questions

Ok, let’s begin. Please remember to think aloud as you answer the questions.

OBSERVATION NOTES:

OBSERVE ANY BEHAVIORS BELOW AND BE SURE TO DESCRIBE THEM IN YOUR SUMMARY.

• DOES R MARK AT LEAST ONE MAIN CATEGORY BOX, DETAILED CATEGORY BOX, AND/OR PROVIDE A WRITE IN, IF APPROPRIATE?

• DOES R GO BACK AND CHANGE ANY ANSWER AT ANY POINT?

• DOES R APPEAR TO SEARCH FOR ANYTHING OR HAVE DIFFICULTY IN LOCATING A RESPONSE (E.G., SCROLLING BACK AND FORTH, TAKING A LONG TIME, ETC.)?

• DOES R ANSWER AS EXPECTED BASED ON RECRUITMENT SCREENER RESPONSES?

• DOES R SEEM TO HAVE ANY DIFFICULTY UNDERSTANDING THE QUESTION?

• IS THERE A LITERACY AND/OR A LANGUAGE BARRIER?

• DOES R ASK FOR CLARIFICATION?

• DO THEY CLICK THE HELP TEXT?

• ANY OTHER NOTABLE BEHAVIORS?

Section 3: Retrospective Probes

Research Questions (Combined Question):

1. How do respondents understand and interpret the combined question?

2. Does the question format allow respondents to easily self-identify with the main and detailed categories?

AFTER PARTICIPANT FINISHES QUESTIONNAIRE: WALK THROUGH RETROSPECTIVE PROBES WHILE THEY REFERENCE RESPONSES TO QUESTIONS AND ANSWERS TO RESPONDENT WHEN POSSIBLE

I see you finished answering the questions. I'm going to ask you about your answer on this question.

PROBES ON RACE QUESTION, MAIN CATEGORIES

SHOW RESPONDENT COMBINED RACE QUESTION (Q5 – MAIN CATEGORIES ONLY)

SPONTANEOUS PROBES:

• If R had any minor or major problem with combined race question, probe that now. Probe specifically on any behaviors noticed during questionnaire administration for self or proxy responses.

  • Example: “I noticed that you [spent a long time/scrolled up and down/went back and forth] looking for an answer (thinking about what to write). Can you tell me what was going on for you then?”

• Interviewers should be aware of respondents’ screener identification so they know what to expect and can probe on unexpected responses.

When you answered this question, you [selected and/or entered] [FILL FROM SCREEN].

• How did you choose your answer for question 5?

• [IF R MARKED MORE THAN ONE]: How did you decide to mark more than one answer?

• What information did you think this question was asking for?

• How much of the question did you read? Where did you stop? What parts did you see?

• Were you able to easily identify yourself in this part of the question?

IF MORE THAN ONE PERSON IN HOUSEHOLD:

• Did you have difficulty answering for other members of your household?

• How did you choose your answer for Person 2? What about for Person 3?

MENA RESPONDENTS ONLY:

• How have you answered this question in the past? Did you consider marking any other categories?

• Have you seen the Middle Eastern North African category in the past?

IF PROBING ON TWO-PAGE VERSION, INSTRUCT R TO SELECT THE MAIN CATEGORIES AGAIN

INSTRUCT R TO ADVANCE TO NEXT PAGE (DETAILED CATEGORIES)

PROBES ON RACE QUESTION, DETAILED CATEGORIES

IF R SKIPPED THE DETAILED CATEGORIES:

• Can you tell me more about why you skipped this question?

IF R ANSWERED THE DETAILED CATEGORIES:

When you answered this question, you [selected and or entered] [FILL FROM SCREEN].

IF PROBING ON VERTICAL VERSION, INSTRUCT R TO SELECT THE MAIN CATEGORIES AGAIN TO UNFOLD.

• How did you choose your answer for this question?

• [AS APPROPRIATE]:

  • For what reason did you write in _____? What does [TERM RESPONDENT WROTE] mean to you?

  • I notice you didn't enter anything in down in the write-in line. Can you tell me for what reason?

• [IF R MARKED MORE THAN ONE]: How did you decide to mark more than one answer?

• What information did you think this question was asking for?

• How much of the question did you read? Where did you stop? What parts did you see?

• Were you able to easily identify yourself in this part of the question?

IF MORE THAN ONE PERSON IN HOUSEHOLD:

• Did you have difficulty answering for other members of your household?

• How did you choose your answer for Person 2? What about for Person 3?

PROBES ON QUESTION STEM

Research Questions (and/or):

3. How do respondents understand and/or as part of the questions stem?

4. How does and/or affect the way respondents interpret and answer the question?

5. Is this interpretation different than “or” alone?

• Can you tell me what “and/or” means to you in this question?

• What if it only said “or,” would you change your answer?

• What does the term race mean to you in this question?

• What does the term ethnicity mean to you in this question?

PROBES ON INSTRUCTIONS

Research Questions (instructions):

1. Does simplifying the instruction at the beginning of the combined question change how respondents understand and the question?

2. How do respondents interpret and understand the simplified instructions?

3. How do the instructions affect the way respondents interpret and answer the combined question?

IF R ANSWERED VERTICAL ONE-PAGE QUESTION:

• Can you tell me what “select all that apply” means to you in this question?

• Can you tell me what “provide details below” means to you in this question?

• Can you tell me what “enter, for example…” means to you in this question?

IF R ANSWERED TWO-PAGE QUESTION

• Can you tell me what “select all that apply” means to you in this question?

INSTRUCT R TO ADVANCE TO NEXT PAGE (DETAILED CATEGORIES)

• [POINT TO OR HIGHLIGHT INTRODUCTION STATEMENT] Can you tell me what this statement means to you on this screen?

• Can you tell me what “enter, for example…” means to you in this question?

TASK AND PROBES: ANSEWRING ALTERNATE VERSION OF QUESTION

INSTRUCT R TO ADVANCE TO NEXT PAGE (ALTERNATE VERSION OF QUESTION)

Thank you for your responses so far. Now we are going to look at another way of asking this same question. After you read the question, think through your response aloud and tell me your answer.

IF R ANSWERED ALTERNATE VERSION DIFFERENTLY THAN ORIGINAL:

• How did you choose your answer for this version of the question?

• [IF R MARKED MORE THAN ONE]: How did you decide to mark more than one answer?

ALL RESPONDENTS

• Was one version of the question easier to respond to?

• [AS APPROPRIATE]:

  • Which version was easier to respond to?

  • What made that version easier to respond to?

PROBES ON OMB RACE CATEGORY DEFINITIONS

Research Question (Help text):

1. How do respondents react to the “help text” or OMB definitions of the minimum categories?

• [If they accessed the help screen]: Tell me more about why you clicked the help link? Was it helpful?

• [IF THEY DID NOT ACCESS THE HELP SCREEN. ASK RESPONDENT TO LOOK AT THE DEFINITION THAT CORRESPONDS TO THEIR SELF-IDENTIFICATION]: Please read the [respondent’s chosen category(ies)] definition.

  • Would this/these definition(s) have helped you answer the question?

  • Do you have any other thoughts or reactions about these descriptions?

DEBRIEFING PROBES

• Anything else you would like to tell us before we close?

Those are all of the questions that I had for you today. Thank you very much for your participation.

HAVE R SIGN VOUCHER, EXPLAIN TIMELINE FOR SENDING OUT INCENTIVE

10