Various Pretesting Activities

Cognitive Testing of Translated ACS Language Assistance Guides

Protocol Guide - English

Cognitive Testing of Translated ACS Language Assistance Guides

Protocol Guide – English

“A”

Phase 1 - Round 1

PARTICIPANT ID #: |_P_|___|_R_|___|___|___|___|___|___| DATE: ____ / ____ / 2010

LANGUAGE (CHECK ONE): 1 CHINESE  Specify dialect: ____________________

2 KOREAN

INTERVIEWER: 1 Ginny 2 Hyunjoo 3 Jiyoung 4 Suzie 5 Younhee

6 Gretchen 7 Guangya 8 Mandy 9 Michelle 10 Lu 11 Grace

LAG CONTENT: Start Here box, Persons 1 and 2, and Housing Questions 1 – 11

START TIME: ____________ AM / PM

SECTION I: INFORMED CONSENT

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

A. Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

 VERBAL CONSENT MUST BE AUDIO-RECORDED

This is a form called “Language Assistance Guide”. It has been translated in [Chinese/Korean] for the American Community Survey conducted by the Census Bureau.

• The Census Bureau counts population in the U.S. and also conducts various kinds of

surveys. They routinely encounter non-English speaking households and therefore would

like to have this form available in other languages beside English. It will be mailed to

people who request them across the U.S.

• Researchers at the Census Bureau are interested in knowing your opinions about the translation.

• They have hired RTI International to conduct this research. Your participation is very

important because it will help the Census Bureau determine if this form is translated well

and is culturally appropriate.

• I’d like you to read through this form and record your answers to the questions. Please try to complete the form on your own, as if I were not here. While you fill out the form, I’ll be taking some notes and then we will discuss the translation together.

• This is NOT any kind of test and there are no right or wrong ways to fill it out. We are practicing with as many different kinds of people as possible in different parts of the country to see if everyone fills out the form the same way and to see if it’s easy or difficult for people to fill out.

Here is the form. When you are ready, begin with “Start Here”.

INTERVIEWER: For Protocol “A”, direct respondents to complete the Start Here box, Person 1, Person 2 (if applicable), and Housing questions 1-11. Note perceived difficulty and errors experienced by the respondent. Discuss these issues during the cognitive interview and probe on translation issues. Sign of confusion or difficulty (reading but not answering, asking questions, etc.) Skipped questions intentionally or unintentionally Didn’t follow instructions (checking wrong boxes, following wrong path, etc.) If R asks questions, encourage R to complete the form on his or her own. Say: What would you do if I weren’t here? End this section if the participant becomes upset or is unable to complete the form on his or her own after 15 minutes.

SECTION II. INTERVIEW OBSERVATIONS

A. Introduction and Practice

Now I have some specific questions about how you understood some of the words and phrases that were used in this form. I will first point out these words and phrases to you and let you read them over before I ask about your thoughts. I really want to hear all of your opinions and reactions.

Let’s do a short practice before we begin. Please look at this question:

How many windows are there in the house or apartment where you live?

What would your answer be?

Now, tell me everything you were thinking about as you went about answering this question.

INTERVIEWER: Probe as necessary.

• How did you come up with that answer?

• What do you think they mean by the term “window”?

• What is this question asking in your own words?

Thank you. Don’t hesitate to speak up whenever something seems unclear, is hard to answer, or doesn’t seem to apply to you. Again, this is NOT any kind of test and there are no right or wrong answers. Just tell me what you think of the translation.

Okay, now let’s get to the form that you filled out.

SECTION III. COGNITIVE INTERVIEW

B. Start Here Box

In this question, “how many people are living or staying at this address”, there are 4 bulleted instructions. Please read them aloud to me and then I will ask you some questions.

INTERVIEWER: WHILE R IS READING, DOCUMENT DIFFICULTIES OBSERVED PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

• PROBE IF APPROPRIATE: “I noticed when you read those instructions…, you were (laughing/ hesitating/pausing), what were you thinking at that point?”).

• Do those instructions sound natural to you in [Chinese/Korean]?

You wrote that there are _#_ people at your address. According to these instructions, in your household, who should have been included?

According to the instructions, who should not have been included?

C. Housing Section (Instruction and Questions 1-11)

Instruction

Okay, let’s look at this instruction. What do you think they mean by “the address on the mailing label” here in this question?

Housing Question 1

What do you think Question #1 is asking?

Was it easy or difficult for you to choose your answer?

What do you think this phrase “Include all apartments, flats, etc., even if vacant” is saying? Does it sound natural in <language?>

Let’s look at these answer categories more closely. You chose ____. How did you choose that answer?

What do you think they mean by “a one-family house detached from any other house”? Could you give me some examples?

What do you think they mean by “a one-family house attached to one or more houses”? Could you give me some examples?

Housing Question 3

You are doing great. Let’s take a look at Question 3. What does “PERSON 1 (listed on page 2)” mean to you?

Routing Instruction A

Now please read the phrase right above Question 4 to yourself. What do you think these phrases are asking you to do?

Did you notice them while filling out the form?

INTERVIEWER: IF R DID NOT FOLLOW THE ROUTING INSTRUCITON, DETERMINE THE CAUSE. Tell me how you decided to answer Question #. Tell me how you decided to skip Question #.

Housing Question 4

Let's look at Question 4. IF R SKIPPED QUESTION 4, SAY: That’s okay. Let’s look at it now.

What is this question asking in your own words?

Do you know how big 1 acre is?

Housing Question 5

Please read Question 5 to yourself. IF R SKIPPED QUESTION 5, SAY: That’s okay. Let’s look at it now.

What do you think this question is asking?

What does the term “property” mean to you here?

When you read “IN THE PAST 12 MONTHS”, what period of time were you thinking about?

Housing Question 6

Okay, let's look at Question 6. IF R SKIPPED QUESTION 5, SAY: That’s okay. Let’s look at it now.

What do you think they mean here?

What are some examples of the term “business”?

Housing Question 7a

You wrote _#_ in Question 7a. How did you come up with your answer? What did you include?

What does the phrase “separate rooms” mean to you here?

POINT TO ITALIC TEXT BELOW QUESTION 7a, AND SAY: This explains what they mean by “separate rooms”. Could you tell me in your own words what it is saying? Did you find something confusing?

INTERVIEWER: ADMINISTER PROBES AND RECORD ANSWERS IN TABLE BELOW. Here are some terms that describe part of the home: porches, foyers, halls, or unfinished basement. Have you heard of [TERM] before? What do you think they mean by [TERM] here? SHOW PICTURE: This is a picture of a [TERM]. Can you think of a better way to say this phrase in your language?

Term	PORCH	FOYER	HALL	UNFINISHED BASEMENT
Heard before? (y/n)
What it is:
Better translation:	[IF NECESSARY: It’s an area adjoining the entrance to a home. It is covered and usually has its own roof.]	[IF NECESSARY: It’s an entrance hallway, like a lobby in a building.]	[IF NECESSARY: It’s a passageway to a room.]	[IF NECESSARY: It’s a basement that has no furnishings.]

Housing Question 7b

You wrote _#_ in Question 7b. How did you come up with your answer? What did you include?

POINT TO ITALIC TEXT BELOW QUESTION 7b, AND SAY: This explains what they mean by “bedrooms”. Could you tell me in your own words what it is saying? Did you find something confusing?

What does the phrase “efficiency/studio apartment” mean to you here?

SHOW PICTURE AND SAY: Here is a picture of an “efficiency/studio apartment”. It is a self-contained apartment that combines living room, bedroom and kitchen into a single unit, barring a bathroom. Can you think of a better way to describe this type of home in your language?

Housing Question 8

Now I’d like you to read question 8 and its answer categories aloud to me then I will ask some questions.

INTERVIEWER: WHILE R IS READING, DOCUMENT DIFFICULTIES OBSERVED PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

• PROBE IF APPROPRIATE: “I noticed when you read this question…, you were (laughing/ hesitating/pausing), what were you thinking at that point?”).

• Does the question here sound natural to you in <language>?

INTERVIEWER: DISCUSS DIFFICULTIES WITH QUESTION LAYOUT.

Housing Question 9

Okay, let’s take a look at Question 9. How did you come up with the answer?

What do you think this question is asking?

Housing Question 10

In Question 10, tell me how you chose your answer. Did you find something confusing?

Are there any terms that you are not familiar with?

Housing Question 11a-c

Was it difficult for you to answer some of these questions here? Which ones?

IF DIFFICULT: Could you tell me more?

In Question 11a, how did you decide what to answer?

What about Question 11b?

When you read “LAST MONTH” in Question 11a, what period of time were you thinking about?

There is an answer category that says “included in rent or condominium fee”. What do you think it means here?

In Question 11c, how did you figure the amount?

What about Question 11d?

When you read “IN THE PAST 12 MONTHS” in Question 11c, what period of time were you thinking about?

There is an answer category that says “no charge”. What do you think it means here?

Cover Page instructions below title

Please read the first part of these phrases aloud to me and then I will discuss it with you.

INTERVIEWER: WHILE R IS READING, DOCUMENT DIFFICULTIES OBSERVED (E.G. CONFUSING TERMS, MISPRONUNCIATION, ETC.)

• PROBE IF APPROPRIATE: “I noticed when you read those instructions…, you were (laughing/ hesitating/pausing), what were you thinking at that point?”).

• According to this phrase, what do you think it is telling you?

• Do those instructions sound natural to you in <language>?

Please read the second part of these phrases aloud to me and then I will discuss it with you.

INTERVIEWER: WHILE R IS READING, DOCUMENT DIFFICULTIES OBSERVED (E.G. CONFUSING TERMS, MISPRONUNCIATION, ETC.)

• PROBE IF APPROPRIATE: “I noticed when you read those instructions…, you were (laughing/ hesitating/pausing), what were you thinking at that point?”).

• According to this phrase, what do you think it is telling you?

• Do those instructions sound natural to you in <language>?

D. Debriefing and Closing

Finally, I have just a few more questions.

1. Have you ever participated in any cognitive interviewing like the one we did today?

1 Yes  Tell me more about it. By what organization?

2 No

2. Have you filled out a survey questionnaire in your country of origin?

1 Yes  Tell me more about it. Was it for market research, demographic surveys, or something else?

2 No

3. How about in the U.S.? Have you filled out a survey questionnaire in the U.S.?

1 Yes  Tell me more about it. Was it for market research, demographic surveys, or something else?

2 No

4. Have you ever participated in any survey interview before? For example, instead of your filling out the answers to questions on a survey questionnaire like today, the survey questions were read to you by an interviewer? This can be done in person or over the phone.

1 Yes  Tell me more about it. Was it by phone or in person? What was it like? By what organization?

2 No

5. Think about your family members, friends and neighbors who speak [Chinese/Korean] primarily. Have you heard that they have ever participated in any survey interview before?

1 Yes

2 No

INTERVIEWER: MAKE ADDITIONAL NOTES HERE.

6. INTERVIEWER: PROBE FOR THE OBSERVED DIFFICULTIES IN SECTION II THAT

YOU HAVE NOT DISCUSSED WITH THE RESPONDENTS. I noticed when you were filling out the questionnaire, you ….., what were you thinking at that point?

7. Are there any phrases or words in this form that did not sound natural to you but we have not discussed? What about things that would not be appropriate for you or someone else who speaks [Chinese/Korean]? For example, offensive or strange wording, etc.

8. [TIME PERMITTING] Let’s look at the Start Here box again. I’m going to read a brief description of a Mr. Yang’s situation: Mr. Yang’s daughter goes to school in another state. She lives in the dormitory during the academic year but spends summer vacation with Mr. Yang. According to the DO NOT INCLUDE instructions, should Mr. Yang count his daughter in this question if he is filling it out during the summer? Why or why not?

9. Do you have any final comments or questions?

I want to thank you very much for your participation. I will now give you $40 and I will ask you to sign a receipt form verifying that you received the money.

INTERVIEWER: TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT.

End Time: ____________ AM / PM

5

RTI Project 0209182.017