Various Demographic Pretesting Activities

Cognitive Testing of Translations of ACS Supporting Materials

in Multiple Languages – Project C2

Version 1

Participant ID # __ __ __ __ __ __ __ __ (Interviewer’s initials / 3-digit sequential # / Location)

Date: ____ /____/_______

Interviewer: _______________________________________

Location (circle one): DC/MD/VA IL NC Other: __ __

Target Language (circle one): Chinese Korean Russian Vietnamese

Tested Forms: Contact Letters Set 1:

Better Understanding Letter (ACS-613B) – Test and control

CATI Refusal Letter (ACS-15(L)) – Test

Letter Order: Test-Control Control-Test

Show card set: A B

START TIME: ____________ AM / PM

SECTION I: INTRODUCTION

PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT

1. INFORMED CONSENT: Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. In addition, the back page asks for your permission to have this session audio recorded. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.

PARTICIPANT READS AND SIGNS FORM. PROVIDE A COPY OF THE FORM TO THE PARTICIPANT

Here is a copy of this form for you to keep.

B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the tape recorder.

TEST TAPE RECORDER AND TURN IT ON AGAIN AFTER TEST

Now, I would like to confirm on tape that it is fine with you to record this interview.

C. BRIEFLY EXPLAIN THE SURVEY AND PROCEDURES USING THE FOLLOWING SCRIPT:

• The Census Bureau counts population in the U.S and also conducts various kinds of surveys. They routinely encounter non-English speaking households and therefore would like to have the letters available in other languages besides English.

• The Census Bureau has hired us to see how well the <TARGET LANGUAGE> versions work. We want to hear your thoughts and opinions on these materials. For example, we might ask you to compare two similar messages and tell us which one you like better.

• There are no right or wrong answers. We are simply trying to make sure the content of these materials is clear and easy for most people to understand.

• In a few minutes, we will ask you to read the materials as if you were selected to participate in the American Community Survey.

• Your participation in this review of the materials is very important because it will help the Census Bureau in revising these materials.

• Any questions before we begin?

SECTION II – A: “BETTER UNDERSTANDING” LETTER (ACS-613B)

CHECK PROTOCOL COVER AND GIVE CONTROL OR TEST VERSION OF THE BETTER UNDERSTANDING LETTER (DEPENDING ON THE PREASSIGNED ORDER)

A. FIRST READING: SILENT READING BY PARTICIPANT

HAND THE LETTER TO THE PARTICIPANT

When an address is selected to be in the survey, a letter like this one is mailed to respondents in order to provide a better explanation of the American Community Survey.

Please read this silently to yourself and let me know when you are finished. I will then ask you some questions about the letter.

B1. DEBRIEFING:

TAKE THE LETTER FROM THE PARTICIPANT

• What did you think this letter was about?

• What were the main messages in the letter? (Anything else)?

• After reading this letter, do you think you can explain to someone else what it is about? Please tell me what you would say in your own words.

• Does this letter ask you to do anything? What is that?

• Do you know who is conducting the survey?

• Do you know how the American Community Survey helps you and your community?

• Is participation in this survey required by law?

• Did anything in this letter sound strange or awkward to you?

GIVE THE SECOND VERSION OF THE LETTER (TEST OR CONTROL, DEPENDING ON PREASSIGNED ORDER)

Now please read this letter and I will ask you some questions.

B2. DEBRIEFING:

TAKE THE LETTER FROM THE PARTICIPANT

• What were the messages in this letter? (Anything else?)

• How was this letter different from the first one? (Anything else?)

• Was there anything new in this letter? (Anything else?)

• Did anything in this letter sound strange or awkward to you?

• Which letter did you like better? Why?

• Which letter is more appropriate for your culture? Why?

• If you had to guess, which letter would you say was directly translated from English into your language (as opposed to being written by a speaker of your language)? Why do you think so?

SAMPLE PROBES TO USE IN QUESTIONS ABOVE: “What was it about the letter that made you think that?” or “Do you remember which part of the letter told you that? Can you show me where?”

C. SECOND READING/DEBRIEFING

IN THIS SECTION: THE PARTICIPANT READS ALOUD ONE MESSAGE/PARAGRAPH AT A TIME FROM SHOWCARD SET (A OR B, DEPENDING ON PREASSIGNED ORDER) INTERVIEWER OBSERVES PARTICIPANT INTERVIEWER DOCUMENTS OBSERVATIONS INTERVIEWER PROBES INTERVIEWER ASKS SCRIPTED QUESTIONS FOR EACH MESSAGE ALL STEPS ARE REPEATED FOR EACH MESSAGE/PARAGRAPH AFTER ALL PARAGRAPHS, ADMINISTER “GENERAL PROBES”

• 1^nd Paragraph: I will present you with several show cards with two messages on each. As we go, please read both messages aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the m …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO MESSAGES ON SHOWCARD]:

• Survey Sponsor: [ORIGINAL MESSAGE: The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sections 141, 193, and 221, of the United States Code, and response to this survey is required by law. ]

PROBE: What do you think they mean by each statement?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

PROBE: Have you heard of Title 13? Did you find its mention useful?

• 2^nd Paragraph: Please read these messages aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the message …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO MESSAGES ON SHOWCARD]:

• Survey Topic Message: [ORIGINAL MESSAGE: The American Community Survey contains questions about household characteristics including topics such as education, employment, and housing.]

PROBE: What do you think they mean by each statement?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

PROBE: What do you think the survey is about?

• Survey Purpose Message:[ORIGINAL MESSAGE: It produces critical, up-to-date information that is used to meet the needs of communities across the United States. For example, results from this survey may be used to decide where new schools, hospitals, and fire stations are needed.

PROBE: What do you think they mean by each statement?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

• Data Uses Message:[ORIGINAL MESSAGE: Survey data are used by federal, state, and local governments to make decisions and to develop programs that will provide health care, education, and transportation services that affect you and your community. This survey information helps communities plan for emergency situations that might affect you and your neighbors.]

PROBE: What do you think they mean by each statement?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

• 3^rd Paragraph: Please read these messages aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the messages …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO MESSAGES ON SHOWCARD]:

• Confidentiality: [ORIGINAL STATEMENT: We want to emphasize that any information you give our representative will be kept confidential. By law, the Census Bureau cannot publish or release to anyone any information that would identify you or your household (Title 13, Section 9).]

PROBE: What do you think they are saying in these messages?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

PROBE: What does the term “confidential” mean to you in this sentence?

• Statistical Purposes: [ORIGINAL STATEMENT: The information you provide can be used only for statistical purposes.]

PROBE: What do you think they meant by this statement?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

PROBE: What does the phrase “statistical purposes” mean to you in this sentence?

• 4^th Paragraph: Please read the following messages aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the messages …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO MESSAGES ON SHOWCARD]:

• Next Step Message: [ORIGINAL MESSAGE: A member of our staff will contact you again in a few days.]

PROBE: What do you think they meant by these statements?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

• Importance of Participation Message: [ORIGINAL MESSAGE: Your participation and cooperation are important to the success of the survey. Your response does make a difference.]

PROBE: What do you think they meant by these statements?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

• Where to Find Information Message: [ORIGINAL MESSAGE: If you have any questions, call us at the phone number provided at the bottom of this letter. We will be pleased to help you.]

PROBE: What do you think they meant by these statements?

PROBE: How are these statements different?

PROBE: Which one do you like better? Why?

PROBE: If you had to guess, which message is a direct translation from English into your language and which was written by someone from your country? Why would you say so?

GENERAL PROBES ABOUT THE “BETTER UNDERSTANDING” LETTER (ACS-613B): GIVE BOTH LETTETS BACK TO PARTICIPANT

Now I have some general questions about this letter.

• Do the words, terms, and ideas used in any of the two letters sound right or appropriate in your language or to your culture? Are there any that do not seem right or appropriate? Are there any terms that have a negative connotation in your language?

• Are there any words or ideas here that you found hard to understand? How would you change them?

SECTION II – B: “CATI REFUSAL” LETTER (ACS-15 (L))

A. FIRST READING: SILENT READING BY PARTICIPANT

HAND THE LETTER TO THE PARTICIPANT

Now imagine you just received this letter in the mail.

This one-page letter is mailed to respondents when they have refused to participate in the American Community Survey telephone interview.

Please read this silently to yourself and let me know when you are finished. I will then ask you some questions about the letter. Please pretend you did not see the previous letter.

B. DEBRIEFING:

TAKE THE LETTER FROM THE PARTICIPANT

When you answer my questions, please think only about this letter and try to ignore the previous letter:

• What were the messages in this letter? (Anything else?)

• How would you explain to someone else what this letter is about?

• Does this letter ask you to do anything? What is that?

• Did anything in this letter sound strange or awkward to you?

GIVE LETTER BACK TO PARTICIPANT

• Was there anything about the letter that might encourage you to participate if you were selected for this survey? What was that?

• Was there any part of the letter that caused you to become concerned? What was that/Why? Do you remember where that was in the letter?

• What would you think if you received this letter in the mail?

• Is participation in this survey required by law?

• Was there anything in this letter you found inappropriate for your culture?

SAMPLE PROBES TO USE IN QUESTIONS ABOVE: “What was it about the letter that made you think that?” or “Do you remember which part of the letter told you that? Can you show me where?”).

C. SECOND READING/DEBRIEFING

IN THIS SECTION: THE PARTICIPANT READS ALOUD ONE PARAGRAPH AT A TIME INTERVIEWER OBSERVES PARTICIPANT INTERVIEWER DOCUMENTS OBSERVATIONS INTERVIEWER PROBES INTERVIEWER ASKS SCRIPTED QUESTIONS ALL STEPS ARE REPEATED FOR EACH PARAGRAPH AFTER ALL PARAGRAPHS, ADMINISTER “GENERAL PROBES”

• 1^st Paragraph: Please read the first paragraph aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the paragraph …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO STATEMENT]:

• Mandatory Survey Message: [ORIGINAL MESSAGE: The U.S. Census Bureau is conducting this survey under the authority of Title 13, Sections 141, 193, and 221, of the United States Code, and response to this survey is required by law.]

PROBE: What do you think they meant by this statement?

• Survey Concerns Message: [ORIGINAL MESSAGE: I understand that you have some concerns about participating in this survey, but your household’s participation is important to the success of this survey.

PROBE: What does this sentence say about your household’s participation in this survey?

• 2^nd Paragraph: Please read the second paragraph aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the paragraph …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO STATEMENT]:

• Survey Purpose Message:[ORIGINAL MESSAGE: The primary goal of this survey is to provide information each year about the social, economic, and housing characteristics of the United States.]

PROBE: What kind of information do you think the survey produces? Is this important or not? Why?

• Importance of Participation Message: [ORIGINAL MESSAGE: Your participation helps provide the information needed by your community, county, state, and nation, to plan and fund programs at all levels]

PROBE: What do you think is meant by these sentences?

PROBE: What kind of information do you think they are referring to?

• Data Uses Message: [ORIGINAL MESSAGE: The American Community Survey will provide communities annually updated, detailed information previously available only when the Census Bureau conducted a census every 10 years.]

PROBE: What do you think they meant by this sentence?

PROBE: Do you think this is important? Why?

• 3rd Paragraph: Please read the second paragraph aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the paragraph …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO STATEMENT]:

• Confidentiality and Statistical Purposes message: [ORIGINAL MESSAGE: We want to emphasize that any information you give our representative will be kept confidential. By law, the Census Bureau cannot publish or release to anyone any information that would identify you or your household (Tile 13, Section 9). The information you provide can be used only for statistical purposes.]

PROBE: What does the term “confidential” mean to you in this sentence?

PROBE: What does the phrase “statistical purposes” mean to you in this sentence?

• 4^th Paragraph: Please read the fourth paragraph aloud to me and then I will ask you some questions.

WHILE R IS READING  OBSERVE AND DOCUMENT BELOW: PLACES WHERE TEXT/SYNTAX SEEMS DIFFICULT FOR RS TO READ AND/OR COMPREHEND. ANY TERMS/CONCEPTS MENTIONED BY RS AND THE ISSUES THEY HAVE WITH THOSE TERMS/CONCEPTS. ANY MISPRONUNCIATIONS. (THIS WILL SERVE AS A CLUE FOR DIFFICULT OR UNKNOWN TERMS/CONCEPTS.)

PROBE IF APPROPRIATE: “I noticed when you read the paragraph …, you were laughing/ hesitating/pausing, what were you thinking at that point?”).

INTERVIEWER PROBING [POINT TO STATEMENT]:

• Second Importance of Participation Message: [ORIGINAL MESSAGE: We hope that you will participate in this survey to help us improve the information that you and others provide about your community. ]

PROBE: How do you think the information you provide matters?

PROBE: Would you be more likely to participate in this survey after reading this letter? Why?

• Where to Call with Questions Message: [ORIGINAL MESSAGE: If you have any questions, call us at 1-888-817-2153.

PROBE: How likely are you to call this number? Does this letter leave any unanswered questions or raise concerns?

GENERAL PROBES ABOUT THE “CATI REFUSAL” LETTER (ACS-15(L)):

Now I have some general questions about this letter.

• Do the words, terms, and ideas used in this letter sound right or appropriate in your language or to your culture? Are there any that do not seem right or appropriate? Are there any terms that have a negative connotation in your language?

• Are there any words or ideas here that you found hard to understand? How would you change them?

• If you had to write this letter, how would you organize the messages in it? Why?

SECTION III: SURVEY EXPERIENCE

Now I have a few additional questions to ask you concerning your experience with survey participation.

• How did you hear about this interview?

• Have you ever participated in any cognitive interviews like the one we did today?

• IF YES

  • Can you tell me more about it? What kind? By what organization? Was it in the U.S. [IF INTERVIEW NOT IN ENGLISH: or your home country]?

• Have you ever participated in any survey interview? Was it by phone or in person?

• IF YES

  • What kind? Was it market research, opinion research, or a demographic survey? Was it in the U.S. [IF INTERVIEW NOT IN ENGLISH: or your home country]?

• Have you ever filled out a survey questionnaire?

• IF YES

  • What kind? Was it market research, opinion research, or a demographic survey? Was it in the U.S. [IF INTERVIEW NOT IN ENGLISH: or your home country]?

Finally, I have a question about the letters you just read.

• After reading these letters, would you be likely to participate in the survey? Why/Why not?

SECTION IV: CONCLUSION

1. FINAL COMMENTS:

Do you have any final comments or any questions?

B. CLOSING: I want to thank you very much for participating today. I will now give you $40 as a token of our appreciation. Please sign this receipt form verifying that you received the money.

TURN OFF THE TAPE RECORDER. HAND THE CASH INCENTIVE TO THE PARTICIPANT AND TAKE THE SIGNED INCENTIVE RECEIPT FORM.

TURN OFF THE TAPE RECORDER IF YOU HAVE A MICROPHONE TURN IT OFF TOO HAND PARTICIPANT THE CASH INCENTIVE HAVE PARTICIPANT SIGN RECEIPT FORM COLLECT CONSENT FORM WITH 2 SIGNATURES THANK PARTICIPANT – YOU ARE DONE!!

END TIME: ____________ AM / PM

38