2020 American Community Survey (ACS) Revised Mailing Materials Cognitive Interviews

Cognitive Testing of ACS 2020 Mail Materials

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM

Introduction

DO NOT MENTION ACS OR DECENNIAL UNTIL PROMPTED TO DO SO.

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we’re going to look at some of the items that might be mailed to you in 2020. We are testing mail for 2020 now so that we have time to plan our mailings in advance. We will go through some different pieces of mail one by one. After you’ve finished with one piece of mail, I will hand you the next. I’ll explain each one to you as we go along.

As I give you these mailings, I’m going to ask you to treat each one as you would if it came to your home. Please go through these as you would at home, as if I were not there, but say out loud what you are thinking as you look at the mailings.

I’m also going to ask you some questions about the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how Census Bureau mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• [HAND RESPONDENT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a respondent, such as that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio (if applicable: and video) recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [RESPONDENT READS FORM THEN SIGNS AND DATES FORM]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think-Aloud

While you are looking at the materials I give you, I’d like you to tell me what’s going through your mind. This could include what information you’re seeing in each material, questions that come to your mind, and reactions you’re having.

No practice think aloud. Encourage respondents to think aloud as needed throughout interview, especially with first few mailing pieces:

• What are you looking at?

• What are you thinking?

• Remember to tell me what you are thinking about as you look at these mailings.

Set 1: Observation

Research Questions:

1. Did respondents understand that the ACS was different from the decennial census?

If so, which component(s) of the mail messaging succeeded in conveying that message?

2. Did respondents understand that they were required to complete both the ACS and the census?

If so, which component(s) of the mail messaging succeeded in conveying that message?

3. Did respondents notice and understand the language on the flap of the 3rd mailing envelope?

4. Did the FAQs answer the questions that respondents had?

Okay, let’s get started.

• Make observation notes below (see questions in red text). Note any nonverbal reactions.

• If respondent indicates they would not look at a piece of mail/would let mail pile up, note that but then ask them to look at it anyway.

• Let the respondent spend no more than 3 minutes looking at each mailing.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings to each other until specified.

Mailing 1, Decennial #1 – letter: In mid-March 2020, you would receive this piece of mail. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

2. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise put a number to indicate the order.

___ Envelope, front ___ Envelope, back ___ Letter, front ___ Letter, back

___ Language assistance sheet, front ___ Language assistance sheet, back

3. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

5. Does R read letter front; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

6. Does R read FAQs on back of letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Other notes:

Mailing 2, Decennial #2 – letter: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

2. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise put a number to indicate the order.

___ Envelope, front ___ Envelope, back ___ Letter, front ___ Letter, back (blank)

3. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

5. Does R read letter front; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 3, ACS #1 – letter: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive about a week later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

2. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise put a number to indicate the order.

___ Envelope, front ___ Envelope, back ___ Letter, front ___ Letter, back
___ FAQ Brochure ___ Instruction card, front ___ Instruction card, back

___ Language brochure

6. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

7. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

3. Does R read letter front; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

4. Does R read FAQs on back of letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Note any comments R had on envelope flap:

Other notes:

Mailing 4, Decennial #3 – postcard: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does R look at both sides?

___ Yes ___ No

2. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

3. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R read content; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 5, ACS #2 – first pressure seal: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R open pressure seal?

___ Address side up ____ Address side down

2. Does R look at both sides (address side and blank side)?

___ Yes ___ No

3. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

5. Does R read content inside pressure seal; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Set 1: Neutral Probing

Thank you. Now I have a few questions I’d like to ask you about the mailings.

• Only ask probe if not mentioned during think aloud.

• If timing is a concern, spend less time on decennial materials than on ACS materials.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings to each other until specified.

Mailing 1, Decennial #1 – letter:

This is the first mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. What would you have done next after reading it? (If needed: Tell me more about that.)

Mailing 2, Decennial #2 – letter:

Thank you. [Take back previous mailing.] This is the second mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as the first mailing, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

Mailing 3, ACS #1 – letter:

Thank you. [Take back previous mailing.] This is the third mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

Mailing 4, Decennial #3 – postcard:

Thank you. [Take back previous mailing.] This is the fourth mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

Mailing 5, ACS #2 – first pressure seal:

Thank you. [Take back previous mailing.] This is the fifth mailing you received. [Give respondent mailing.]

1. If needed: When I first gave you this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

Set 2: Observation

Thank you. Now I’m going to ask you to switch gears a bit and go back to looking at some things you might receive in the mail if you had not responded to the earlier pieces of mail.

• Make observation notes below (see questions in red text). Note any nonverbal reactions.

• If respondent indicates they would not look at a piece of mail/would let mail pile up, note that but then ask them to look at it anyway.

• Let the respondent spend no more than 3 minutes looking at each mailing.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings to each other until specified.

Mailing 6, Decennial #4 – questionnaire package: In mid-April, 2020, you might receive this piece of mail. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

2. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise put a number to indicate the order.

___ Envelope, front ___ Envelope, back ___ Letter, front ___ Letter, back (blank)

___ Questionnaire ___ Return envelope, front ___ Return envelope, back

___ Language assistance sheet, front ___ Language assistance sheet, back

3. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

5. Does R read letter front; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

6. Does R read FAQs on back of letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Other notes:

Mailing 7, ACS #3 – questionnaire package: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

2. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise put a number to indicate the order.

___ Envelope, front ___ Envelope, back ___ Letter, front ___ Letter, back
___ FAQ Brochure ___ Instruction card, front ___ Instruction card, back

___ Questionnaire ___ Return envelope, front ___ Return envelope, back

3. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

5. Does R read letter front; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

6. Does R read FAQs on back of letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Note any comments R had on envelope flap:

Other notes:

Mailing 8, ACS #4 – postcard: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does R look at both sides?

___ Yes ___ No

2. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

3. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R read content; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 9, Decennial #5 – postcard: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a few days later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does R look at both sides?

___ Yes ___ No

2. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

3. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

4. Does R read content; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 10, ACS #5 – second pressure seal: [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive a couple weeks later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

6. Which way does R open pressure seal?

___ Address side up ____ Address side down

7. Does R look at both sides (address side and blank side)?

___ Yes ___ No

8. Does R comment on whether this is for decennial or ACS?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

9. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be more than one place): __________________________________

10. Does R read content inside pressure seal; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Sets 1 and 2: Directed Probing

Research Questions:

1. Did respondents understand that the ACS was different from the decennial census?

If so, which component(s) of the mail messaging succeeded in conveying that message?

2. Did respondents understand that they were required to complete both the ACS and the census?

If so, which component(s) of the mail messaging succeeded in conveying that message?

3. Did respondents notice and understand the language on the flap of the 3rd mailing envelope?

Was one version of the flap wording more successful than the other in getting respondents to understand that the ACS is different from the census and that they are required to complete both?

4. Did the FAQs answer the questions that respondents had?

What questions did respondents have that weren’t answered by the FAQs?

Thank you. Now I have a few more questions I’d like to ask you about the mailings. It’s okay if you are unsure or do not know the answer. There are no right or wrong answers to these questions. Here are the mailings you saw today. [Give all mailings to respondent; reassemble in envelopes if necessary]

• Only ask probe if not mentioned during think aloud.

• Respondents can compare mailings if they want to but do not instruct them to do so.

Overall

1. If needed: Why would the Census Bureau be sending out these pieces of mail?

   1. If needed: Were these mailings sent for the same survey, or for different surveys? Which one(s)?

   2. If respondent still does not seem to pick up on ACS/2020 distinction: Did you notice that some of these pieces of mail were for the 2020 Census and some were for the American Community Survey? Go to #3.

2. If respondent notices there are both ACS and 2020 mailings on their own:

   1. At what point did you realize some mail was for the American Community Survey and some was for the 2020 Census? Was there a specific mailing, or part of the mailings that made you realize this?

   2. What difference, if any, did you notice between the mail for the 2020 Census and mail for the American Community Survey?

3. If needed: Is responding to the American Community Survey optional, or required?

   1. If respondent says it is required: At what point did you realize responding was required?

4. If needed: Is responding to the 2020 Census optional or required?

   1. If respondent says it is required: At what point did you realize responding was required?

Envelope

I’d like to go back to one of the envelopes you saw earlier. [Give mailing #7 envelope (ACS questionnaire package) to respondent with flap side up.]

1. If needed: Did you notice the wording on the flap of this envelope? [Point to language]

   1. If no: Please take a moment to look it over. [Give respondent a moment to review.]

2. In your own words, what is this saying?

Now I’d like to show you another version of this envelope flap. Please take a moment to look it over. [Give to respondent; give respondent a moment to review.]

1. In your own words, what is this saying? [Point to language]

2. If needed: Which of these envelopes do you think better explains that the 2020 Census and the American Community Survey are different?

3. If needed: [If respondent thinks 2020 and/or ACS are optional: Responding to both the 2020 Census and the American Community Survey is required.] Which of these envelopes do you think better explains that responding to the 2020 Census and the American Community Survey is required?

FAQs

Let’s go back to one of the letters you saw earlier. [Give mailing #3 letter (first ACS mailing) to respondent.]

1. If needed: Did you notice the back of this letter?

   1. If no: Please take a moment to look it over. [Give respondent a moment to review.]

2. In your own words, what is this saying? [Gesture to back of letter]

3. Are there any questions that you think people might have that aren’t answered here?

Now I’d like you to look back at another one of the letters. [Give mailing #7 letter (ACS questionnaire package) to respondent.]

1. If needed: Did you notice the back of this letter?

   1. If no: Please take a moment to look it over. [Give respondent a moment to review.]

2. Which version of the back of the letter do you think is clearer, if any? [Gesture to mailing #3 and mailing #7 back of letter]

Flier

Research Question: After seeing the field flier, did respondents understand that the ACS was different than the census?

We are almost done with the interview. I am going to give you one more thing to look at. If you had not responded to the mail we looked at in this interview, you might receive an in-person visit from a Census Bureau interviewer. The interviewer might show you this flier, or leave it at your door if you are not home. Please take a moment to look it over, and remember to tell me what you are thinking. [Give respondent a moment to review.]

• Make observation notes below. Note any nonverbal reactions.

• Let the respondent spend no more than 3 minutes looking at flyer.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which sides does R look at, and in what order? Mark “X” if respondent did not look at a side. Otherwise put a number to indicate the order.

___ FAQ side ___ ACS paper form/online form screenshot side

2. Does R notice that response is required?

___ Yes ___ No

If yes, where? (Can be more than one place): __________________________________

3. Does R read content on FAQ side; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

4. Does R read content on paper form/online form screenshot side; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any comments made or questions R asked about the flier:

Other notes:

Probe:

1. What did you think about this flier?

   1. If needed: Was there anything you think people might find confusing?

2. Are there any questions you think people might have that are not answered by this flier?

   1. If needed: Is there any other information that you think we should put on the flier?

3. What do you think about this line describing the census? [Point to line.]

4. If R only looks at one side:

   1. Please take a moment to look at this side of the flier. [Give respondent a moment to review.]

   2. What did you think about this side of the flier?

Debriefing

Thank you. I have just a few more questions for you to wrap up.

1. Was there a specific piece of mail we looked at today that stood out to you for any reason?

   1. If needed: Which piece? Why did that piece stand out to you?

2. If you had received this mail in 2020, do you think you would complete the 2020 Census, the American Community Survey, both, or neither?

   1. If needed: Tell me more about that.

3. Is there anything we could change about the mailings that would make it clearer that some mail is for the 2020 Census, and some mail is for the American Community Survey?

4. Is there anything we could change about the mailings that would make it clearer that responding to both the 2020 Census and the American Community Survey is required?

5. Was there anything else that we haven’t talked about that you would like to mention?

Great, thank you very much. Those are all of the questions that I had for you today. Thank you very much for your participation. [TURN OFF THE RECORDER.] Here is an envelope containing your $40, and a voucher form that verifies I gave you the money. Please complete the information in the highlighted areas and sign and date. [HAND THE CASH INCENTIVE TO THE RESPONDENT AND COLLECT VOUCHER FORM.]

29