American Community Survey (ACS) Strategic Framework Mailing Materials Cognitive Interviews

Cognitive Testing of ACS Strategic Framework Mail Materials

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM VERSION: ________________

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we are going to look at some of the envelopes, letters, postcards, and the questionnaire that you would get in the mail if your address had been selected to respond to a national survey called the American Community Survey.

[If needed: Because these are example mailings, they may not look quite as nice as they would if you received them in the mail.] We will go through some different pieces of mail one by one. After you’ve finished with one piece of mail, I will hand you the next. I’ll explain each one to you as we go along.

As I give you these mailings, I’m going to ask you to treat each one as you would if it came to your home. Please go through these as you would at home, as if I were not there, but say out loud what you are thinking as you look at the mailings. I want you to note things that you are looking at or reading, things that stand out to you, that you like or dislike, things like that.

I’m also going to ask you some questions about the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how the mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• [HAND RESPONDENT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a respondent, such as that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio (if applicable: and video) recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [RESPONDENT READS FORM THEN SIGNS AND DATES FORM]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF RESPONDENT PROVIDES CONSENT TO RECORD THE SESSION]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think-Aloud

While you are looking at the materials I give you, I’d like you to tell me what’s going through your mind. This could include what information you’re seeing in each material, questions that come to your mind, and reactions you’re having. You do not have to read aloud the mailing materials if you would not do that at home.

No practice of think aloud. Encourage respondents to think aloud as needed throughout interview, especially with first few mailing pieces:

• What are you looking at?

• What are you thinking?

• Remember to tell me what you are thinking about as you look at these mailings.

Observation

Research Questions:

1. Does the use of color, images, and novel structure enhance the appeal of the letters?

2. Does the use of color cause any difficulties? (Readability/believability, ‘official’ government mail)

3. Do the strategies of reducing the amount of text and using plain language in the bodies of the letters improve readability and clarity of letter content?

4. Does including “OFFICIAL U.S. GOVERNMENT MAIL” on the envelope increase “believability”?

5. Do reducing text, using plain language, and the use of novel visual design elements make the letters look too “unofficial”?

6. Does Spanish messaging evoke any reaction?

Okay, let’s get started.

• Make observation notes below (see questions in red text). Note any nonverbal reactions.

• If respondent indicates that they would not look at a piece of mail/would let mail pile up, note that but then ask them to look at it anyway.

• Let the respondent spend no more than 5 minutes looking at each mailing.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings to each other until specified.

• Hand materials to respondents SIDEWAYS, so that it is neither face up or face down.

Mailing 1 - initial letter: Okay, let’s get started. Imagine you received this piece of mail at your house. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does R comment on “OFFICIAL US GOVERNMENT MAIL” on the envelope?

___ Yes ____ No If yes, what did they say? _____________________________

2. Which way does R take materials out of the envelope? ___ Address side up ____ Address side down

3. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise, put a number to indicate the order.

___ Envelope, front ___ Letter, front ___ Instruction card, front

___ Envelope, back ___ Letter, back ___ Instruction card, back

4. Does R comment on whether the mail pieces look legitimate or unofficial?

___ Yes, legitimate ___Yes, unofficial ___ No ___ Not sure

If yes, what piece(s) were they referring to when they commented? (Can be in more than one place):________________________________________________________________________

5. Does R comment on the icon/sidebar/bullet format?

___ Yes ___ No ___ Not sure

6. Does R comment on the coloring used?

___ Yes ___ No ___ Not sure

7. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be in more than one place): _______________________________________

8. Does R comment on the Spanish messaging?

___ Yes ___ No ___ Not sure

If yes, what did they say? __________________________________

9. Does R read letter front- How closely?

___ Did not look ___ Glanced ___ Skimmed ___Read closely

10. Does R read FAQs on back of letter; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Other notes:

Mailing 2 - first reminder letter, pressure seal: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive one week later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R open pressure seal?

___ Address side up ____ Address side down

2. Does R look at both sides (address side and blank side)?

___ Yes ___ No

3. Does R comment on whether the mail pieces look legitimate or unofficial?

___ Yes, legitimate ___Yes, unofficial ___ No ___ Not sure

If yes, what piece(s) were they referring to when they commented? (Can be in more than one place): _______________________________________________________________________

4. Does R comment on whether the mail pieces looks like it belongs with previous mailings?

___ Yes ___ No ___ Not sure

If yes, what did they comment on? __________________________________

5. Does R comment on the icon/sidebar/bullet format?

___ Yes ___ No ___ Not sure

6. Does R comment on the community benefits mentioned in the letter?

___ Yes ___ No ___ Not sure

If yes, what did they comment? : __________________________________

7. Does R comment on the uses of the ACS data?

___ Yes ___ No ___ Not sure

If yes, what did they comment? __________________________________

8. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be in more than one place): ____________________________

9. Does R read content inside pressure seal; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 3 - paper questionnaire package: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive about a week later if you did not respond to the earlier requests [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Does R comment on “OFFICIAL US GOVERNMENT MAIL” on the envelope?

___ Yes ____ No If yes, what did they say? _________________________________

2. Which way does R take materials out of the envelope?

___ Address side up ____ Address side down

3. What materials does R look at, and in what order? Mark “X” if respondent did not look at a material. Otherwise, put a number to indicate the order.

___ Outer envelope, front ___ Letter, front ___ Questionnaire ___ Return envelope, front

___ Outer envelope, back ___ Letter, back ___ Return envelope, back

4. Does R comment on whether the mail pieces look legitimate or unofficial?

___ Yes, legitimate ___Yes, unofficial ___ No ___ Not sure

If yes, what piece(s) were they referring to when they commented? (Can be in more than one place): __________________________________

5. Does R comment on the icon/sidebar/bullet format?

___ Yes ___ No ___ Not sure

6. Does R comment on the coloring used?

___ Yes ___ No ___ Not sure

7. Does R make unprompted comments about the text about civic duty?

___ Yes ___ No ___ Not sure

If yes, what did they comment? : __________________________________

8. Does R make unprompted comments on the line about millions of households/many households completing survey?

___ Yes ___ No ___ Not sure

If yes, what did they comment? __________________________________

9. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be in more than one place): __________________________________

10. Does R comment on the Spanish messaging?

___ Yes ___ No ___ Not sure

If yes, what did they say? __________________________________

11. Does R read letter front- How closely?

___ Did not look ___ Glanced ___ Skimmed ___Read closely

Note any questions R asked/comments R made about this mailing:

Note any comments R had on the back of the letter:

Other notes:

Mailing 4 - postcard: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive about four days later if you received the third mailing. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which side(s) does R look at, and in what order? Mark “X” if respondent did not look at one of the sides. Otherwise, put a number to indicate the order.

___ Front ___ Back

2. Does R comment on whether the mail pieces look legitimate or unofficial?

___ Yes, legitimate ___Yes, unofficial ___ No ___ Not sure

If yes, and if obvious, what were they referring to when they commented? __________________________________

3. [ONLY IN ‘Classic With a Twist’]: Does R comment on the bullet format?

___ Yes ___ No ___ Not sure

4. Does R comment on the coloring used?

___ Yes ___ No ___ Not sure

5. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, what did they comment on: _____________________________

6. Does R read content; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Mailing 5 – final reminder, pressure seal: Thank you. [Take previous mailing back.] Let’s move onto the next piece of mail, which you would receive about two weeks later. [Give respondent mailing.] Please treat it as you would if you were receiving this piece of mail at your home.

OBSERVE (DO NOT READ TO RESPONDENT):

1. Which way does R open pressure seal?

___ Address side up ____ Address side down

2. Does R look at both sides (address side and blank side)?

___ Yes ___ No

3. Does R comment on the icon/sidebar/bullet format?

___ Yes ___ No ___ Not sure

4. Does R comment on the due date?

___ Yes ___ No ___ Not sure

If yes, what did they comment? ____________________________

5. Does R comment on the commitment device?

___ Yes ___ No ___ Not sure

If yes, what did they comment? ____________________________

6. Does R comment on response being required?

___ Yes ___ No ___ Not sure

If yes, where? (Can be in more than one place): ____________________________

7. Does R comment on the Spanish messaging?

___ Yes ___ No ___ Not sure

If yes, what did they say? __________________________________

8. Does R read content inside pressure seal; if so, how closely?

___ Did not look ___ Glanced ___ Skimmed ___ Read closely

Note any questions R asked/comments R made about this mailing:

Other notes:

Probing

Thank you. Now I have a few questions I’d like to ask you about the mailings.

• Only ask probe if not mentioned earlier in the interview.

• Take back each mailing after respondent is finished—do not let them compare the mailings to each other until specified.

Mailing 1 - initial letter:

Research Questions:

1. Does including “OFFICIAL U.S. GOVERNMENT MAIL” on the envelope increase “believability”? (Does this sound like a scam to R? Does R believe that only the government would say this, or is this something they’ve seen other places?)

2. Do changes to mailings intended to increase trust and legitimacy actually convey that to the reader? (i.e., adding FAQs about the survey, highlighting the Census logo, testimonies on back of letters from credible experts, using consistent coloring throughout mailings)

This is the first mailing you received. [Give respondent letter 1.]

1. What do you think you would do after receiving this in the mail?

2. When I first gave you this, what did you notice about it?

3. What do you think is the main point of this mailing?

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. If needed: After getting this letter, would you respond to the survey? Tell me more about that.

6. Is there anything in this letter that would make you more likely to respond to this survey?

7. Is there anything in this letter that would make you less likely to respond to this survey?

8. If needed: Do you think you have to respond to the survey?

9. Did you notice the back of this letter? What do you think about it?

Now let’s take a look at the envelope this letter came in.

10. If R didn’t comment on the wording during observation: Did you notice this wording on this envelope? [Point to ‘Official U.S. Government Mail]

11. Would this make you more or less likely to open up the envelope? If needed: Tell me more about that.

12. If needed: What do you think about this wording?

Mailing 2 - first reminder pressure seal:

Research Questions:

1. Does highlighting community benefits resonate with the reader?

2. Are they turned off by some benefits or conversely are they inspired by some benefits (enough so to feel that their response is important)?

3. Does highlighting the use of ACS data by nonprofit organizations resonate with the reader?

4. [Classic w/a Twist and To The Point] Does including a “Reminder…” line in the top row of the second letter make the letter seem more friendly... less demanding?

Thank you. [Take back previous mailing.] This is the second mailing you received. [Give respondent mailing.]

1. What do you think you would do after receiving this in the mail?

2. When I first gave you this, what did you notice about it?

3. Did this mailing seem the same as the first mailing, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. If you hadn’t responded to the survey already, after getting this letter do you think you would you respond to the survey? (If needed: Tell me more about that.)

6. ONLY IF CLASSIC WITH A TWIST OR TO THE POINT: Please read the first line of the letter that says “Reminder.” Did you notice this sentence when reading the letter earlier in our interview? What do you think about this reminder?

   1. If needed: Do you think this reminder is helpful?

   2. If needed: Does the reminder make you more or less likely to reply? (If needed: Tell me more about that.)

7. If needed: Did you notice anything in the letter about why you should complete the survey? Tell me more that.

8. If R mentioned the benefits or use of ACS data: What do you think about that?

9. If participant did not mention the benefits or use of ACS data: [POINT TO THE APPLICABLE SECTIONS ON BENEFITS AND USES OF DATA] Please read this section(s). What do you think about this part of the letter?

10. Do you think the ACS would be used to benefit your community? (If needed: Tell me more about that.)

11. Is there anything in this letter that would make you more likely to respond to this survey?

12. Is there anything in this letter that would make you less likely to respond to this survey?

Mailing 3 - paper questionnaire package:

Research Questions:

1. Does including “OFFICIAL U.S. GOVERNMENT MAIL” on the envelope increase “believability”?

2. Does using messages about civic duty resonate with the reader?

3. Does reading about “jury duty” and “paying taxes” NOT resonate? (Are they irritating?)

4. Does using normative messaging resonate with the reader?

Thank you. [Take back previous mailing.] This is the third mailing you received. [Give respondent mailing.]

1. What do you think you would do after receiving this in the mail?

2. When I first gave you this, what did you notice about it?

3. Is there anything in this letter that would make you more likely to respond to this survey?

4. Is there anything in this letter that would make you less likely to respond to this survey?

5. Did this third mailing seem the same or different from any of the two previous mailings? (If needed: Tell me more about that.)

6. What would you have done next after reading it? (If needed: Tell me more about that.)

7. What do you think about this sentence here? [POINT TO SENTENCE THAT TALKS ABOUT “CIVIC DUTY”]

8. What do you think about this sentence here? [POINT TO SENTENCE ABOUT ‘MANY HOUSEHOLDS’(CWT & TTP) OR MILLIONS OF HOUSEHOLDS (FL)]

Mailing 4 - postcard:

Thank you. [Take back previous mailing.] This is the fourth mailing you received. [Give respondent mailing.]

1. What do you think you would do after receiving this in the mail?

2. When I first gave you this, what did you notice about it?

3. Is there anything in this postcard that would make you more likely to respond to this survey?

4. Is there anything in this postcard that would make you less likely to respond to this survey?

5. Did this mailing seem the same as any of the three previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

6. What would you have done next after reading it? (If needed: Tell me more about that.)

Mailing 5 - final reminder pressure seal:

Research Questions:

1. Does including deadline/due dates help to increase the message of urgency for the reader?

2. Do Rs think they would be more likely to respond given a firm date?

3. How does implementing a commitment device affect self-response?

4. Would this commitment device make R think he/she should mail the letter in?

Thank you. [Take back previous mailing.] This is the fifth mailing you received. [Give respondent mailing.]

1. What do you think you would do after receiving this in the mail?

2. When I first gave you this, what did you notice about it?

3. Is there anything in this letter that would make you more likely to respond to this survey?

4. Is there anything in this letter that would make you less likely to respond to this survey?

5. Did this fifth mailing seem the same as any of the four previous mailings, or did it seem different?

(If needed: Tell me more about that.)

6. What would you have done next after reading it? (If needed: Tell me more about that.)

7. Did you notice a due date? What do you think about it?

8. If needed: What do you think about all of the different options for completing the survey? Do you think that would make you more or less likely to complete the survey, or would it not affect your decision to complete the survey?

Side by side of all treatments:

Research Questions:

1. What stands out to respondents in a side-by-side comparison of all treatments?

2. Does color cause any difficulties? (Readability/believability, ‘official’ government mail)

Initial Letter:

Now I’m going to show you a few different versions of the first letter you looked at today [Place all four versions of mailing 1 in front of the respondent in a line].

1. What do you see as the difference, if any, between these four letters?

2. Which version of the letter do you prefer, if any? Can you tell me more about that

3. What do you think of the color used in these letters?

4. If needed: Did you find the green text easy or difficult to read? Can you tell me more about that?

5. If needed: Does the use of color look like ‘official’ government mail? Can you tell me more about that?

6. [IF NOT ADMINISTERED TO THE POINT DESIGN] What do you think of these column headers? [POINT TO COLUMN HEADERS IN TO THE POINT].

7. [IF NOT ADMINISTERED FRESH LOOK DESIGN] What do you think of this? [POINT TO SIDEBAR IN FRESH LOOK].

8. If needed: What do you like about it, if anything? What do you dislike about it, if anything? Can you tell me more about that?

9. [IF NOT ADMINISTERED CLASSIC WITH A TWIST DESIGN] What do you think of the icons in this version? [POINT TO ICON IN CLASSIC WITH A TWIST].

10. If needed: What do you like about the icons, if anything? What do you dislike about the icons, if anything? Can you tell me more about that?

Postcard: Thank you. [Take back the four letters.] Now let’s look at four different versions of the postcard? [Place all four versions of postcard in front of the respondent in a line].

1. Is one version of the postcard easier for you to read than the others? If yes: What makes it easier to read?

2. Which version of the postcard do you prefer, if any? Please tell me more about that?

3. Is there anything in any of the postcards that would make you more likely to respond to this survey?

4. Is there anything in any of the postcards that would make you less likely to respond to this survey?

Thank you. [Take back the four postcards]

Debriefing

I have just a few more questions for you to wrap up.

1. In general, how likely would you be to read these materials if they came in the mail to you?

2. Was there a specific piece of mail we looked at today that stood out to you for any reason?

If needed: Which piece? Why did that piece stand out to you?

3. If you had received this mail, do you think you would complete the American Community Survey?

If needed: Tell me more about that.

4. Is there anything we could change about the letters, or postcards, questionnaire, or even the envelopes that would make people more likely to respond to this survey?

5. Was there anything else that we haven’t talked about that you would like to mention?

Great, thank you very much. Those are all of the questions that I have for you today.

Thank you very much for your participation. [TURN OFF THE RECORDER.]

Here is an envelope containing your $40, and a voucher form that verifies I gave you the money.

Please complete the information in the highlighted areas and sign and date.

[HAND THE CASH INCENTIVE TO THE RESPONDENT AND COLLECT VOUCHER FORM.]

12