HUP_SmaRT Testing of Decennial Mail Materials Protocol March 2024

SmaRT Testing of Decennial Mail Materials (HUP)

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM EDDM VERSION (Plain/Patriotic):______

LOCATION: _____________

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we are going to look at some of the letters and postcards that you would receive in the mail, and the questionnaire that you would answer online if you were completing the Census.

[If needed: Because these are example mailings, they may not look quite as nice as they would if you received them in the mail.] We will go through several different pieces of mail one by one. After you’ve finished with one piece of mail, I will have you look at the next.

As we go through these mailings, I’m going to ask you to treat each one as you would if it came to your home. Please go through these as you would at home (as if I were not [here/there]), but I’d like you to say out loud what you are thinking as you look at the mailings.

I’m also going to ask you some questions about the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how the mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• IF IN PERSON [HAND RESPONDENT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [RESPONDENT READS FORM THEN SIGNS AND DATES FORM]

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

IF REMOTE

[Participant was emailed consent form]. We emailed you information about the session today and your rights as a participant. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document.

I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think Aloud

I’d like you to answer the questions as you would if you were at home without me there, but with one big difference. What I would like you to do is tell me what you are thinking and feeling as you come up with your answers. 

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[GIVE FEEDBACK ON THE THINK ALOUD THEY DID]

IF REMOTE

Camera

So, just like we practiced in the tech check – I am going to need you to go ahead and turn your camera on by clicking on the camera icon.

[If able to see P] Okay great I can see you. Now we can get started with the task.

[If unable to see P will need to troubleshoot it]

Mailing 1 - initial HUP letter:

Okay, let’s get started.

[Remote] Please open the priority mail envelope. There are a number of pieces of mail in there so be sure you are in a place to lay them out. We will be referring to each one twice in this session. You will see several letters inside. There are 2 groups of letters that are paperclipped together. Please find the one labeled Mailing 1 and take it out. There should be a letter with a handwritten note at the top. What does it say? [Mailing 1]

[In person] [Hand the participant Mailing 1]

Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on two-step instruction?

___ Yes ___ No ___ Not sure

2. Does P comment on “protected by law”?

___ Yes ___ No ___ Not sure

3. Does P comment on why their response matters?

___ Yes ___ No ___ Not sure

4. Does P comment on the data use message – including roads, housing, parks, hospitals?

___ Yes ___ No ___ Not sure

4a. (if yes) were the comments positive or negative or neutral

___ Positive ___ Negative ___ Neutral

5. Does P comment on 10-minute survey length?

___ Yes ___ No ___ Not sure

6. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

7. Does P spontaneously mention wanting to know when it’s due?

___ Yes ___ No ___ Not sure

8. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

Mailing 2 – reminder HUP letter:

[Remote] Please look at Mailing 2. There should be a letter with a handwritten note at the top. What does it say? [Mailing 2]

[In person] [Hand the participant Mailing 2]

A few days after you get the first letter, you would receive this reminder letter. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on two-step instruction?

___ Yes ___ No ___ Not sure

2. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

3. Does P comment on “answers are confidential”?

___ Yes ___ No ___ Not sure

4. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

5. Does P comment on why their response matters?

___ Yes ___ No ___ Not sure

6. Does P comment on the data use message – compliance with anti-discrimination laws?

___ Yes ___ No ___ Not sure

6a. (if yes) were the comments positive or negative or neutral

___ Positive ___ Negative ___ Neutral

7. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

Mailing 3 - EDDM:

[TA note: Remote Ps receive patriotic EDDM; in-person see plain EDDM]

[Remote] Please take a look at Mailing 3.

[In person] [Hand the participant Mailing 3]

A few days after you receive the reminder letter, you would receive this postcard. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on format/design?

___ Yes ___ No ___ Not sure

2. Does P comment on “your response helps your local government…”?

___ Yes ___ No ___ Not sure

3. Does P comment on Census ID?

___ Yes ___ No ___ Not sure

4. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that postcard aside. We will come back to it later.

Mailing 4 – final HUP letter:

[Remote] Please look at Envelope 4.

[In person] [Hand the participant Mailing 4]

If you had not completed the survey, about one week after the postcard you would receive this mailing. Imagine you received this piece of mail at your house. Take a look at as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on due date?

___ Yes ___ No ___ Not sure

2. Does P comment on text “choose ONE way to respond”?

___ Yes ___ No ___ Not sure

3. Does P comment paper questionnaire enclosure?

___ Yes ___ No ___ Not sure

4. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

5. Does P comment on “help your community”/ why response matters?

___ Yes ___ No ___ Not sure

6. Does P comment on interviewer collecting data in person?

___ Yes ___ No ___ Not sure

7. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

8. Does P comment on the data use message – direct trillions of dollars?

___ Yes ___ No ___ Not sure

8a. (if yes) were the comments positive or negative or neutral

___ Positive ___ Negative ___ Neutral

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

NTL(H) postcard: Now look at the postcard labeled NTL(H).

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on “it’s not too late”?

___ Yes ___ No ___ Not sure

2. Does P comment presence/absence of Census ID?

___ Yes ___ No ___ Not sure

3. Does P comment “paper questionnaire sent earlier?”

___ Yes ___ No ___ Not sure

4. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Probes

Thank you. Now I have a few questions I’d like to ask you about the mailings. We’ll go through them again one at a time.

• Only ask probe if not mentioned during think aloud.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings

• Hand materials to respondents SIDEWAYS, so that it is neither face up or face down.

Mailing 1 - initial HUP letter:

Research Questions:

1. Do they understand the purpose of the letter?

2. Do they understand that they are required to complete the census?

[Remote] Please take a look at the first mailing that we had you look at. It should be labeled as Mailing 1.

[In person] This is the first mailing you received. [Give respondent mailing with letter on top.]

1. If needed: [If in person] When you first looked at the letter, what did you notice about it?

[If remote] When you first looked at the letter, what did you notice about it?

2. What do you think the main point of this mailing is? [Do they understand that they have to complete the census?]

3. What would you have done next after reading it? (If needed: Tell me more about that.)

4. [If not mentioned during think aloud] What are your thoughts about the length of the survey?

5. [If not mentioned during think aloud] Did you notice anything in the letter about why you should complete the survey? Tell me more that.

6. [If needed] Did you notice the sentence after the statement that “your response matters?” Can you take a moment and look at that paragraph. What are your thoughts? Would you be more or less likely to complete the survey after reading it?

7. [If needed] Did you notice the sentence that starts with “your answers to the survey are protected by law?” Can you take a moment and look at that paragraph. What are your thoughts? Would you be more or less likely to complete the survey after reading it?

8. [If not mentioned during think aloud] Do you have any feedback about the instructions to respond?

9. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 2 - first HUP reminder:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the second mailing that we had you look at. It should be labeled as Mailing 2.

[In person] This is the second mailing you received. [Give respondent mailing.]

1. If needed: [If in person] When you first looked at the letter, what did you notice about it?

[If remote] When you first looked at the letter, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as the first mailing, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. If you hadn’t responded already, after getting this letter, do you think you would respond to the survey?

6. What does the word “confidential” mean to you in this sentence?

7. [In person, TA points to second sentence in second paragraph] Did you notice this sentence? What do you think about it? Would it make you more or less likely to complete the survey?

[If remote, TA share screen with sentence highlighted] I’m going to share my screen and ask you about some of the sentences in the letter. Did you notice this sentence? What do you think about it? Would it make you more or less likely to complete the survey?

8. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 3 – EDDM postcard:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the postcard/mailing 3.

[In person] This is the third mailing you received. [Give respondent mailing.]

1. If needed: What did you notice about it?

[If remote] When you first saw this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. Do you have thoughts on the sentence that starts with “Your response helps your local government?”

5. What would you have done next after reading it? (If needed: Tell me more about that.)

6. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 4 -final HUP:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the fourth mailing that we had you look at. It should be labeled as Mailing 4.

[In person] This is the fourth mailing you received. [Give respondent mailing.]

1. If needed: [In person] When I first gave you this, what did you notice about it?

[If remote] When you first looked at the letter, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. This mailing came with the paper questionnaire for you to complete. What do you think about that?

6. Do you think you can complete the questionnaire over the phone?

7. [If not mentioned during think aloud] Did you notice the due date on this letter? What did you think about that?

8. Did you notice the sentence that starts with “Your response helps your community?” Please read that paragraph. What are your thoughts? Would you be more or less likely to complete the survey after reading it.

9. [If any observations marked as yes above] I noticed you/you commented on…

NTL(H) postcard: Now look at the postcard labeled NTL(H).

Thank you. [Take back previous mailing.]

[Remote] Please take a look at mailing NTLH.

[In person] This is mailing NTLH that you received. [Give respondent mailing.]

1. If needed: [In person] When I first gave you this, what did you notice about it?

[If remote] When you first looked at the postcard, what did you notice about it?

2. [If any observations marked as yes above] I noticed you/you commented on…

[After all HUP mail materials]

IVR message:

OK, so we have one final thing to do with you with these mail materials. Please look at letter/postcard labeled number 7. Now imagine that you decided that you wanted a paper questionnaire. What would you do?

[If P says they would call number] OK, let’s imagine you called the number and this is what you heard. “Thank you for calling the 2024 Census Survey Information Line. For general information about the 2024 Census Survey, including how the Census Bureau protects your responses, press 1. For questions about mail, including how to request a paper questionnaire, press 2. For help finding your Census ID to respond online, press 3. For information to help you respond to the survey, press 4. To repeat this menu, press 8.”

What would you do now?

[If P says they would press 2] OK, if you pressed 2, this is what you’d hear, “The Census Bureau is minimizing the use of paper questionnaires. However, if you do not want to respond to the survey online, a paper questionnaire can be mailed to you. If you would like to request a paper questionnaire, press 1. If you already responded to the survey, but continue to receive mailings from the Census Bureau, press 2. To repeat this menu, press 8. To go back to the previous menu, press 9.”

What would you do now?

[If P says, press 1] OK, if you pressed 1, this is what you’d hear, “To request a paper questionnaire in the mail, you must provide your Census ID. Your Census ID is a series of 12 digits found on the front of the letter. If you received a postcard, the 12-digit Census ID is located below the barcode and separated by dashes. Please note that it may take up to xx days to receive your paper questionnaire in the mail. To request a paper questionnaire using your Census ID now, press 1. To repeat this menu, press 8. To go back to the previous menu, press 9.”

What would you do now?

[If P says press 1] OK, if you pressed 1, this is what you’d hear, “Please enter your 12-digit Census ID using the keypad on your phone.”

Any thoughts about what you’d hear or the process?

Note any questions P asked/comments P made about this text message:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just heard?

You’re done with this part. Now we’re going to focus on the survey itself.