Decennial Mail Materials Protocol

SmaRT Testing of Decennial Mail Materials

RESPONDENT ID #: _________________________ DATE: ____ / ____ / _____

INTERVIEWER’S NAME: _________________________________________________

START TIME: ______: ______ AM / PM

LOCATION: _____________

Introduction

Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to help us today. Let me start by telling you a little about what we will be doing today.

Every year, the Census Bureau conducts a number of surveys. Today we are going to look at some of the letters and postcards that you would receive in the mail, and the questionnaire that you would answer online if you were completing the Census.

[If needed: Because these are example mailings, they may not look quite as nice as they would if you received them in the mail.] We will go through several different pieces of mail one by one. After you’ve finished with one piece of mail, I will have you look at the next.

As we go through these mailings, I’m going to ask you to treat each one as you would if it came to your home. Please go through these as you would at home (as if I were not [here/there]), but I’d like you to say out loud what you are thinking as you look at the mailings.

I’m also going to ask you some questions about the mailings and about your reactions to them. There are no right or wrong answers. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear. Please feel free to be honest.

[IF OBSERVER PRESENT]: I want to let you know that this interview is being observed by other Census Bureau staff. They are observing the interview because they are also interested in how the mailings work for people like you.

Do you have any questions about what we’ll be doing?

Informed Consent

• IF IN PERSON [HAND RESPONDENT COPY OF THE CONSENT FORM.]

Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document. Once you have finished reading the document, please sign and date it.

• [RESPONDENT READS FORM THEN SIGNS AND DATES FORM]

• [INTERVIEWER SIGNS AND DATES FORM]

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Thank you for signing the consent form. I will also sign this form to show that I agree to everything in the form.

IF REMOTE

[Participant was emailed consent form]. We emailed you information about the session today and your rights as a participant. This document explains a little bit about this interview and provides information about your rights as a participant, that your participation is optional, all information you provide is confidential, and we won’t use your name in any report. It also asks for your permission to have this session audio recorded and for your screen to be recorded so that I can concentrate on what you are saying rather than having to take extensive notes. Please ask me any questions you have about this document.

I just need to capture your consent orally. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Do you give your consent?

• [IF RESPONDENT PROVIDES CONSENT TO HAVE THE SESSION RECORDED]: I will now turn on the recorder.

• [TURN ON RECORDER]

Think Aloud

I’d like you to answer the questions as you would if you were at home without me there, but with one big difference. What I would like you to do is tell me what you are thinking and feeling as you come up with your answers. 

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[GIVE FEEDBACK ON THE THINK ALOUD THEY DID]

IF REMOTE

Camera

So, just like we practiced in the tech check – I am going to need you to go ahead and turn your camera on by clicking on the camera icon.

[If able to see P] Okay great I can see you. Now we can get started with the task.

[If unable to see P will need to troubleshoot it]

Mailing 1 – initial letter:

Okay, let’s get started.

[Remote] Please open the priority mail envelope. There are a number of pieces of mail in there so be sure you are in a place to lay them out. We will be referring to each one twice in this session. You will see several letters inside. Please find the one labeled Mailing 1 and take it out. There should be a letter with a handwritten note at the top. What does it say? [Mailing 1]

[In person] [Hand the participant Mailing 1]

Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on Respond By Date?

___ Yes ___ No ___ Not sure

2. Does P comment the Census ID?

___ Yes ___ No ___ Not sure

3. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

4. (Spanish only) Does P find Spanish text with no assistance?

___ Yes ___ No ___ Not sure

5. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

6. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

Mailing 2 – reminder letter :

[Remote] Please look at Mailing 2. There should be a letter with a handwritten note at the top. What does it say? [Mailing 2]

[In person] [Hand the participant Mailing 2]

A few days after you get the first letter, you would receive this letter. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on “send you a paper questionnaire”?

___ Yes ___ No ___ Not sure

2. Does P comment on the Respond By Date?

___ Yes ___ No ___ Not sure

3. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

4. Does P comment on the Census ID?

___ Yes ___ No ___ Not sure

5. Does P comment on “required by law”?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

Mailing 3 - Postcard:

[Remote] Please take a look at Mailing 3.

[In person] [Hand the participant Mailing 3]

A few days after you receive the reminder letter, you would receive this postcard. Imagine you received this piece of mail at your house. Take a look at it as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on format/design?

___ Yes ___ No ___ Not sure

2. Does P comment on “your response helps your local government…”?

___ Yes ___ No ___ Not sure

3. Does P notice or comment on Census ID?

___ Yes ___ No ___ Not sure

4. Does P comment on bilingual aspect?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that postcard aside. We will come back to it later.

Mailing 4 – Final letter and First page of paper questionnaire:

[Remote] Please look at Mailing 4.

[In person] [Hand the participant Mailing 4]

If you had not completed the survey, about one week after the postcard you would receive another mailing that would include a paper questionnaire. The mailing includes the letter you would receive, as well as the first page of that questionnaire. Imagine you received these materials at your house. Take a look at as if I were not [here/there] and remember to think out loud as you do. You can also let me know if anything stands out to you.

[If participant reads the letter aloud.] You do not have to read aloud unless that is what you would do at home.

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P comment on the Census ID or the arrow pointing to it?

___ Yes ___ No ___ Not sure

2. Does P comment on bilingual aspect of the letter?

___ Yes ___ No ___ Not sure

3. Does P comment on the Due Date?

___ Yes ___ No ___ Not sure

Note any questions P asked/comments P made about this mailing:

Other notes:

[If P does not provide enough information during think aloud] Do you have any reactions or any immediate thoughts about what you just read? 

Thank you. You can put that letter aside. We will come back to it later.

Probes

Now we will look at some draft envelopes.

Time capsule envelope 1

[Remote] Please take a look at a piece of paper with the label Envelope 1 on it.

[In person] [Hand the participant Envelope 1]

Pretend you received an envelope from the U.S. Census Bureau and this was what the back of the envelope looked like. Please share with me your thoughts.

Record thoughts

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P understand what a time capsule is?

___ Yes ___ No ___ Not sure

2. Does P comment positively, negatively, or neutrally on the messaging?

___ Positive ___ Negative ___ Neutral

Would you be more or less likely to open the envelope with this message or would it not matter?

___ More likely ___ Less likely ___ Would not matter

Please explain your answer:

Time capsule envelope 2

[Remote] Please take a look at a piece of paper with the label Envelope 2 on it.

[In person] [Hand the participant Envelope 2]

Pretend you received an envelope from the U.S. Census Bureau and this was what the back of the envelope looked like. Please share with me your thoughts.

Record thoughts

OBSERVE (DO NOT READ TO PARTICIPANT):

1. Does P understand what a time capsule is?

___ Yes ___ No ___ Not sure

2. Does P comment positively, negatively, or neutrally on the messaging”?

___ Positive ___ Negative ___ Neutral

Would you be more or less likely to open the envelope with this message or would it not matter?

___ More likely ___ Less likely ___ Would not matter

Please explain your answer:

Is there one envelope message that you liked more?

___ Yes ___ No

Which one was preferred?

___ Envelope 1 ___ Envelope 2 ___ N/A

Please explain why you liked one message more than the other?

Thank you. Now I have a few questions I’d like to ask you about the mailings. We’ll go through them again one at a time.

• Only ask probe if not mentioned during think aloud.

• Take back each mailing after respondent is finished with it—do not let them compare the mailings

• Hand materials to respondents SIDEWAYS, so that it is neither face up or face down.

Mailing 1 - initial letter:

Research Questions:

1. Do they understand the purpose of the letter?

2. Do they understand that they are required to complete the census?

[Remote] Please take a look at the first mailing that we had you look at. It should be labeled as Mailing 1.

[In person] This is the first mailing you received. [Give respondent mailing with letter on top.]

1. If needed: When you first looked at the letter, what did you notice about it?

2. What do you think the main point of this mailing is? [Do they understand that they have to complete the census?]

3. What would you have done next after reading it? (If needed: Tell me more about that.)

4. [If not mentioned during think aloud] What are your thoughts about when you would complete the survey?

5. If not mentioned – point to Please respond by box. Did you see this information? What does it mean to you?

   1. Do you think you can complete the survey after that date? Why or why not.

6. [If not mentioned during think aloud] Do you have any feedback about the instructions to respond?

7. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 2 – reminder letter:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the second mailing that we had you look at. It should be labeled as Mailing 2.

[In person] This is the second mailing you received. [Give respondent mailing.]

1. If needed: [If in person] When you first looked at the letter, what did you notice about it?

[If remote] When you first looked at the letter, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as the first mailing, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. If you hadn’t responded already, after getting this letter, do you think you would respond to the survey?

6. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 3 – Postcard:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the postcard/mailing 3.

[In person] This is the third mailing you received. [Give respondent mailing.]

1. If needed: What did you notice about it?

[If remote] When you first saw this, what did you notice about it?

2. What do you think the main point of this mailing is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. What would you have done next after reading it? (If needed: Tell me more about that.)

5. [If any observations marked as yes above] I noticed you/you commented on…

Mailing 4 – Final letter and first page of Paper Questionnaire:

Thank you. [Take back previous mailing.]

[Remote] Please take a look at the fourth mailing that we had you look at. It should be labeled as Mailing 4.

[In person] This is the fourth mailing you received. [Give respondent mailing.] Let’s look at the letter first.

1. If needed: [In person] When I first gave you this, what did you notice about it?

[If remote] When you first looked at this page, what did you notice about it?

2. What do you think the main point of this letter is?

3. Did this seem the same as any of the previous mailings you saw, or did it seem different? (If needed: Tell me more about that.)

4. Do you think you can complete the questionnaire over the phone?

5. [If not mentioned during think aloud] Did you notice the due date on this letter? What did you think about that?

6. [If any observations marked as yes above] I noticed you/you commented on…

7. Now, let’s look at the first page of the paper questionnaire. What do you notice about it?

8. Is there anything confusing?

You’re done with this part. Now we’re going to focus on the survey itself. We are going to use this last mailing for this survey part.