Cognitive Testing of ACS 2017 Mail Test Materials
PARTICIPANT ID #: _________________________ DATE: ____ / ____ / 2016
INTERVIEWER’S NAME: _________________________________
START TIME: ______: ______ AM / PM
SECTION I. INTRODUCTION |
Hello, I’m [NAME OF INTERVIEWER]. I work for the Census Bureau. Thanks for agreeing to help us today. Let me start by telling you a little about what we will be doing today. The United States Census Bureau counts the population in the U.S and also conducts various kinds of surveys.
Today we are going to look at some of the envelopes, letters, postcards, and the questionnaire that you would get in the mail if your address had been selected to respond to a national survey called the American Community Survey. There are five things to look at. I’ll explain each one to you as we go along. I’m going to ask you to look at the envelopes and read the letters, postcards, and questionnaire, and then ask you some questions about them: what they mean to you, and what are your reactions to them.
Remember, there are no wrong answers, because only you know what you are thinking. I really want to hear your opinions and reactions to these pieces of mail, so don’t hesitate to speak up whenever something is unclear, or not easy to understand. Also just so you know, I did not have anything to do with creating anything I will show you today, so please feel free to be honest. Do you have any questions about what we’ll be doing?
|
|
|||
|
|
|
|
|
PLACE THE CONSENT FORM IN FRONT OF PARTICIPANT
Before we start, I would like you to read over the document in front of you. This document explains a little bit about this interview and provides information about your rights as a participant. Our session today is completely confidential, so all information you provide is confidential and we won’t use your name in any report. Your participation in this study is completely voluntary. You may decline to answer any particular question. This form also asks for your permission to have this session audio recorded. That way, I can focus today on what you’re saying rather than having to concentrate on taking notes or having to rely on my memory later. Please ask me any questions you have about this document. Once you have finished reading the document, please sign it.
PARTICIPANT READS FORM THEN SIGNS AND DATES FORM
INTERVIEWER SIGNS AND DATES FORM
B. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED: I will now turn on the audio recorder.
SECTION III. MAIL MATERIALS |
SET-UP: Okay, let’s get started. Imagine you received this piece of mail at your house, and that this is your address. [HAND R ENVELOPE WITH INITIAL PACKAGE, POINT TO “123 Any Street...”] First, I have some questions about the envelope itself.
What do you think you would do after receiving this envelope in the mail?
What was the first thing you noticed about the envelope?
Do you see anything about the envelope that might make you more likely to open it? [If necessary] Why would that make you more likely to open the envelope?
Is there anything on this envelope that stood out to you? [If necessary] Why did that stand out to you?
[If necessary] Do you like or dislike that being on the envelope? Why?
Now I’d like you to take the envelope and treat it as you would if you received it in the mail. [Let the respondent take as much time looking at the materials as they want before starting the probes]
BY OBSERVATION:
Does R open the envelope? ___ Yes ___ No
[If R indicates that they wouldn’t even open the envelope, note that, then instruct them to open the envelope and look through the materials inside]
Does R open the envelope with the address facing them or facing away from them?
___ Address facing them ___ Address facing away
Does R look at both the letter and the brochure? ___ Yes ___ No
If yes, which does R look at first – the letter or the brochure?
___ Letter ___ Brochure ___N/A
If no, which one does R look at?
___ Letter ___ Brochure ___N/A
Note unsolicited comments R makes about the brochure here, if any:
[If R did not look at anything inside the envelope] I noticed that you didn’t look at anything inside the envelope. Can you tell me why?
[If R did not look at the letter/the brochure] I noticed that you didn’t look at [the letter/the brochure]. Can you tell me why?
[If R did not look at the letter at all, say “Please take a look at the letter that was in the envelope.”]
Does R turn the letter over and find text on back?
___ Yes ___ No
If yes, does R read the text on the back?
___ Yes ___ No ___ N/A
STANDARD PROBES (when R indicates he/she is done looking through the materials and reading the letter):
Which parts of the letter did you read?
What was the first thing you noticed about the letter?
What do you think the main point of the letter was? [If they give a brief answer such as “respond to a survey”] Tell me more about that.
Was there any part of the letter you spent more time on than other parts? [If necessary] Why did you spend more time on that part?
[Optional probe if they are brief] Did anything pop out at you right away?
[AT THIS TIME, HAND THEM THE PINK AND GREEN HIGHLIGHTER]
Now with the green highlighter I would like you to highlight anything in the letter that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say,” Okay, then please highlight the top three things that you noticed in this letter.”] [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that/those?
Now I am going to ask you to use the pink highlighter to mark anything you think some people might find difficult to understand. If you want to mark something in pink that you already highlighted with green, you can underline it. [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that?
What do you think you would do after receiving this letter?
[If necessary] After getting this letter, would you respond to the survey? Why/why not?
[If yes to A13/If R mentioned responding] How can you respond to the survey?
BY OBSERVATION: Did R seem to know that the internet response was the only option at this point, or did they have to look back over the letter to find out how to respond?
___Seemed to know off-hand ___Had to re-read letter to answer
___I couldn’t tell from observation ___N/A (R said wouldn’t respond)
[Ask of all respondents] If you couldn’t respond by internet what would you do?
Do you think you have to do the survey?
What makes you think that?
Did you notice any reasons for doing the survey? [If they say yes and don’t elaborate] What were they? Anything else?
What do you think about these reasons to respond to the survey?
There is a sentence that mentions your response being required by law. [POINT TO SENTENCE AT END OF FIRST PARAGRAPH BELOW THE BOX] What is your reaction to that sentence?
What do you think about the sentence where we mention sending a paper questionnaire to your house in about three weeks? [POINT TO “If you are unable to complete…” IN THE SECOND PARAGRAPH BELOW THE BOX]
What do you think about these reasons to respond to the survey online? [POINT TO “Answering online saves taxpayer money…” IN SECOND PARAGRAPH RIGHT ABOVE BOX]
Of these, which reason do you think is most important? Why?
Are there other reasons we could list that would make you more likely to respond online?
[TAKE THE MAIL MATERIALS BACK BUT HAVE THEM KEEP THE MARKERS]
SET-UP: About a week after you get the first letter you would receive another letter reminding you to complete the survey online. Again, we will start by looking at the envelope. [HAND R ENVELOPE FOR FIRST REMINDER LETTER]
What was the first thing you noticed about this envelope?
What do you think you would do after receiving this next envelope in the mail?
Do you see anything about the envelope that might make you more likely to open it? [If necessary] Why would that make you more likely to open the envelope?
Is there anything on this envelope that stood out to you? [If necessary] Why did that stand out to you?
[If necessary] Do you like or dislike that being on the envelope? Why?
Now, please take the envelope and treat it as you would if you received it in the mail. Remember to think aloud as you do.
BY OBSERVATION:
Does R open the envelope? ___ Yes ___ No
[If R indicates that they wouldn’t even open the envelope, note that, then instruct them to open the envelope and look through the materials inside]
[IF R HASN’T LOOKED AT THE LETTER] Please take a look at the letter now.
Does R appear to read the letter closely? ___ Yes ___ No
STANDARD PROBES (when R indicates he/she is done reading):
What was the first thing you noticed about the letter?
Which parts of the letter did you read?
Was there anything that stood out to you as different from the last letter? [If yes] Why did that stand out to you?
What do you think the main point of the letter was? [If they give a brief answer such as “respond to a survey”] Tell me more about that. Why did you say [ANSWER]?
Was there any part of the letter you spent more time on than other parts? [If necessary] Why did you spend more time on that part?
[Optional probe if they are brief] Did anything pop out at you right away?
Now with the green highlighter I would like you to highlight anything in the letter that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say, “Okay, then please highlight the top three things that you noticed in this letter.”] [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that/those?
Now I am going to ask you to use the pink highlighter to mark anything you think some people might find difficult to understand. [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that?
What do you think you would do after receiving this letter?
[Optional probe if they just say “I would respond”] Tell me more about how you would respond.
[If they say they would have already responded after the first letter in response to A16] If you had already responded to the survey online and then you received this letter, what would your reaction be?
If you hadn’t responded already, after getting this letter would you respond to the survey? Why/why not?
Did the letter mention any new reasons for doing the survey? [If they say yes and don’t elaborate] What were they?
[If R mentioned reasons] What do you think about these reasons to respond to the survey?
[If necessary] What do you think about the part where we mention sending a paper questionnaire to your house in about ten days? [If they say didn’t notice: POINT TO SENTENCE RIGHT BELOW BOX]
[TAKE THE MAIL MATERIALS BACK BUT HAVE THEM KEEP THE MARKERS]
SET-UP: If you haven’t completed your online survey, about two weeks after you get the reminder letter you would receive another package. If you respond online but we didn’t receive it right away for some reason, you could still get the paper questionnaire. Let’s take a look at the envelope. [HAND R THE PAPER QUESTIONNAIRE PACKAGE]
What was the first thing you noticed about this envelope?
What do you think you would do after receiving this envelope in the mail?
Do you see anything about the envelope that might make you more likely to open it? [If necessary] Why would that make you more likely to open the envelope?
Is there anything on this envelope that stood out to you? [If necessary] Why did that stand out to you?
[If necessary] Do you like or dislike that being on the envelope? Why?
Now, please take the envelope and treat it as you would if you received it in the mail.
BY OBSERVATION:
Does R open the envelope? ___ Yes ___ No
[If R indicates that they wouldn’t even open the envelope, note that, then instruct them to open the envelope and look through the materials inside]
Does R open the envelope with the address facing them or facing away from them?
___ Address facing them ___ Address facing away
Does R look at all three materials? ___ Yes ___ No
___ Letter ___ Questionnaire
___Return envelope ___N/A
If no, which ones does R look at?
___ Letter ___ Questionnaire
___Return envelope ___N/A
I have some specific questions about the letter and the questionnaire. I see that you looked at the [Letter/Questionnaire] first, so let's start with that. [Go to probes for the correct mail piece first, then move on to the remaining mail piece.]
LETTER:
“Please take a look at the letter that was in the envelope.”
[Once R is looking at the letter] The next questions are only about the letter.
Does R appear to read the letter closely? ___ Yes ___ No
STANDARD PROBES (when R indicates he/she is done reading):
What was the first thing you noticed about the letter?
Which parts of the letter did you read?
Was there anything that stood out to you as different from the last letter? [If yes] Why did that stand out to you?
Was there any part of the letter you spent more time on than other parts? [If necessary] Why did you spend more time on that part?
What do you think the main point of the letter was?
Now with the green highlighter I would like you to highlight anything in the letter that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say, “Okay, then please highlight the top three things that you noticed in this letter.] [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that/those?
Now, with the pink highlighter, I would like you to mark anything you think some people may find difficult to understand. [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that?
What do you think you would do after receiving this letter? [Optional probe] Can you tell me more about?
Did you notice any new reasons to respond to the survey that you hadn’t seen yet? [If they say yes and don’t elaborate] What were they?
Do you think this letter would encourage you to respond to the survey? Why/why not?
[If yes] How would you respond to the survey after receiving this letter?
Did you notice any other ways you could respond to the survey?
Let’s say that you completed the survey online, and then received this package with the paper questionnaire. What would your reaction be?
QUESTIONNAIRE:
The next questions are about the paper questionnaire that was in the envelope. [If necessary] Please take a minute to look at first page of the questionnaire.
STANDARD PROBES (when R indicates he/she is done reading):
Now with the green highlighter I would like you to highlight anything on the first page of the questionnaire that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say, “Okay, then please highlight the top three things that you noticed on the first page.”] Why did you choose to highlight that/those?
Now, with the pink highlighter, I would like you to mark anything you think some people may find difficult to understand. Why did you choose to highlight that?
What was the first thing you noticed about the first page of the questionnaire?
Just looking at the first page, is it clear what you are supposed to do?
After looking at the first page, how would you respond to the survey?
[If R says they would fill out the paper form] Did you notice any other ways you could respond to the survey?
Do you have any other thoughts or comments you’d like to share about the first page?
[TAKE THE MATERIALS BACK]
SET-UP: About four days after receiving the previous package, you would receive this postcard. Please look over it. [HAND RESPONDENT THE POSTCARD, ADDRESS SIDE UP]
BY OBSERVATION:
Did R turn the postcard over? ___ Yes ___ No
Does R appear to read the postcard closely? ___ Yes ___ No
STANDARD PROBES (when R indicates he/she is done looking at the postcard):
What was the first thing you noticed about the postcard?
Which parts of the postcard did you read?
Was there anything that stood out to you as different from the last letter? [If R only mentions that this is a postcard and that was a letter: “Is there anything else?”] [If yes] Why did that stand out to you?
As with the letter, with the green highlighter I would like you to highlight anything in the postcard that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say, “Okay, then please highlight the top three things that you noticed in this postcard.”] [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that/those?
Now use the pink highlighter to mark anything that you think some people might find difficult to understand. [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that?
What do you think the main point of the postcard was?
What do you think you would do after receiving this postcard?
Did the postcard mention any new reasons for doing the survey? [If they say yes and don’t elaborate] What were they?
[TAKE THE POSTCARD BACK]
SET-UP: About three weeks after you get the postcard, if you haven’t responded to the ACS either by mail or online, we will send you another letter. Let’s take a look at the envelope. [HAND R ENVELOPE FOR THE SECOND REMINDER LETTER]
What was the first thing you noticed about this envelope?
What do you think you would do after receiving this envelope in the mail?
Do you see anything about the envelope that might make you more likely to open it?? [If necessary] Why would that make you more likely to open the envelope?
Is there anything on this envelope that stood out to you? [If necessary] Why did that stand out to you?
[If necessary] Do you like or dislike that being on the envelope? Why?
Now, please take the envelope and treat it as you would if you received it in the mail.
BY OBSERVATION:
Does R open the envelope? ___ Yes ___ No
[If R indicates that they wouldn’t even open the envelope, note that, then instruct them to open the envelope and look through the materials inside]
[If R does not look at the letter, say, “Please take a look at the letter that was in the envelope.”]
Does R appear to read the letter closely? ___ Yes ___ No
STANDARD PROBES (when R indicates he/she is done reading):
What was the first thing you noticed about the letter?
What parts of the letter did you read?
Was there anything that stood out to you as different from the last postcard? [If R only mentions that this is a letter and that was a postcard: “Is there anything else?”] [If yes] Why did that stand out to you?
Now with the green highlighter I would like you to highlight anything in the letter that would make you more likely to respond to the survey. [If R says nothing would make them more likely to respond, say, “Okay, then please highlight the top three things that you noticed in this letter.”] [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that/those?
Now, with the pink highlighter, I would like you to mark anything you think some people may find difficult to understand. [WAIT FOR RESPONDENT TO FINISH HIGHLIGHTING] Why did you choose to highlight that?
Was there any part of the letter you spent more time on than other parts? [If necessary] Why did you spend more time on that part?
What do you think the main point of the letter was?
What do you think you would do after receiving this letter? [Optional probe] Can you tell me more about?
Did you notice any new reasons to respond to the survey that you hadn’t seen yet? [If they say yes and don’t elaborate] What were they?
Do you think this letter would encourage you to respond to the survey? Why/why not?
[If yes] How would you respond to the survey after receiving this letter?
Did you notice any other ways you could respond to the survey?
SECTION IV. DEBRIEFING |
I just have a few more questions for you.
In general, would you say you were likely or not very likely to read these materials if they came in the mail to you?
[If they don’t mention it then ask]How is the mail usually handled in your household? [If necessary] Who usually gets the mail from the mailbox? Who usually opens the mail in your household?
[If not mentioned already] How often do you look at your mail?
[If not mentioned already] When [you/the person who usually handles the mail] comes across mail that requires you to do something, like respond to a survey or pay a bill, what do you usually do with it?
[If necessary] How long does it usually take you to follow up on it?
Is there anything we could change about the envelopes that would make you more likely to open them? You can look at the envelopes again if you’d like.
[If there is time for R to review brochure] In one of the mailings you looked at earlier, there was a brochure we didn't discuss [HAND RESPONDENT A COPY OF THE BROCHURE]. Looking at it now, is there anything about it that stands out to you?
Is there anything we could change about it that would make you more likely to read it?
Did the messages in the letters bring up any issues in your mind that some people might find sensitive or uncomfortable?
[If necessary] How did you feel about the number and types of communications you’d receive if your address was selected to respond to the survey?
Was there a specific piece of mail that stood out and convinced you to respond to the survey? [If necessary] Which piece? Why did that piece convince you to respond?
Thinking of all of them together, were there any mailings that you thought were not necessary?
Was there anything else that we haven’t talked about that you would like to mention?
Thank you for participating in this study. Your feedback has been very helpful.
[TURN OFF TAPE RECORDER AND GIVE INCENTIVE; HAVE R SIGN THE PAYMENT VOUCHER]
Here is your money for helping me today. I need you to sign this payment voucher so the government knows that I gave you the money.
[To be completed by interviewer immediately after interview]:
What were your general impressions of this respondent and interview?
Is there anything you would like us to know about this interview for the analysis?
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Laurel K Schwede |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |