OMB Memo

2020 American Community Survey (ACS) Revised Mailing Materials Cognitive Interviews

Submitted Under Generic Clearance for the Collection of Routine Customer Feedback

Request: The Census Bureau plans to conduct additional research for the collection of routine customer feedback (OMB number 0690-0030). We will be conducting cognitive interviews to evaluate materials proposed for the 2020 American Community Survey (ACS).

Purpose: There has been evidence from previous decennial census years that the census has an impact on ACS response rates (Chesnut and Davis, 2011; Baumgardner 2013). In 2000 and 2010, self-response rates to the ACS increased prior to census day, perhaps because respondents thought the ACS was the first request to complete the census. ACS response rates then declined after census day, perhaps because respondents thought the ACS was the census and they had already completed it.

After observing this trend in 2000, the Census Bureau developed alternative ACS mailing materials for 2010, with the goal of minimizing the impact of the census on ACS response rates. In these alternative materials, we explained to respondents that the ACS was different from the 2010 Census, and they were required to complete both. Multiple letter and envelope changes were cognitively tested (Schwede 2013), and then three treatments were tested in 2010. The treatments were compared to each other and to the standard ACS production materials, which did not attempt to differentiate the ACS from the 2010 Census. All of the treatments improved ACS participation compared to the production materials (Chesnut and Davis, 2011). We also developed and cognitively tested a field flier that ACS field representatives (FRs) could use during nonresponse followup to further differentiate ACS from the 2010 Census (Schwede and Sorokin, 2010). This flier was viewed positively by FRs, who reported using it frequently. Because of this success in 2010, we would like to implement similar changes for 2020.

The ACS currently sends sampled mailable addresses up to 5 pieces of mail:

1. Initial mailing (all mailable addresses)

2. Pressure seal mailer reminder (all mailable addresses)

3. Questionnaire mailing (only mailable addresses that have not responded to the two first mailings)

4. Reminder postcard (only mailable addresses that have not responded to the two first mailings)

5. Pressure seal mailer reminder (only mailable addresses that have not responded to the previous four mailings)

We have modified these materials to emphasize ACS branding, explain that the ACS and 2020 Census are different, and that respondents are required to complete both. The purpose of the cognitive testing is to evaluate whether changes are effective in communicating these messages. Below is a summary of the changes that were made to these mailings (ACS mailings are also shown in Enclosure 1 and the ACS questionnaire, which remains unchanged from 2018 production, can be found in Enclosure 2):

1. Initial mailing: The front of the envelope now emphasizes the ACS (“The U.S. Census Bureau’s American Community Survey”) above the address window; the back of the envelope has the Census Bureau logo. The letter has been modified to include language explaining the difference between ACS and the 2020 Census and that response to both is required. A frequently asked questions section has been added to the back of the letter. All other inserts in this mailing remain unchanged from 2018 production.

2. Pressure seal mailer reminder: Text has been added to the front that reads “The U.S. Census Bureau’s American Community Survey.” Text inside the mailer has been added to explain the difference between ACS and the 2020 Census and that response to both is required.

3. Questionnaire mailing: The look-and-feel of the front of the envelope has been changed as in the first mailing. The back of the envelope has the Census Bureau logo as well as a few sentences explaining the difference between ACS and the 2020 Census and that response to both is required. We are testing two versions of the back of the envelope in cognitive testing. The letter has been modified to include language explaining the difference between ACS and the 2020 Census and that response to both is required. A frequently asked questions section has been added to the back of the letter. All other inserts in this mailing remain unchanged from 2018 production.

4. Reminder postcard: A sentence has been added explaining that response to both the ACS and the 2020 Census is required by law.

5. Pressure seal mailer reminder: Text has been added to the front that reads “The U.S. Census Bureau’s American Community Survey.” There is a minor wording change to the back to clarify that the mailing is for the ACS.

Because each of the mailings has been modified in some way, we plan to include all of them in cognitive testing. We also plan to use the most recently available set of draft 2020 Census mailing materials/questionnaire in the cognitive interviews (see Enclosures 3 and 4), in order to simulate the experience of receiving mail for both ACS and census in 2020. We used this approach when cognitively testing 2010 materials (Schwede, 2013).

We have also updated the 2010 field flier to improve clarity and reflect changes to the ACS and the census for 2020 (e.g., an internet response option will be offered). We will include the flier in the cognitive testing (see Enclosure 5).

Population of Interest: The ACS mail materials are sent to a nationally-representative sample of households in the United States.

Timeline: One round of 20 cognitive interviews will be conducted in October 2018.

Language: Interviews will be conducted in English only.

Method: Staff from the Center for Survey Measurement will conduct one round of cognitive interviews with 20 respondents. Interviews will be conducted in-person at the Census Bureau headquarters and at locations convenient to interviewees in the local metropolitan area (DC, Maryland, Virginia).

Sample and Recruitment: We will recruit respondents using the methods described below. Our recruiting efforts will target respondents who are most likely to handle the mail and react to it in a household. We will recruit respondents who read at different literacy levels, using education as a proxy measure. We will also recruit respondents with diverse demographic characteristics and household composition to the maximum extent possible.

Recruitment: Respondents will be recruited through fliers posted at local community organizations, advertisements on Craigslist.com, and broadcast messages distributed through the Census Bureau’s daily online newsletter. All recruiting materials are attached (see Enclosure 6).

Protocol: We will conduct our cognitive interviews using printed drafts of all mail materials assembled into mail packages. Respondents will be presented with mail packages individually, in the approximate order that they would be received by sampled households in the April ACS panel:

1. 2020 Census letter #1

2. 2020 Census letter #2

3. ACS initial mailing

4. 2020 Census postcard

5. ACS pressure seal reminder

6. 2020 Census questionnaire mailing

7. ACS questionnaire mailing

8. ACS postcard

9. 2020 Census postcard #2

10. ACS pressure seal reminder #2

Researchers will observe respondents’ interactions with these mailings and ask respondents about their reactions to them. We will ask probes as needed to determine whether respondents noticed some materials were for the ACS while others were for the 2020 Census and that response to both is mandatory. We will also ask probes about specific features of interest, such as the third mailing envelope flap (and an alternative option) and the FAQs on the back of the letter. The interview will conclude by asking respondents for their impressions of the field flier as well as general debriefing probes, such as if the mailing materials could be improved to better communicate that ACS and the 2020 Census are different and mandatory (see Enclosure 7).

Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio and/or videotaped to facilitate analysis of the results (see Enclosure 8). Participants who do not consent to be video and/or audio-taped will still be allowed to participate.

Incentive: Participants will receive $40 to offset the costs of participation in this research, such as travel and parking.

Length of Interview: We estimate that each of the 20 interviews will take approximately one hour. This results in a burden of 20 hours.

The pre-approved generic screening questionnaire will take approximately ten minutes per person, and the additional screening questions specific to this research will take two minutes per person (see Enclosure 9). We estimate that we will screen three people for each successful recruit for each of the 20 interviews. Therefore, for the first round of interviewing, we estimate a total of 60 people screened for a total of 2 hours (60 people at 2 minutes each).

Thus, the total estimated burden for the first round of this research is 22 hours.

Table 1. Total Estimated Burden

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	60	2 minutes	2 hours
Cognitive Interviews	20	60 minutes	20 hours
Totals			22 hours

Below is a list of materials to be used in the current study:

Enclosure 1. Modified 2020 ACS mailing materials

Enclosure 2. 2018 ACS questionnaire

Enclosure 3. Draft 2020 Census mailing materials

Enclosure 4. Draft 2020 Census questionnaire

Enclosure 5. 2020 field flier

Enclosure 6. Recruitment materials

Enclosure 7. Cognitive interview protocol

Enclosure 8. Consent form

Enclosure 9. Additional screening questions

The contact person for questions regarding data collection and the design of this research is listed below:

Jessica Holzberg

Center for Survey Measurement

U.S. Census Bureau

Room 5K026B

Washington, D.C. 20233

(301) 763-2298

Jessica.Holzberg@census.gov

References:

Baumgardner, S. 2013. Tracking American Community Survey Mail Response During the 2010 Census. Available at: https://www.census.gov/library/working-papers/2013/acs/2013_Baumgardner_01.html.

Chesnut, J. and Davis, M. 2011. Evaluation of the ACS Mail Materials and Mailing Strategy during the 2010 Census. Available at: https://www.census.gov/library/working-papers/2011/acs/2011_Chesnut_01.html.

Schwede, L. 2013. Cognitive Testing of Modified American Community Survey Envelopes and Letters for Use during the 2010 Census: ACS Messaging Project Final Report on Phases 1 and 2. Available at: https://www.census.gov/library/working-papers/2013/acs/2013_Schwede_01.html.

Schwede, L. and Sorokin, A. 2010. Cognitive Testing Results for the ACS Field Flyer for Use in 2010: ACS Messaging Project Phase 3 Final Report. Available at: https://census.gov/library/working-papers/2010/acs/2010_Schwede_01.html.

5