OMB Memo

2017 American Community Survey (ACS) Revised Mailing Materials Cognitive Interviews

Submitted Under Generic Clearance for the Collection of Routine Customer Feedback

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research. We will be conducting cognitive interviews to aid in the preparation of a 2017 American Community Survey (ACS) mail test.

Purpose: The purpose of this cognitive research is to (1) evaluate the effectiveness of modified mailing materials (including envelopes, letters, postcards, and the ACS questionnaire) incorporating design elements recently recommended by a preeminent survey methodologist (Dr. Donald Dillman), and (2) assess respondent reactions to the modified materials and identify strengths and weaknesses. Below is a summary of the mail materials to be tested:

• Envelopes: The envelopes for the ACS mail materials have been revised per recommendations provided by Dr. Dillman. The modified envelopes are intended to be more appealing to respondents and non-essential information has been removed or minimized. The test will include four envelopes in total.

• Letters and Postcard: The standard ACS letters and postcard have been revised per recommendations provided by Dr. Dillman to offer more information to respondents about the data collection process and to have a more consistent, polite tone. The letter content is sequential, offering new information as the data collection effort progresses and moving from an effort to obtain internet responses to a choice between internet and paper response modes. The test will include four letters and one postcard.

• Informational Brochure: The informational brochure "How Your Responses Help America" informs respondents about the types of information collected on the American Community Survey and how each type of information is used by policy makers and communities.

• First Page of ACS Paper Questionnaire: The first page of the standard ACS paper questionnaire has been revised per recommendations provided by Dr Dillman. The first page of the modified questionnaire now has a stand-alone design; it is intended to give enough information so that a respondent can complete the questionnaire even if they do not read any of the other mail materials provided in the mailing package.

The full set of materials is attached (see Attachment I: 2017 ACS Revised Mailing Materials). Because each of the mail materials has been modified in some way, we plan to cognitively test each of the mail materials using specific probes designed as part of the cognitive test .The cognitive testing will focus on elements of the mailing materials that would encourage individuals to read them and respond to the survey invitation while also looking for aspects that respondents find confusing or concerning. In order to get the best sense of how the revisions to the materials might affect public perceptions and response rates for the ACS, we will recruit individuals who handle the mail for their households and represent diverse respondent populations (with a particular focus on respondent age and likelihood of completing the survey by internet or by mail). By combining the responses across interviews, we can evaluate the strengths of the revised mail materials and identify any potential problems in terms of likelihood of reading the mail materials, respondent comprehension, or willingness to respond to the survey invitation. The results of the cognitive testing will inform the final revised ACS mail materials that will be used in a field test in 2017.

Population of Interest: The ACS mail materials are sent to a nationally-representative sample of households in the United States.

Timeline: From November 2016 to January 2017, staff from the Center for Survey Measurement will conduct one round of cognitive interviews with 25 respondents. In the event that there are any outstanding issues in the questionnaire that have not been addressed by the first round of interviewing, we will reserve the possibility to conduct an optional second round of cognitive interviews in February and March of 2017 with 10 respondents.

Sample: We will conduct the interviews with a total of 25 respondents (15 who are identified as likely internet responders and 10 who are identified as likely paper responders) using printed drafts of all mail materials assembled into mail packages.

Our recruiting efforts will target respondents with the following characteristics related to survey response behavior and likelihood of reviewing the ACS mail materials:

 

• Device ownership and level of internet use

• Younger than 30 years or older than 65 years

• Handles the mail in his or her household

Recruitment: Respondents will be recruited through fliers posted at local community organizations; advertisements on Craigslist.com; and broadcast messages distributed through the Census Bureau’s daily online newsletter. All recruiting materials are attached (see Attachment II: Recruitment Advertisements).

Location of Interviews: Interviews will be conducted at the Census Bureau’s Response Research Laboratory and at locations convenient to interviewees. The interviews will be conducted in-person in the local metropolitan area (DC, Maryland, Virginia), and possibly in nearby Pennsylvania, New Jersey, and West Virginia as well.

Protocol: We will conduct our cognitive interviews using printed drafts of all mail materials assembled into mail packages. Respondents will be presented with mail packages in the order that they would be received by sampled households and asked probes specific to each mail package as they view each piece. General probes will be asked at the end during respondent debriefing (see Attachment III: ACS Revised Mailing Materials Protocol).

Informed Consent: We will inform participants that their response is voluntary and that the information they provide is confidential and will be accessed only by employees involved in the research project. The consent form will also indicate that the respondent agrees that the interview can be audio-taped to facilitate analysis of the results. Participants who do not consent to be audio-taped will still be allowed to participate.

Incentive: Participants will receive $40 for their participation in this research. A copy of the consent form is attached (see Attachment IV: Consent Form).

Length of Interview: For the first two rounds of interviews, we estimate that each of the 25 interviews will take approximately one hour for a total of 25 hours. The pre-approved generic screening questionnaire will take approximately ten minutes per person, and the additional screening questions specific to this research will take two minutes per person (see Attachment V: Recruitment Eligibility Screeners).

We estimate that we will screen three people for each successful recruit for each of the 25 interviews. Therefore, for the first round of interviewing, we estimate a total of 75 people screened for a total of 2.5 hours (75 people at 2 minutes each). Thus, the total estimated burden for the first round of this research is 27.5 hours.

Table 1. Total Estimated Burden for Round 1

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	75	2 minutes	2.5 hours
Cognitive Interviews	25	60 minutes	25 hours
Totals			27.5 hours

For the optional second round of interviewing, each of the 10 interviews will also take approximately one hour for a total of 10 hours. The screening questions specific to this research will take two minutes per person. We estimate we will screen three people to find each of the 10 respondents. Therefore, for the optional third round of interviewing, we estimate a total of 30 people screened (1 hour) (30 people at 2 minutes each). Thus, the total estimated burden for the optional second round of this research is 11 hours.

Table 2. Total Estimated Burden for Optional Round 2

Category of Respondent	No. of Respondents	Participation Time	Burden
Screening	30	2 minutes	1 hour
Cognitive Interviews	10	60 minutes	10 hours
Totals			11 hours

The total estimated burden for both rounds of this research is 38.5 hours (27.5 hours for Round 1 plus 11 for optional Round 2).

The following documents are included as attachments:

Attachment I: 2017 ACS Revised Mailing Materials

Attachment II: Recruitment Advertisements

Attachment III: ACS Revised Mailing Materials Protocol

Attachment IV: Consent Form

Attachment V: Recruitment Eligibility Screeners

The contact person for questions regarding data collection and the design of this research is listed below:

Casey Eggleston

Center for Survey Measurement

U.S. Census Bureau

Room 5K020H

Washington, D.C. 20233

(301) 763-6144

Casey.m.eggleston@census.gov

Page 4 of 4