ACS Methods Panel NSC- IMPS and ROIL Tests

1

OMB Nonsubstantive Change Request
Department: Commerce
Agency: U.S. Census Bureau
Title: American Community Survey Methods Panel Tests
OMB Control Number: 0607-0936
Expiration Date: 08/31/2021
The U.S. Census Bureau plans to conduct two tests in 2021 under the American Community
Survey (ACS) Methods Panel Tests clearance. These include testing:
• A pressure seal mailer as the first mailing in place of a mail package.
• An updated letter used by field representatives during nonresponse follow-up
activities.
Each of these tests is described below.
Initial Mailing Pressure Seal Test
Request: The Census Bureau is requesting to test the use of a pressure seal mailer as the first
mailing. The purpose of the test is to understand the effect of sending a pressure seal mailer
instead of the initial mail package on self-response rates. The Census Bureau proposes
conducting a field test as part of production ACS. This field test will require a change in the mail
materials for a subset of the sampled addresses. No additional burden is expected.
Background: Because of the emergent health situation caused by the coronavirus pandemic
and the impact on the Census Bureau’s National Processing Center (NPC), it was necessary to
change the ACS self-response contact strategy beginning in 2020. Included in the changes to the
contact strategy was the use of a pressure seal mailer for the initial mailing instead of a mail
package because of limited staff availability and materials. Observations of return rates of the
pressure seal letter suggest that it may garner higher response than the mail package. This
proposed test will experimentally compare return rates associated with the use of a pressure
seal mailer versus a mail package in the initial mailing.
The coronavirus pandemic resulted in the closure of many operations at NPC. No ACS selfresponse mailings were sent from mid-March through June 2020. When mailings restarted for
the July 2020 panel, the ACS printing, assembly, and mailout operations were negatively
impacted by a reduction in operations and significant shortages of available staff. Therefore,
the ACS program changed the self-response contact strategy to lessen the impact on the survey
operations and to utilize the limited NPC staff available.
The ACS self-response contact strategy is designed to send up to five mailings to eligible
sampled housing units. Starting with the July 2020 panel, the contact strategy changed from up
to five mail contacts per housing unit to two. This was increased to three mailings beginning
with the October 2020 panel. The first mailing ACS typically sends to respondents is the initial
mailing package. This package includes: a large outgoing envelope, a letter, an instruction card,
and a multilingual brochure.

2

Because of inventory shortages and staffing issues brought on by the coronavirus pandemic,
ACS is using two types of initial mailings to send to a sampled unit. In August 2020, a pressure
seal mailer was sent to sampled households instead of the initial mailing package. The pressure
seal mailer was also used in October 2020 and will also be used January through March of 2021.
In August 2020, returns were nominally higher than what was observed in July 2020 (about
6,000 more returns received than at the same point in time in July). We hypothesized that
sending a pressure seal mailer would reduce response rates because a larger, thicker package,
like the initial mailing package, would be more likely to be noticed by respondents. Additionally,
previous Census Bureau research tested the use of pressure seal mailers compared with letters
and postcards and found no difference in response rates. Thus, the observed increase in return
rates in August was surprising.
There are many confounding factors with comparing August 2020 to July 2020 rates, which we
want to mitigate with a planned experimental design. Comparisons across months are typically
not made because historical data show variation in response rates from month to month.
Additionally, at the end of July when the August ACS mailing was sent, an additional reminder
postcard was sent to some housing units for the 2020 Census. This additional mailing plus
increased media coverage of the 2020 Census may have contributed to an increase in ACS
response rates in August. We know that respondents often confuse the two surveys during a
decennial year.
Finally, wording differences between the letters may be contributing to differences in response
rates. The initial mail package lets respondents know that if they are unable to complete the
survey online, then we will send a paper questionnaire in a few weeks. The pressure seal
mailers did not mention this. Because of the coronavirus pandemic, we were unable to mail
paper questionnaires to all households that did not respond early. We theorize that telling
respondents that they will have another opportunity to respond later delays their response.
Additionally, the internet User ID in the initial mail package is included on the instruction card,
not in the letter. In the pressure seal mailer, the User ID is included very clearly in a call-out
box. We theorize that making the log-in instructions clear and easy to find increases internet
response rates.
We propose a controlled experiment, the “Initial Mailing Pressure Seal Test,” to determine the
effect on self-response of a pressure seal mailer in the initial mailing versus a mail package
containing a letter, instruction card, and mulitingual brochure.
Method:
This test will include a control treatment and four experimental treatments: an initial mail
package (control), a modified initial mail package, and three variations of the pressure seal
mailer. Addresses not part of the test will receive the initial mail package. The design of this test
allows us to assess the impact of using a pressure seal mailer instead of an initial mail package
and how information about a paper questionnaire being mailed impacts response rates.

3

Table 1 shows the experiment’s treatments and the main factors in the design.
Table 1. Initial Pressure Seal Mailer Test Experimental Design
Treatment
Mailing Type
User ID in Letter?

Paper Questionnaire
Reference

Control

Initial Mail Package

No

Yes

Treatment 1

Initial Mail Package

No

No

Treatment 2

Pressure Seal Mailer

Yes

Yes

Treatment 3

Pressure Seal Mailer

Yes

No

Treatment 4

Pressure Seal Mailer

Yes

Modified

In Treatment 4, the reference to the paper questionnaire is included, but it is not in bold font
and is moved to the middle of the paragraph rather than the first sentence.
A few other modifications to the pressure seal mailers were made because of the type of
mailing. We are not able to isolate these differences (they are inherent to using a pressure seal
mailer). These differences include the following:
• The return address is located on the top right in the pressure seal mailer (for space
reasons) and under the logo in the initial mail package letter.
• FAQ text is in small font on the front of the pressure seal letter; it is in regular font on
the back of the initial mail package letter.
• The pressure seal mailer is 8.5 inches by 5.5 inches; the initial mail package envelope is
11.5 inches by 6 inches.
• The pressure seal mailer only includes text in English; the initial mail package contains a
a multilingual brochure.
Documentation of all differences between the treatments can be found in Attachment 1.
Analysis Metrics: The Census Bureau will compare self-response rates and final response rates
between treatments to assess the impact of the mailing. We will also monitor the volume of
phone calls to Telephone Questionnaire Assistance and communications received about the
paper questionnaire (concerns that the respondent can’t do the survey online or requests for a
paper questionnaire, for example).
Sample: The monthly ACS production sample has approximately 295,000 addresses, of which
approximately 288,000 have sufficient addresses to which to send mail. Each monthly sample is
divided into 24 groups, where each group contains approximately 12,000 mailable addresses.
Each group is a representative subsample of the entire monthly sample, and each monthly
sample is representative of the entire yearly sample and the country. The Census Bureau will
use two randomly selected groups for each treatment. In total, approximately 96,000 addresses
will be used for the four experimental treatments and 24,000 for the control. The remaining

4

ACS sample will receive production materials. The sample size will be able to detect differences
of approximately 1.74 percentage points between the self-response return rates of two
treatments. The power of the test, which is 80 percent and where α=0.1, assumes a 50 percent
response rate.
Timeline: We are proposing to conduct the Initial Mailing Pressure Seal Test in 2021, in the late
spring or summer (targeting the May panel). The timing of the test is not fixed and is contingent
on when the ACS is able to return to the five-mailing contact strategy (currently planned for
April 2021).
Use of Incentives: None.
Burden: There is no change in burden to the public associated with this test. The test is being
conducted using production ACS sample. No additional contacts are being made.
Contact Information:
Please contact Elizabeth Poehler for further information about this test:
Elizabeth Poehler
Decennial Statistical Studies Division
U.S. Census Bureau
Washington, D.C. 20233
(301) 763-9305
Elizabeth.Poehler@census.gov

5

Regional Office Internet Letter Test
Request: The Census Bureau is requesting to test updated versions of a letter mailed to
nonresponders in the CAPI universe prior to visits from field representatives (FRs). FRs use
letters to further increase response during the third month of data collection. The Census
Bureau proposes to conduct field tests as part of production ACS. This will require a change in
materials for a subset of the sampled addresses. No additional burden is expected.
Background:
In the third and final month of data collection, the Census Bureau contacts residents from a
sample of housing units who did not respond in the first two months of data collection for the
ACS. This phase of data collection is a nonresponse follow-up operation, referred to as CAPI
because data is collected via Computer Assisted Personal Interview. Note that CAPI interviews
may be conducted in person or over the phone. Response rates during the third month of data
collection have been declining over the last five years, with rates now hovering around 80
percent.
There are various reasons for the declining response rates during CAPI. Barriers to cooperation
include privacy concerns or an inaccessible building. One method used to address these barriers
to cooperation is to send informational letters to respondents based on the situation observed
by a FR at an address. For example, an FR who cannot access a locked building may request a
“management letter” designed to alleviate concerns raised by apartment managers when an FR
requests access to their building. In total, the FRs have 12 letters and brochures available to
them, four of which they carry with them and leave at the door (as needed). The remaining
eight are sent to the respondents by the regional office (RO) at the request of the FR.
One of the most often used letters by FRs is the Internet Letter, which reminds the recipient
that they have been selected to participate in the ACS and provides login instructions for
accessing the survey online. Some regional offices send this letter to every address in the CAPI
operation during the first week of data collection (even before an FR visits). Despite the decline
in overall response during the third month of data collection, internet response has been
increasing. In 2013, the portion of internet responses received during CAPI was less than 1
percent. In the last year, it accounted for 4.5 to 6.0 percent.
Based on this increase, the ACS program has decided to send the Internet Letter to all mailable
addresses in the CAPI sample. This pressure-seal mailer will be processed and mailed to CAPI
households from the National Processing Center during the first week of CAPI data collection.
The Internet Letter is expected to increase self-response rates during the third month of data
collection. A response via a self-response mode is less costly than a CAPI interview.
The Census Bureau plans to test content variations to this letter as discussed below.
Method:

6

The Census Bureau plans to test two factors that can potential increase response. One factor is
a statement of urgency on the address side of the pressure seal mailer: a “Past Due” message
or “Required by Law” message. The other factor provides the available response options:
internet only vs. internet and TQA. The experimental design isolates each of the content factors
being studied. There is one control and three experimental treatments.
Table 2. Regional Office Internet Letter Test Experimental Design
Treatment

Address-Side Message

Response Option

Control

Past Due

Internet Only

English Version 1

Past Due

Internet + TQA

English Version 2

Required by Law

Internet Only

English Version 3

Required by Law

Internet + TQA

Analysis Metrics:
This research will answer the following questions.
1. What is the effect of using Past Due message vs. a Required by Law message on the
address side of the pressure seal mailer?
2. What is the effect of using an internet response option only vs. an internet and TQA
response option inside the letter?
For each research question, we will look at self-response rates during the CAPI operation, as
well as the overall CAPI response rate, refusal rate, and other outcomes. The TQA call volume
will also be monitored and compared. Other metrics may be analyzed.
Sample: The monthly ACS production sample has approximately 295,000 addresses, of which
approximately 288,000 have sufficient mailing addresses. The Census Bureau sends the ACS
self-response materials to these addresses. Each monthly sample is divided into 24 groups,
where each group contains approximately 12,000 mailable addresses. Each group is a
representative subsample of the entire monthly sample, and each monthly sample is
representative of the entire yearly sample and the country.
At the time of the CAPI subsample, the number of mailable addresses in each methods panel
group is approximately 2,600 address. The Census Bureau will use six randomly selected groups
for each treatment. The Census Bureau proposes to use six randomly selected methods panel
groups for each treatment. Each treatment will have a sample size of approximately 15,600
addresses. In total, approximately 46,800 addresses will be used for the three experimental
treatments and 15,600 for the control. All mailable CAPI cases are included in the experiment.
The sample size will be able to detect differences of approximately 1.4 percentage points

7

between the self-response return rates of two treatments. The power of the test, which is 80
percent and where α=0.1, assumes a 50 percent response rate.
Timeline: The Census Bureau is proposing to conduct testing of variations to the Internet Letter
in 2021. The test is tentatively proposed as part of the June ACS panel, with mailout of the
letter at the end of July for the August CAPI data collection. The timing of the test is contingent
on when ACS is able to return to the five-mailing contact strategy. It is important to wait until
then because fewer self-response mailings decrease self-response during the first two months
of data collection and therefore change the universe of cases eligible for CAPI.
Use of Incentives: None.
Burden: There is no change in burden to the public associated with this test. The test is being
conducted using production ACS sample. No additional contacts are being made.
Contact Information:
Please contact Elizabeth Poehler for further information about this test:
Elizabeth Poehler
Decennial Statistical Studies Division
U.S. Census Bureau
Washington, D.C. 20233
(301) 763-9305
Elizabeth.Poehler@census.gov
Attachments
Attachment A: Mail Materials for the IMPS Test
Attachment B: Mail Materials for the ROIL Test
Attachment C: IMPS Study Plan
Attachment E: ROIL Study Plan
Attachment F: 30 Day Federal Register Notice

Methods Panel Burden Tracker
3-year American Community Survey Methods Panel Tests (0607-0936) (August 2018 - August 2021)
Total Burden Hours Requested over 3 years: 348,167
Estimated
Number of
Respondents

Burden
Estimated
Hours
Burden Hours
Requested
Used

Methods Panel Submissions for Tests in
August 2018-August 2021

Field Date

Mail Materials Test
2019 Census Test

September 2018
June 2019

480,000

241,440

$0
$0

0
40,000

2019 ACS Due Dates Test

September 2019

144,000

72,432

$0

40,000

March 2020 through
September 2020

2,016,000

1,014,048

2020 Specialized Mail Materials Test

Initial Mailing Pressure Seal Test

May 2021 ACS Panel

Regional Office Internet Letter Test

July and August 2021

Total

Sample
Size (HUs)

62400

56160

Cost to
Respondent

$0

40,000

$0

40,000

$0

40,000

200,000

Notes
There is no change in burden to the public
associated with this test. The test is being conducted
using production ACS sample
0
40,240
There is no change in burden to the public
associated with this test. The test is being conducted
using production ACS sample
0
There is no change in burden to the public
associated with this test. The test is being conducted
0
using production ACS sample
There is no change in burden to the public
associated with this test. The test is being conducted
using production ACS sample. The initial mailing for
0 the May 2021 panel will be sent at the end of April.
There is no change in burden to the public
associated with this test. The test is being conducted
using production ACS sample. The letter will be sent
at the end of July through early August for August
CAPI interviews.
0

40,240

12/14/2020