ACS MP IMPS and ROIL NSC- IMPS Study Plan Attachment C

ACS MP IMPS and ROIL NSC_2021 IMPS Test Study Plan_Attachment C.pdf

American Community Survey Methods Panel Tests

ACS MP IMPS and ROIL NSC- IMPS Study Plan Attachment C

OMB: 0607-0936

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0607-0936 can be found here:

Document [pdf]

Download: pdf | pdf

American Community Survey Research and Evaluation Program
December 10, 2020

ACS Research & Evaluation Analysis Plan (REAP)

2021 Initial Mailing
Pressure Seal Test

THIS PAGE INTENTIONALLY BLANK

U.S. Census Bureau

TABLE OF CONTENTS
1.

INTRODUCTION ................................................................................................................ 1

BACKGROUND .................................................................................................................. 2
2.1
2.2
2.3

Normal ACS Data Collection Strategy .......................................................................... 2
2020 ACS Data Collection Strategy Before the Coronavirus Pandemic ........................ 3
2020 ACS Data Collection Strategy During the Coronavirus Pandemic ........................ 5

LITERATURE REVIEW ......................................................................................................... 6

RESEARCH QUESTIONS AND METHODOLOGY ................................................................... 7
4.1
4.2

Sample Design ............................................................................................................. 7
Experimental Design.................................................................................................... 7
4.2.1
4.2.2
4.2.3
4.2.4
4.2.5

4.3
4.4

Research Questions ..................................................................................................... 9
Analysis Metrics........................................................................................................... 9
4.4.1
4.4.2
4.4.3
4.4.4

Control – Initial Mail Package ........................................................................ 8
Treatment 1 – Initial Mail Package without a Paper Questionnaire
Reference ...................................................................................................... 8
Treatment 2 – Pressure Seal Letter with a Paper Questionnaire
Reference ...................................................................................................... 8
Treatment 3 –Pressure Seal Letter without a Paper Questionnaire
Reference ...................................................................................................... 8
Treatment 4 – Pressure Seal Letter with a De-emphasized Paper
Questionnaire Reference .............................................................................. 8

Self-Response Return Rates .......................................................................... 9
Final Response Rates ................................................................................... 10
Cost Analysis ............................................................................................... 11
Additional Analysis Metrics ......................................................................... 11

ASSUMPTIONS AND LIMITATIONS .................................................................................. 12
5.1
5.2

Assumptions .............................................................................................................. 12
Limitations ................................................................................................................. 12

TABLE SHELLS.................................................................................................................. 12

POTENTIAL CHANGES TO ACS ......................................................................................... 14

REFERENCES ................................................................................................................... 14

Appendix A. Images of the August 2020 Pressure Seal Letter ................................................ 16
Appendix B. Images of the January 2021 Pressure Seal Letter ............................................... 18
Appendix C. Images of the Initial Mail Package (Control) ....................................................... 22
Appendix D. Images of the Experimental Treatments (Treatment 1 – 4) ................................ 27
Appendix E. Comparison of Treatments................................................................................. 35
ii

U.S. Census Bureau

1. INTRODUCTION
Beginning in March 2020, the U.S. Census Bureau’s National Processing Center (NPC) operations
were impacted by the coronavirus (COVID-19) pandemic. The NPC did not send any American
Community Survey (ACS) mailings from mid-March through June 2020.1
In late June, staffing levels at the NPC were sufficient to resume ACS mailings. However,
because of the interruption in operations and continued staffing constraints, it was necessary
to reduce the number of ACS mailings. Starting with the July 2020 panel, the ACS self-response
mail contact strategy changed to two mail contacts per sampled housing unit.2 For the October
2020 panel, a third mailing was added. In addition to the staffing shortages, there was an
insufficient supply of pre-assembled mail packages for the remainder of the calendar year.
Therefore, the Census Bureau decided to alternate between two types of initial mailings. The
July, September, November, and December panels received the initial mailing package used in
the normal mailout strategy. The August and October panels received a pressure seal letter as
the initial mailing, inviting them to respond online.
We created the new pressure seal letter by combining content from the initial mailing package
and reminder letter and by modifying that content to fit the new format and mailout strategy.
While this approach solved the staffing and supply issue, there were some concerns about how
effective the pressure seal letter would be compared to the typical initial mailing package.
Intuitively, it seems that a larger initial mail package would be more noticeable than a pressure
seal letter. Thus, we anticipated a decrease in response with the smaller mailing. Previous
testing showed no significant difference in the self-response rates when comparing a pressure
seal letter to a letter in a regular-sized envelope (Risley 2017).
Surprisingly, data from the August 2020 panel, which was sent the pressure seal letter, showed
higher response rates compared to the July 2020 panel which received the initial mail package.
However, the response rates for August 2020, were still lower than the August 2019 response
rates. There are several factors that confound these comparisons. Differences between July and
August panels may be due to the difference in the mail type, but could also be a consequence
of an additional mailing sent for the 2020 Census one week prior to the August ACS mailing,
other 2020 Census communications, or differences in respondent behavior due to the
pandemic. Differences between 2020 and 2019 may also have occurred because we did not
send a reminder letter as a second mailing in 2020. Additionally, comparisons are limited due to
wording differences in the mailings (e.g., the August 2020 pressure seal letter did not mention a
paper questionnaire would be coming, but both the July 2020 mailing and August 2019 mailing

1
2

The April, May, and June 2020 panels did not receive any mail contacts.
Under normal circumstances, the mail contact strategy includes up to five mail contacts.

U.S. Census Bureau

did). For the October 2020 panel, we again sent a pressure seal letter as the initial mailing, and
again we saw an increase in responses.3
The purpose of this test is to determine if using a pressure seal letter for the initial mailing
results in higher response rates in a situation where we can control for some of the above
confounding factors such as wording differences, lack of a second mailing, and the effect of
2020 Census communications. If response rates using the pressure seal letter are not
significantly lower than response rates using the initial mail package, then it would be cost
effective to change the ACS contact strategy because a pressure seal letter is cheaper than an
initial mail package.

2. BACKGROUND
This section presents information on:
1.
2.
3.
2.1

the normal ACS data collection strategy
the 2020 ACS data collection strategy before the coronavirus pandemic
the 2020 ACS data collection strategy during the coronavirus pandemic
Normal ACS Data Collection Strategy

When the NPC returns to normal operations, the 2021 ACS mail contact strategy will follow the
plan outlined below. This test will not occur until the normal ACS data collection strategy is in
place. Figure 1 outlines the 2021 ACS mail contact strategy.
Figure 1. 2021 ACS Mail Contact Strategy

The first two mailings are sent to all mailable addresses in the monthly sample. The first mailing
is a package that includes a letter, a multilingual brochure, and a card with instructions on how
to respond via the internet. The letter contains an invitation to participate in the ACS online and

The increase was nominal; there was no statistical testing performed.

U.S. Census Bureau

more information in a frequently asked questions (FAQs) format on the back of the letter. The
letter also provides the Telephone Questionnaire Assistance (TQA) phone number if anyone at
the address has any questions. A week later, the same addresses are sent a second mailing
(reminder letter as a pressure seal letter).
Responding addresses are removed from the address file after the second mailing to create a
new mailing universe of nonrespondents; these addresses are sent the third and fourth
mailings.4 The third mailing is a package that includes a letter, a paper questionnaire, and a
business reply envelope. Four days later, these addresses are sent a fourth mailing (reminder
postcard) which encourages them to respond.
After the fourth mailing, responding addresses are again removed from the address file to
create a new mailing universe of nonrespondents. The remaining sample addresses are sent the
fifth mailing (a more urgent final reminder letter with a due date in a pressure seal letter).
Two to three weeks later, responding addresses are removed and unmailable and undeliverable
addresses (from the initial sample) are added to create the universe of addresses eligible for
the Computer-Assisted Personal Interview (CAPI) nonresponse followup operation.5 Of this
universe, a subsample is chosen to be included in the CAPI operation. Field representatives
attempt to call to interview those selected for CAPI by phone. If they cannot reach them by
phone, or do not have a phone number, they visit the addresses to conduct in-person
interviews.
Additional information can be found in the ACS Design and Methodology Report (U.S. Census
Bureau 2014).
2.2

2020 ACS Data Collection Strategy Before the Coronavirus Pandemic

The data collection strategy for 2020 was not planned to be different in terms of the number
and types of mailings. However, the language of some of the materials was changed to address
potential respondent confusion between the ACS and the 2020 Census.
Table 1 outlines the changes that were made to help distinguish the ACS from the 2020 Census.
There were no changes made to the first mailing instruction card and multilingual brochure,
third mailing questionnaire and return envelope, the fourth mailing, or the interior of the fifth
mailing. Due to the coronavirus pandemic, the mailings in Table 1 were sent only for the March
2020 panel; some of the language distinguishing the 2020 Census from the ACS was used in
mailings developed during the pandemic.

Addresses deemed “undeliverable as addressed” (UAA) by the United States Postal Service are also removed
from the address files for subsequent mailings.
5
CAPI interviews start at the beginning of the month following the fifth mailing.

U.S. Census Bureau

Table 1. Differences Between the Normal ACS Materials and ACS Materials During the 2020
Census
Mailing Mail Material What Was Changed
Specific Wording Differences
First
Envelope
Mailing
Letter

Second Exterior of
Mailing Pressure Seal
Letter
Interior of
Pressure Seal
Letter
Third
Envelope
Mailing
Letter

Form number is
different to help with
processing
2020 Census text
(front of letter)

2020 Census FAQs
included (back of
letter)
Mentions the ACS on
the exterior
2020 Census text
included

Form number is
different to help with
processing
2020 Census text
(front of letter)

Added text: “The American Community
Survey is not the 2020 Census. This
survey asks questions about topics not
on the 2020 Census, such as veteran
status, transportation, and internet
access.”
Added two additional FAQs about the
2020 Census
Added text:
U.S. Census Bureau
American Community Survey
Added text: “Some households,
including yours, will receive both the
American Community Survey and the
2020 Census this year.”

Added text: “This year, the Census
Bureau is also conducting the 2020
Census. The American Community
Survey is different from the 2020
Census.”
Also changed “Your response to this
survey is required by law” to “Your
response to the American Community
Survey is required by law”.

Fifth
Exterior of
Mailing Pressure Seal
Letter

2020 Census FAQs
included (back of
letter)
Mentions the ACS on
the exterior

Added two additional FAQs about the
2020 Census
Added text:
U.S. Census Bureau
American Community Survey

U.S. Census Bureau

2.3

2020 ACS Data Collection Strategy During the Coronavirus Pandemic

Due to the temporary shutdown of operations at NPC between mid-March and June, the
mailing contact strategy for July 2020 through the beginning of 2021 was changed from five
mailings to two or three mailings. July through September panels were sent two mailings, while
the October panel and subsequent months will receive three mailings.
In addition to the reduction in the number of mailings, the type of mailing sent to each panel
varied due to supply and staffing shortages. Depending on the panel, the first mailing could be
an initial mail package or a pressure seal letter. The second mailing varied within panel (July to
September), with nonrespondents receiving either a paper questionnaire package or a
reminder pressure seal letter. Table 2 outlines the mailing contact strategy for the second half
of 2020. Changes to staffing may result in changes to this plan.
Table 2. Mailing Contact Strategy July 2020 to December 2020
Panel
First Mailing
Second Mailing
Paper Questionnaire Package or
Jul 2020
Initial Mail Package
Pressure Seal Letter
Paper Questionnaire Package or
Aug 2020
Pressure Seal Letter
Pressure Seal Letter
Paper Questionnaire Package or
Sep 2020
Initial Mail Package
Pressure Seal Letter
Oct 2020
Pressure Seal Letter
Paper Questionnaire Package
Nov 2020
Initial Mail Package
Paper Questionnaire Package
Dec 2020
Initial Mail Package
Paper Questionnaire Package

Third Mailing
None
None
None
Pressure Seal Letter
Pressure Seal Letter
Pressure Seal Letter

We combined content from the first and second 2020 ACS mailing materials to create the new
initial pressure seal letter. We modified some of the content to fit a pressure seal letter, such as
moving the Census Bureau address to the upper right corner and condensing the FAQs that
appear on the back of the initial mail package letter to the bottom in small font. We also
modified the benefits listed in the second paragraph to address potential new concerns due to
the pandemic. In addition, we dropped the reference to a paper questionnaire.6 See Appendix A
for images of the pressure seal letter used as the first mailing in August 2020.
The contact strategies for 2021 have not yet been decided. The current plan is to send three
mailings and use a pressure seal letter for the first mailing for at least January 2021. The contact
strategy will most likely mirror that of the October 2020 panel. Ideally, we will send the initial
mail package (instead of the pressure seal letter) and add a fourth mailing starting in February,

For the July, August, and September panels, it was not true for every housing unit that a paper questionnaire
would be sent. Some housing units were sent the questionnaire while others received a pressure seal letter
reminding them to respond online. See Table 2.

U.S. Census Bureau

but there are concerns that these mailings may not be feasible. In April 2021 we hope to return
to normal five-mailings strategy.
Since the paper questionnaire will be sent to all addresses in 2021, unlike in 2020, we designed
a new initial pressure seal letter to use starting January 2021 that includes a reference to the
paper questionnaire. The 2021 initial pressure seal letter also removes the 2020 specialized
language that distinguishes the 2020 Census from the ACS. See Appendix B for images of the
pressure seal letter to be used as the first mailing in January 2021 and how it differs from the
one used in August 2020.

3. LITERATURE REVIEW
In 2014, the Census Bureau collaborated with Reingold, Inc. to research ways to improve the
ACS mail materials. Focus groups and interviews were conducted in which participants were
asked to rank ACS mail packages as to how likely they were to notice and open the packages.
Pressure seal letters were ranked as one of the most effective. Participants saw these letters as
more official (government-like) and considered them more confidential (Reingold 2014).
Statistics Canada used a pressure seal letters for the 2016 Canadian Census and also found that
pressure seal letters were considered more official than a traditional envelope in the eyes of
the public (Graziadei 2016).
Statistics Canada found the following benefits of using pressure seal letters:
•
•
•

The letters offered the ability to include personal or confidential information (e.g., login
information).7
The use of pressure seal letters reduced both costs and paper waste due to the
elimination of a separate envelope.
The printer had the capacity to produce 1.6 million pressure seal letters a day,
increasing the efficiency of mail assembly.

In 2017, the Census Bureau conducted a field test to see if replacing some of the ACS mail
materials (reminder letters and postcards) with pressure seal letters would affect response
rates. The results of that test showed that replacing the reminder letter (second mailing) with a
pressure seal letter would not negatively impact self-response and would be a cost-saving
change (Risley 2017). This test did not however test replacing the initial mail package with a
pressure seal letter.

As opposed to a postcard.

U.S. Census Bureau

4. RESEARCH QUESTIONS AND METHODOLOGY
This section discusses the sample design, experimental design, and research questions and
metrics of the 2020 Initial Mailing Pressure Seal (IMPS) Test. The goal of this test is to assess if
sending a pressure seal letter as the initial mailing affects response metrics.
4.1

Sample Design

The 2021 IMPS Test will be conducted using the May 2021 ACS production sample.8 The
monthly ACS production sample consists of approximately 290,000 housing unit addresses and
is divided into 24 nationally representative groups (referred to as methods panel groups) of
approximately 12,000 addresses each. Each of the treatments in this test will use two randomly
assigned methods panel groups (approximately 24,000 mailing addresses per treatment). The
control treatment will use two methods panel groups and will receive production ACS
materials, but will be sorted and mailed separately from production. All remaining methods
panel groups will receive production ACS materials.
4.2

Experimental Design

This test will include a control and four experimental treatments: an initial mail package
(control), an initial mail package with a modified letter, and three modified pressure seal
letters. Addresses not part of the test will receive the initial mail package production materials.9
In addition to testing for the effect of using a pressure seal letter versus an initial mail package,
we will test the effect of mentioning that a paper questionnaire will be sent in a later mailing.
The pressure seal letter used in the August 2020 panel did not include a sentence mentioning
the paper questionnaire because, due to staff shortages at NPC, not every address received the
paper questionnaire package as the third mailing (see Table 2).10 Therefore, we are also testing
the effect of the paper questionnaire sentence on response metrics.
We will conduct this test when the ACS mailing strategy returns to the five-mailing contact
strategy. Having all five mailings will account for how changes in the first mailing interact with
subsequent mailings, especially the second mailing, which was not sent in the August 2020 and
October 2020 panels. All of the treatments will adhere to the same overall ACS mailing strategy
(the number of mailings, types of mailings, and timing of mailings).
8

The current plan is to conduct this test in May 2021, but we may conduct it in a later panel depending on the
contact strategy. We will only run this test once the ACS returns to a five-mailing contact strategy.
9
Previous research indicates that in ACS experiments, postal procedures alone could cause a difference in
response rates at a given point in time between smaller experimental treatments and larger control treatments,
with response for the small treatments having a negative bias (Heimel 2016). Thus, the treatments are
structured to be of similar size, and the control will be sorted and mailed separately from the rest of production
cases so that the control and treatments have similar mail delivery timing.
10
Due to staff shortages at NPC from the coronavirus pandemic, only around 60 percent of nonresponders were
sent a paper questionnaire. The remaining 40 percent were sent a pressure seal reminder letter.

U.S. Census Bureau

4.2.1

Control – Initial Mail Package

As mentioned in Section 2.1, the initial mail package includes a letter inviting residents of the
sampled address to participate in the ACS online and stating that a paper questionnaire will be
sent in a few weeks to those unable to respond online. Along with the letter, the package also
contains a multilingual brochure and an instruction card. The address mailing label of the
instruction card contains the User ID, which is used when responding online. See Appendix C for
images of all contents of the Initial Mail Package.
4.2.2

Treatment 1 – Initial Mail Package without a Paper Questionnaire Reference

Treatment 1 will be an initial mail package with a new letter that excludes the reference to a
paper questionnaire. The only difference between the contents of this treatment and the
control will be the paper questionnaire sentence in the letter. It is possible that self-response
increased in August 2020 due to recipients being unaware of the paper response option.11 See
Appendix D for images of the Treatment 1 letter.
4.2.3

Treatment 2 – Pressure Seal Letter with a Paper Questionnaire Reference

Treatment 2 will be a pressure seal letter that includes a User ID and a reference to the paper
questionnaire. The wording of this pressure seal letter will mirror the wording in the control
initial mail package letter as closely as possible. However, the User ID will be included in the
letter. We decided not to test a pressure seal letter without the User ID included, because we
would never implement a pressure seal letter in production that did not include a User ID.12
Additionally, FAQs that appear on the back of the initial mail package letter were condensed
and included in small font in the pressure seal mail letter. See Appendix D for images of the
Treatment 2 pressure seal letter.
4.2.4

Treatment 3 –Pressure Seal Letter without a Paper Questionnaire Reference

Treatment 3 will be a pressure seal letter with the reference to a paper questionnaire removed.
The only difference between the wording of this treatment and Treatment 2 will be the paper
questionnaire sentence. Treatment 3 will be the closest match to what was sent out in August
2020 (see Appendix A). See Appendix D for images of the Treatment 3 pressure seal letter.
4.2.5

Treatment 4 – Pressure Seal Letter with a De-emphasized Paper Questionnaire
Reference

Treatment 4 will be a pressure seal letter that includes a reference to the paper questionnaire
that is de-emphasized (un-bolded and in the middle of the paragraph). Besides the paragraph
11

Treatment 1 will use the same envelope and contain the same multilingual brochure and instruction card as the
control (see Appendix C for images).
12
All pressure seal letters will include a User ID on the letter. A User ID is provided in the initial mail package on the
instruction card, but is not provided on the letter itself due to how the initial mail package is printed and
assembled.

U.S. Census Bureau

containing the paper questionnaire reference, the rest of the wording of this treatment will be
the same as Treatment 2 and Treatment 3. See Appendix D for images of the Treatment 4
pressure seal letter.
See Appendix E for a comparison of the treatments.
4.3

Research Questions

The 2021 IMPS Test will answer the following questions:
RQ1. What is the impact of using a pressure seal letter in the initial mailing on self-response
return rates (overall and by mode)?
RQ2. What is the impact of removing the reference to a paper questionnaire on self-response
return rates (overall and by mode)?
RQ3. What is the impact of de-emphasizing the reference to a paper questionnaire on selfresponse return rates (overall and by mode)?
RQ4. What is the overall impact of each of the experimental treatments on final response rates
and data collection costs (overall and by mode)?
4.4

Analysis Metrics

All self-response analyses, except for the cost analysis, will be weighted using the ACS base
sampling weight (the inverse of the probability of selection). Cases in the CAPI subsample will
have a CAPI subsampling factor that will be multiplied by the base weight, unless they are selfresponses. The sample size will be able to detect differences of approximately 1.25 percentage
points between the self-response return rates of the experimental treatments (with 80 percent
power and α=0.1). We will use a significance level of α=0.1 when determining significant
differences between treatments.
4.4.1

Self-Response Return Rates

To determine the effect of each treatment on self-response, we will calculate the self-response
return rates at two points in time in the data collection cycle – before the third mailing and
before the start of CAPI. Self-response return rates will be calculated for total self-response
combined and separately for internet, mail, and TQA responses. If there are no significant
differences in TQA rates between treatments, we may combine mail and TQA rates.
The self-response return rates will be calculated using the following formula:

U.S. Census Bureau

Self-Response
Return Rate

Number of mailable and deliverable sample addresses that
either provided a non-blank13 return by mail or TQA, or a
complete or sufficient partial14 response by internet
Total number of mailable and deliverable sample addresses15

* 100

To assess the impact on self-response return rates of using a pressure seal letter (RQ1), we will
compare the control to Treatment 2 and compare Treatment 1 to Treatment 3. To assess the
impact on self-response return rates of removing the reference to a paper questionnaire (RQ2),
we will compare the control to Treatment 1 and compare Treatment 2 to Treatment 3. To
assess the impact on self-response return rates of de-emphasizing a reference to a paper
questionnaire (RQ3), we will compare both Treatment 2 and Treatment 3 to Treatment 4. See
Appendix E for a table of the comparisons.
If we receive more than one return from an address, then the return received first will be
considered the response. In the rare case that we receive two returns on the same day, then
we will choose the mail return.
4.4.2

Final Response Rates

To determine the effect of the experimental treatments on overall response to the survey, we
will calculate final overall response rates and how each response mode contributes to the
overall final response rate. The final response rates will be calculated using the following
formula:

Final Response
Rate

Number of eligible sample addresses that either provided a
non-blank return14 by mail or TQA, a complete or sufficient
partial15 response by internet, or a complete CAPI interview
Total number of sample addresses eligible to reply to the survey
and not sampled out of CAPI

* 100

The denominator does not include UAAs (unless the address did respond or is in the CAPI
sample) and does not include addresses that are found to be a business, demolished, under
construction, etc.

A blank form is a form in which there are no persons with sufficient response data and there is no telephone
number listed on the form.
14
A sufficient partial internet response is one in which the respondent reached the Pick Next Person screen for a
household with two or more individuals on the roster or has gone through the place of birth question for a 1person household.
15
We will remove addresses deemed to be UAA by the Postal Service if no response is received.
13

U.S. Census Bureau

If we receive more than one response from an address, then the response received first will be
considered the response. In the rare case that we receive two responses on the same day, then
we will choose the response in the following order: (1) mail, (2) TQA, (3) internet, and (4) CAPI.
4.4.3

Cost Analysis

In evaluating the different experimental treatments, it is not sufficient to compare only the selfresponse return rates and final response rates. If one or more of the experimental treatments
increases self-response, subsequent mailings and the CAPI workloads (which cost more per case
to complete than self-response cases) would be smaller.
Because the only changes will occur in the initial mailing, we are interested in the effect on selfresponse prior to determining the second mailing universe (M2). An increase in self-response
before the M2 cut decreases the number of mailing pieces that need to be sent out and
reduces cost.16 Calculating the return rates before the M2 cut and CAPI cut will give us an idea
of how the experimental treatments would affect operational and mailing costs if they were
implemented into a full ACS production year.
We will conduct a cost analysis to estimate the costs of putting each of the treatments into
production. We may not perform a cost analysis for the control or Treatment 1 if their selfresponse return rates are significantly lower than the other treatments because the initial mail
package is more expensive than a pressure seal letter. Since the cost model uses projected
workload differences to project survey costs, this part of the analysis will not be weighted.
4.4.4

Additional Analysis Metrics

Prior to answering the research questions, we will investigate the underlying data to ensure
there are no differences between treatments in metrics that could affect the research question
results. We will examine the rate at which addresses are flagged by the USPS as being UAA, as
return rates and response rates can be influenced by UAA rates.
If there are significant differences between treatments in how each response mode contributes
to the overall final response rate, then we will look at major demographic distributions of
Person 1 (who is typically the respondent) from sufficiently complete responses. We assume
respondents in all of the treatments will have similar demographic characteristics, but mode
differences in response rates could indicate an experimental difference in the treatments. Only
significant findings from these analyses will be reported.
We will also examine TQA workload during the months the test is active. Not telling recipients
that a paper questionnaire will be sent in a few weeks could increase calls to TQA. Since the
TQA workload is collected daily for all calls and not differentiated by panel, we cannot know for
certain if a change in workload is because of a particular treatment or the test overall.
16

The M2 mailing universe cut occurs before sending the third mailing (paper questionnaire).

U.S. Census Bureau

However, we will still examine the workload and compare it to previous months and years to
see if there are any substantial changes during the months the test is active.

5. ASSUMPTIONS AND LIMITATIONS
5.1

Assumptions
1. A single ACS monthly sample is representative of an entire year (twelve panels) and
the entire frame sample, with respect to both response rates and cost, as designed.
2. A single methods panel group (1/24 of the full monthly sample) is representative of
the full monthly sample, as designed.
3. We assume that there is no difference between treatments in mail delivery timing or
subsequent response time. The treatments will have the same sample size and use
the same postal sort and mailout procedures. Previous research indicated that
postal procedures alone could cause a difference in response rates at a given point
in time between experimental treatments of different sizes, with response for the
smaller treatments lagging (Heimel 2016).

5.2

Limitations
1. Group quarters and sample housing unit addresses from remote Alaska and Puerto
Rico are not included in the sample for the test.
2. The cost analysis uses estimates to make cost projections. These estimates do not
account for monthly variability in production costs such as changes in staffing,
production rates, or printing price adjustments.
3. There are materials and information in the initial mail package that will not be
provided with the pressure seal letter (e.g., the multilingual brochure). Thus, we
cannot draw separate conclusions about the effects of the mailer-type distinguished
from the inclusion of extra materials.
4. It is unknown how long the coronavirus pandemic will last and what effects it could
have on the implementation of this field test. A coronavius exposure at NPC could
slow or stop operations and delay this test. In addition, impacts of the pandemic on
respondents’ lives may affect their likelihood of responding or how they respond
even when a normal mailing strategy resumes.

6. TABLE SHELLS
Below are samples of tables that will be used in the final report to show results from this test.
12

U.S. Census Bureau

Table 3. Sample Table for Overall Self-Response Return Rates
Point in Data Collection Cycle
Treatment X Treatment Y
Before Third Mailing
%%.%
%%.%
Before CAPI
%%.%
%%.%

Difference
%%.% (#.#)
%%.% (#.#)

P-value
#.##
#.##

Source: U.S. Census Bureau, American Community Survey, 2021 Initial Mailing Pressure Seal Test, DRB #########
Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a
statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level.

Table 4. Sample Table for Internet Self-Response Return Rates
Point in Data Collection Cycle
Treatment X Treatment Y
Before Third Mailing
%%.%
%%.%
Before CAPI
%%.%
%%.%

Difference
%%.% (#.#)
%%.% (#.#)

P-value
#.##
#.##

Table 5. Sample Table for Mail Self-Response Return Rates
Point in Data Collection Cycle
Treatment X Treatment Y
Before Third Mailing
%%.%
%%.%
Before CAPI
%%.%
%%.%

Difference
%%.% (#.#)
%%.% (#.#)

P-value
#.##
#.##

Table 6. Sample Table for TQA Self-Response Return Rates
Point in Data Collection Cycle
Treatment X Treatment Y
Before Third Mailing
%%.%
%%.%
Before CAPI
%%.%
%%.%

Difference
%%.% (#.#)
%%.% (#.#)

P-value
#.##
#.##

Table 7. Sample Table for Final Response Rates
Treatment X
Production
Overall Response
%%.%
%%.%
Internet
%%.%
%%.%
Mail
%%.%
%%.%
TQA
%%.%
%%.%
CAPI
%%.%
%%.%

Difference
%%.% (#.#)
%%.% (#.#)
%%.% (#.#)
%%.% (#.#)
%%.% (#.#)

P-Value
#.##
#.##
#.##
#.##
#.##

U.S. Census Bureau

Table 8. Sample Table for Self-Response Return Rates at Closeout for Cost Analysis
Mode
Control
Treatment 1 Treatment 2 Treatment 3 Treatment 4
Self-Response
%%.%
%%.%
%%.%
%%.%
%%.%
Internet
%%.%
%%.%
%%.%
%%.%
%%.%
Mail
%%.%
%%.%
%%.%
%%.%
%%.%
TQA
%%.%
%%.%
%%.%
%%.%
%%.%
Source: U.S. Census Bureau, American Community Survey, 2021 Initial Mailing Pressure Seal Test, DRB #########
Note: Minor additive discrepancies are due to rounding. Standard errors are in parentheses. An asterisk (*) indicates a
statistically significant result. Significance was tested based on a two tailed t-test at the α=0.1 level.

Table 9. Sample Table for ACS Annual Total Cost Estimates
Treatment
Estimate of the Cost Difference from Control
Treatment 1
$#,###,###
Treatment 2
$#,###,###
Treatment 3
$#,###,###
Treatment 4
$#,###,###
Source: U.S. Census Bureau, American Community Survey, 2021 Initial Mailing Pressure Seal Test, DRB #########
Note: Negative values are denoted with parentheses and indicate a cost savings.

7. POTENTIAL CHANGES TO ACS
This test could result in a change to the ACS contact strategy if the response rates using the
pressure seal letter are significantly higher or not significantly different than the response rates
using the initial mail package. Specifically, the ACS contact strategy would change whether the
first mailing is a pressure seal letter instead of a mail package. Alternatively, if the pressure seal
letter is not shown to be cost effective, but the self-response rates for the treatments that do
not mention the paper questionnaire are higher than the treatments that do mention it, the
initial mail package letter could be modified to remove the reference to the paper
questionnaire.

8. REFERENCES
Graziadei, C. (2016). “Overview of the 2016 Canadian Census” [PowerPoint slides]. Informal
meeting with division chiefs at the US Census Bureau in October, 2016.
Heimel, S. (2016). “Postal Tracking Research on the May 2015 ACS Panel.” 2016 American
Community Survey Research and Evaluation Report Memorandum Series #ACS16-RER01, April 1, 2016.
Reingold, Penn Schoen Berland, Decision Partners, (2014). American Community Survey
Messaging and Mail Package Assessment Research: Cumulative Findings. Washington
DC: U.S. Census Bureau. Retrieved on September 10, 2020 from
https://www.census.gov/library/working-papers/2014/acs/2014_Walker_02.html
14

U.S. Census Bureau

Risley, M., Barth, D., Cheza, K., Rabe, M. (2017). “2017 Pressure Seal Mailing Materials Test,”
2017 American Community Survey Research and Evaluation Report Memorandum Series
# ACS17-MP-05, December 4, 2017.
U.S. Census Bureau (2014). “American Community Survey Design and Methodology,” Retrieved
on November 18, 2020 from http://www2.census.gov/programssurveys/acs/methodology/design_and_methodology/acs_design_methodology_ch12_2
014.pdf.

U.S. Census Bureau

Appendix A. Images of the August 2020 Pressure Seal Letter
Figure 2. Inside of August 2020 Pressure Seal Letter

U.S. Census Bureau

Figure 3. Outside of August 2020 Pressure Seal Letter

U.S. Census Bureau

Appendix B. Images of the January 2021 Pressure Seal Letter
Figure 4. Inside of January 2021 Pressure Seal Letter

U.S. Census Bureau

Figure 5. Outside of January 2021 Pressure Seal Letter

U.S. Census Bureau

Figure 6. Differences Between Inside of August 2020 and January 2021 Pressure Seal Letters

U.S. Census Bureau

Figure 7. Differences Between Outside of August 2020 and January 2021 Pressure Seal Letters

U.S. Census Bureau

Appendix C. Images of the Initial Mail Package (Control)
Figure 8. Front of Initial Mail Package Envelope

Figure 9. Back of Initial Mail Package Envelope

U.S. Census Bureau

Figure 10. Front of Initial Mail Package Letter

U.S. Census Bureau

Figure 11. Back of Initial Mail Package Letter

U.S. Census Bureau

Figure 12. Outside of Multilingual Brochure

Figure 13. Inside of Multilingual Brochure

U.S. Census Bureau

Figure 14. Front of Instruction Card

Figure 15. Back of Instruction Card

U.S. Census Bureau

Appendix D. Images of the Experimental Treatments (Treatment 1 – 4)
Figure 16. Front of Treatment 1 Letter

U.S. Census Bureau

Figure 17. Back of Treatment 1 Letter

U.S. Census Bureau

Figure 18. Front of Treatment 2 Pressure Seal Letter

U.S. Census Bureau

Figure 19. Back of Treatment 2 Pressure Seal Letter

U.S. Census Bureau

Figure 20. Front of Treatment 3 Pressure Seal Letter

U.S. Census Bureau

Figure 21. Back of Treatment 3 Pressure Seal Letter

U.S. Census Bureau

Figure 22. Front of Treatment 4 Pressure Seal Letter

U.S. Census Bureau

Figure 23. Back of Treatment 4 Pressure Seal Letter

U.S. Census Bureau

Appendix E. Comparison of Treatments
Table 10. Comparison of Treatments
Item
Description

Comparison(s)

Control
Treatment 1
Initial Mail Package used Initial Mail Package
in production.
without a paper
questionnaire reference.
Treatment 1
Control
Treatment 2
Treatment 3

Return Address*
Response call out box‡
Questionnaire Reference†

Under logo
No User ID
If you are unable to
complete the survey
online, we will send you
a paper questionnaire
in a few weeks.
‡
Response Motivation –
After paper
“The Census Bureau is using questionnaire reference
the internet to collect this
information to conserve
natural resources, save
taxpayers’ money, and
process data more
efficiently.”
Legal Text*
In FAQs on back of letter

Size (in inches)*

Envelope - 11.5x6

Under logo
No User ID
Sentence omitted.

Own paragraph

Treatment 2
Pressure Seal Letter with
paper questionnaire
reference.
Control
Treatment 3
Treatment 4
Upper right corner
Contains user ID
If you are unable to
complete the survey
online, we will send you
a paper questionnaire
in a few weeks.
After paper
questionnaire reference

In FAQs on back of letter In small print at bottom
of letter, text
rearranged to fit
Envelope - 11.5x6
Bi-fold mailer -8.5x5.5

Treatment 3
Pressure Seal Letter
without paper
questionnaire reference.
Treatment 1
Treatment 2
Treatment 4
Upper right corner
Contains user ID
Sentence omitted.

Own paragraph

In small print at bottom
of letter, text
rearranged to fit
Bi-fold mailer - 8.5x5.5

*Difference will not be tested and is a consequence of changing to pressure seal letter.
Ɨ Difference will be tested.
‡Difference will be tested in conjunction with other changes

U.S. Census Bureau

Treatment 4
Pressure Seal Letter with
de-emphasized paper
questionnaire reference.
Treatment 2
Treatment 3
Upper right corner
Contains user ID
If you are unable to
complete the survey
online, we will send you a
paper questionnaire in a
few weeks (no bold).
Before the paper
questionnaire reference

In small print at bottom
of letter, text rearranged
to fit
Bi-fold mailer - 8.5x5.5

File Type	application/pdf
Subject	American Community Survey
Author	U.S. Census Bureau
File Modified	2020-12-14
File Created	2020-12-14