2015_NCT_OMBPartB final

SUPPORTING STATEMENT

U.S. DEPARTMENT OF COMMERCE

U.S. CENSUS BUREAU

2015 National Content Test

OMB CONTROL NUMBER 0607-XXXX

Part B – Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The 2015 National Content Test (NCT) sample will be selected from housing units in the 50 states, the District of Columbia, and Puerto Rico, using the Census Bureau’s Master Address File. Group quarters will not be included in the sampling frame. To reduce burden on respondents, any housing units selected for the 2015 American Community Survey (ACS) sample (entire year) will be excluded from the sample.

The total sample size for the 2015 NCT is 1,200,000 housing units and the scope of the test is primarily driven by content testing which needs to measure differences for relatively small population groups, such as people who identify as Middle Eastern or North African. The sample design will: (1) use sampling strata to oversample certain race and ethnic groups of interest; (2) oversample areas with relatively high concentrations of people who may sometimes live or stay somewhere else (in order to produce better metrics on within-household coverage); and (3) stratify geographic areas into three groups using high, medium, and low internet response propensities.

There will be six different sample strata for the 2015 NCT:

• Tracts with 10% or more MENA

• Tracts with 10% or more AIAN

• Tracts with 15% or more Asian/NHOPI

• Tracts with 25% or more Black

• Tracts with 45% or more Hispanic/Latino

• All other tracts

The sample is designed to ensure that all major race and ethnic groups are adequately represented in the test.  For example, this includes an oversampling of areas with relatively high concentrations of people who identify as Middle Eastern or North African.  The sample will include addresses in all states and counties in the country (excluding some addresses selected for other census tests to reduce respondent burden: the counties in the Savannah designated marketing area, where the spring 2015 Census Test to study different methods to optimize self-response was held; addresses in the Maricopa county sample from the 2015 Census Test; and the 2015 American Community Survey sample).  Within the MENA sampling stratum, the largest percentage of households are in: California, Michigan, New York, Illinois, and New Jersey.

The universe for sampling the population of Middle Eastern and North African origin in the United States has been expanded to include additional groups that are not specifically included within the current Census Bureau working classification of 19 MENA groups. We have added the following groups to the list for the sampling frame as part of the MENA sampling strata to enable us to cast a wider net: Turkish, Somali, Sudanese/South Sudanese, Mauritanian, Armenian, Sudanese, Djiboutian, Cypriot/Turkish Cypriot, Afghan, Azerbaijani, and Georgian.

This inclusion will help oversample the MENA strata and potentially survey households with respondents of these origins to see how they respond to the race and ethnic question(s), which will afford a broader view of who does or does not identify as MENA. In the end, the 2015 NCT research will provide critical information for us to understand how respondents with origins from various parts of the Middle East, North Africa, and adjacent areas, respond to different versions of the questions that include, or do not include, a distinct MENA category.

A sub-sample of respondents from the 2015 NCT will be selected for a content reinterview, focused on race and origin and within-household coverage, with a goal of assessing accuracy and reliability of the different designs. The reinterview sample will be preselected in advance with 1-in-6 respondents being selected for either the race reinterview or the coverage reinterview. All preselected households for the race reinterview that respond by any mode (Internet, phone, or mail) will be included in the reinterview workload. For the coverage reinterview, all preselected households that respond by internet or phone will be included in the reinterview workload. The coverage reinterview will not include paper questionnaire respondents because no coverage testing is being done on paper.

The Census Bureau estimates an overall self-response rate of 55 percent, with 40 percent of the sample responding online and 15 percent responding using telephone questionnaire assistance or mailing back a paper questionnaire.  These estimates are based on analyses from prior testing, including the 2014 Census Test, 2012 National Census Test, and literature review of other studies of self-response methods. 

Statistically, with an alpha level of 0.10 and power of 0.80, we expect to be able to detect statistically significant differences between two contact strategy panels of 0.5 percentage points or more. For analysis of response data, including race and ethnicity and relationship, we expect to be able to detect statistically significant differences between two panels of 0.1 to 1.0 percentage points (depending on the groups and measures of interest). Finally, for analysis of within-household coverage measures, such as the percentage of count discrepancies and the population count gross difference rate, we expect to be able to detect statistically significant differences between two panels of 1 percentage point or less.

2. Procedures for Collecting Information in 2015 National Content Test

Census Day for the 2015 NCT will be September 1, 2015. The test will employ three response modes: Internet self response on a secure web site, self response on a paper questionnaire, and Telephone Questionnaire Assistance (TQA). The initial mail-out is planned for late August 2015. The first mailing will explain why we are conducting the mandatory 2015 NCT, assure respondents that their answers are confidential, and inform them of the measures we take to keep their personal information secure. The second mailing provides an initial reminder and is sent to all housing units. All contacts after the second mailing are sent to non-respondents only.

The Census Bureau will accept responses from respondents who do not have a unique Census ID at the time of the response. By accepting responses without a unique ID, the Census Bureau is continuing to test its ability to increase self-response rates and reduce respondent burden. Non-ID cases need to be validated against the Census Bureau’s address frame. The key survey measure for this data collection is the number of Non-ID cases successfully resolved (i.e., matched to an address on the frame or assigned to a census block) through automated matching and geocoding during batch processing, which will occur for those Non-ID respondent addresses not matched to a valid record in the Master Address File at the time of response. Additionally, the test provides further opportunities to test improvements to the automated Non-ID processing utilized during previous 2020 Non-ID research and testing in the 2013 National Census Contact Test, the 2014 Census Test, the 2015 Census Test and the 2015 Optimizing Self Response Test.

Respondents are encouraged to respond to the 2015 NCT by Internet but may also provide information by phone or paper questionnaire. The test will be conducted nationally in all 50 U.S. states, the District of Columbia, and Puerto Rico.

3. Methods to Maximize Response

Several panels of the 2015 NCT incorporate the “Internet Push” approach, in which sample housing units will not initially receive a paper questionnaire. Instead, sample households receive either a letter or a postcard that describes the survey and includes instructions about how to respond to the survey online. Traditional paper questionnaires will be sent to non-respondents in these panels about three weeks after the initial contact notification. The paper questionnaire is an opportunity for those without Internet access or a telephone, or with reservations about providing their data online, to respond to the data collection. Households that receive the paper questionnaire will still have the option of responding to the survey online or through the telephone questionnaire assistance line.

Several panels incorporate the “Internet Choice” approach, which provides the questionnaire mailing first or earlier than the “Internet Push” contact strategy.

The nine proposed contact strategies for optimizing self response (OSR) are summarized as follows:

Internet Push (Control): This is the same Internet Push strategy used in our most recent series of self response tests, including the 2014 Census Test and the 2015 Optimizing Self Response Test. This panel will serve as a control panel against which to compare the experimental strategies.

Internet Push with Early Postcard: The objective for this panel is to study the timing of reminders. Our hypothesis is that sending the first reminder sooner (closer to the initial Internet push) will provide for a better connection between the two mailings, and could increase response. We also hypothesize this could reduce the volume of later targeted mailings since responses may be quicker overall.

The objective for the following sequence of three panels is based on recent American Community Survey (ACS) research, which has found depressed self response rates among certain respondents/areas with lower Internet usage. Testing the delivery of the paper questionnaires at various points in the response process will allow us to have complete response measures under several scenarios for the cost/benefit analysis needed to inform 2020 Census planning.

• Internet Push with Early Questionnaire: questionnaire sent at third mailing, one week sooner

• Internet Push with Even Earlier Questionnaire: questionnaire sent at second mailing, two weeks sooner

• Internet Choice: questionnaire sent at first mailing, providing a choice of Internet or paper from the beginning

Internet Push with Postcard as third Reminder: The objective for this panel is to further encourage self response, after the questionnaire mailing, but prior to nonresponse follow-up. Numerous survey research studies have concluded that, while there is a point of diminishing returns, further reminders will usually increase self response rates.

Internet Push Postcard: The objective for this panel is to study the impact of sending a postcard at the first mailing instead of a letter. There are two potential benefits. First is the possible cost savings of printing and mailing a postcard compared to the envelope package (with letter and instruction card). Second is the potential for increased self response because reading a postcard requires less effort by a respondent. In this panel, we will send a letter at the third contact (sent to non-respondents only), in place of a postcard, to vary the types of contacts received.

Internet Push with Early Postcard and second Letter Instead of Mail Questionnaire: The objective for testing an approach in which we do not send a mail questionnaire is to address the high-level goal of greatly reducing paper responses in the 2020 Census. By testing an approach in which we send an Internet push letter in place of a paper questionnaire at the fourth mailing, we will have a more robust set of response measures for informing cost/benefit analyses.

Internet Push with Postcard and Email as 1^st Reminder (same time): The objective for this panel is to determine if we can take advantage of the email addresses in the supplemental contact frame maintained by the Center for Administrative Records Research and Applications. The hypothesis is that by sending a postcard and email at the same time, we may be able to elicit increased response.

	Panel	#1	#2 (7 days later)	#3* (7 days later)	#4* (7 days later)	#5* (7 days later)
O1	Internet Push (Control)	Letter	Postcard	Postcard	Mail Q’nnaire
O2	Internet Push with Early Postcard	Letter	Postcard (3 days sooner)	Postcard	Mail Q’nnaire
O3	Internet Push with Early Questionnaire	Letter	Postcard	Mail Q’nnaire	Postcard
O4	Internet Push with Even Earlier Questionnaire (Low only)	Letter	Mail Q’nnaire	Postcard	Postcard
O5	Internet Choice (Low only)	Mail Q’nnaire	Postcard	Postcard	Mail Q’nnaire
O6	Internet Push with Postcard as 3rd Reminder	Letter	Postcard	Postcard	Mail Q’nnaire	Postcard
O7	Internet Push Postcard	Postcard	Postcard	Letter	Mail Q’nnaire
O8	Internet Push with Early Postcard and 2nd Letter Instead of Mail Q	Letter	Postcard (3 days sooner)	Postcard	Letter
O9	Internet Push with Postcard and Email as 1st Reminder (Same time)	Letter	Postcard and Email (3 days sooner)	Postcard	Mail Q’nnaire

* Targeted only to nonrespondents.

This test will also explore additional options for non-English speakers to complete the questionnaires.

In the two mailings that contain a letter for each Optimizing Self Response strategy, three different methods will be used to encourage response. In particular, by altering the language support provided in the letter, the goal is to study how each method impacts response for respondents with Limited English Proficiency (LEP).

The control panel is similar to the 2014 Census Test design, in which the mailing materials are in English with a single Spanish sentence directing respondents to the website or the telephone assistance line.

One of the goals of language research is to maximize the number of non-English speakers that receive the same message as English speakers prior to going online to respond. Two panels provide equality between the English and Spanish content in the letter and test whether one method is better at eliciting Spanish responses. The swim-lane design has been used in the past, such as with the bilingual questionnaire in the 2010 Census. The dual-sided letter provides English content on one side and Spanish content on the other side. In addition, because research has shown that Spanish-speaking respondents do not always open the mailings because they may not know that language resource information is provided inside, the outgoing envelope for both panels will include the census test website URL and a brief message in both languages.

	Treatments
L1	Control: English materials with Spanish sentence (similar to 2014)
L2	Dual-sided letter: Cover letter redesigned with English on front and Spanish on back, with website and languages (English and Spanish) on outside of envelope
L3	Swim-lane letter: Cover letter redesigned to include both English and Spanish content on the same side, with website and languages (English and Spanish) on outside of envelope

4. Tests of Procedures or Methods

After the initial design phase, the Internet instrument will undergo comprehensive usability testing by internal experts. Usability testing will include observing the participants’ behaviors, noting difficulties and comments, and conducting post-testing interviews to gain qualitative feedback about potential confusion. In addition, quantitative measures will also be gathered, including time to complete the survey and paradata to indicate how respondents visually navigate the form on screens. Testing will incorporate accessibility testing to check for compliance with Section 508 of the U.S. Rehabilitation Act.

All associated contact materials, such as the 2015 NCT invitation and reminders, will undergo an internal expert review. In selecting members of the expert review panel, we sought a diverse group of methodological and subject-matter experts. The review panel includes experts with experience in different areas of expertise, including survey methodology, questionnaire design, and research psychology. Additionally, the content reinterview questionnaires (for race and origin, and for household coverage) both underwent an extensive internal expert review in both English and Spanish languages.

The mail questionnaires will undergo comprehensive cognitive testing using participants recruited from outside the Census Bureau. The cognitive testing will seek feedback on the content and wording of questionnaire items, focusing primarily on the race and Hispanic origin items.

5. Contacts for Statistical Aspects and Data Collection

For more information, contact Erin Love (301) 763-2034, erin.s.love@census.gov.

5