Approved memo from 2017 testing

Generic Information Collection Request

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct an iterative cognitive/usability evaluation of the 2017 Census Test online questionnaire. As a part of this current submission we are seeking approval for usability/cognitive interviews conducted by Census Bureau staff.

Purpose: The usability goal for 2017 is to improve upon the 2016 Census Test instrument. Testing will evaluate the usability of the screen design for different types of devices including desktop/laptop, tablets and smartphones. There was not enough time before the 2016 Census Test production to program all the changes recommended from the prior round of usability testing. This is especially true for the mobile-optimized version. Accordingly, changes recommended from the 2016 testing as well as other design improvements will be tested for the 2017 test. These include programming field lengths which match the length of the data expected, converting all fields which required horizontal scrolling to a vertical layout design, using a keypad for numeric data entries such as ZIP and phone fields, implementing more space between radio buttons, etc.

Population of Interest: The planned usability evaluation will focus on assessing and improving the user experience for the general population. Although usability testing for respondents residing on American Indian reservations and Puerto Rico is important to the 2017 Census Test, these groups will not be asked to participate in research under this clearance request.

Timeline: Testing will be conducted iteratively from June 2016 through February 2017. Recruiting will start early June 2016.

Language: Testing will be conducted in English and Spanish only.

Method: If participants are testing on mobile devices, they will use their own smartphones or tablets to complete the survey. If participants are testing on desktop or laptop devices, those will be provided by Census Bureau or the contractor.

Sample: We plan on testing approximately six people each month for a total of 54 participants. The sample size necessary for this test was determined by experience. We have found that running three participants a day is optimal. We want to do all testing in English in one day and likewise for Spanish, which means that we would run a maximum of six participants a day. If we are not able to test one month, we will conduct a larger study at the end of the testing period to validate changes made during the testing period.

Participants will be from the general public. Different household sizes will be recruited – from single-person households to larger households with six or more people. Some participants will live with people who are not related to them. Participants will not have participated in previous testing of a census online instrument. Participants will have at least one year’s experience with using the Internet for things other than email. For those participants using their own smartphone or tablet, they will know how to use that device to access the Internet.

Recruitment: Participants will be recruited using a combination of word-of-mouth, flyers, online and paper advertisements.

Protocol: Participants will first be asked to complete a demographics questionnaire and mobile experience questionnaire. Then participants will complete the most up-to-date 2017 Census Test online survey using a think-aloud protocol. Participants will either be given mailing materials with their login information containing a Census ID or they will be told to complete the census without a Census ID. After completing the online form, each participant will be asked to complete a satisfaction questionnaire, and asked debriefing questions about different screens.

Use of Incentive: Due to the length and complexity of the task as well as the necessity to travel to test locations, and use of their data plan for mobile devices, we plan to offer an incentive of $60 to offset the costs of participation, such as travel and parking.

Below is a list of materials to be used in the current study. Included is a note on whether each attachment is new or has already been approved by OMB.

1. Protocol used for the study (Enclosure 1) including possible debriefing probes and vignettes

2. Placeholder mailing materials (Enclosure 2) (This is an example from the 2016 Census Test, mocked up for the usability tests.)

3. PowerPoint of the current 2016 Census Test screens as a placeholder (Enclosure 3) (Approved by OMB for the 2016 Census Test.)

4. Demographic questionnaire (Enclosure 4) (Previously approved by OMB for usability testing in the spring of 2015 for the 2015 Census Test.)

5. Mobile experience questionnaire (Enclosure 5) (Previously approved by OMB for usability testing with the November 2015 Compass-NRFU submission.)

6. Satisfaction questionnaire (Enclosure 6) (Previously approved by OMB for usability testing in the spring of 2015 for the 2015 Census Test.)

Length of interview: We estimate 90 minutes per respondent, overall. (Respondents will be screened using the Census Bureau’s generic screener; thus the burden hours for screenings are covered under a separate request.) The total estimated respondent burden for this request is 81 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Elizabeth Nichols

Center for Survey Measurement

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-1724

Elizabeth.May.Nichols@census.gov

2