Cognitive & Usability Testing Plan, ACS Internet Test Letters & Postcard

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The objective of this research is to conduct combined cognitive and usability testing to evaluate the letters and postcard that are associated with the American Community Survey (ACS) Internet Test as well as a low-fidelity wireframe version of the online interface. We plan to measure user efficiency, accuracy, and user satisfaction with these materials.

Between March and April of 2010, staff from the Center for Survey Methods Research (CSMR) will conduct a maximum of 50 combined cognitive and usability interviews, 10 with each of the five panels in the test. The panels are as follows: Production, Prominent Internet Choice, Not Prominent Internet Choice, Internet Push Regular Schedule, and Internet Push Modified Schedule.

Participants will be randomly assigned to one of the five panels. During the session, each participant will read the letters and postcard for their assigned condition and then will complete a representative portion of the online ACS interface for themselves only. Participants’ eye movements for both the mail materials and the online form will be tracked using Tobii eye trackers in order to obtain empirical evidence about which areas of the ACS letters, postcard, and Internet interface they look at as well as about important elements that they overlook. We expect that the eye-tracker will not be able to calibrate some participants’ eyes. We will continue without eye tracking for those participants whose eyes do not calibrate. We will also track participants’ mouse movements using a Java application in order to investigate the pattern of movements made by the participants as they interact with the interface. All participants will be asked to provide commentary on their experience during a retrospective debriefing.

Attachment A contains all of the letters and postcards used in the testing. Attachment B contains images of each screen of the online ACS instrument. The test protocol, including debriefing questions, is attached in Attachment C. Participants will also be asked to complete a background survey about their demographics and computer experience. (Attachment D). Additionally, they will be asked to complete paper-and-pencil questionnaires designed to measure their satisfaction with the online form (Attachment E).

All sessions will be conducted in the Usability Lab at the Census HQ building. Participants will be recruited from the general public using the SRD Usability Lab's recruiting database and will be reimbursed $40 for their participation.

These sessions will be video-recorded, with the participants' permission, to facilitate summary of the results. All participants will be informed that their response is voluntary and that the information they provide is confidential.

The estimated time for completion of each of the cognitive and usability sessions is an hour. Thus, the estimated burden for this research is 50 hours.

The contact person for questions regarding data collection and study design is:

Kathleen T. Ashenfelter

Center for Survey Methods Research

U.S. Census Bureau

Room 5K020K

Washington, D.C. 20233

(301) 763-4922

kathleen.t.ashenfelter@census.gov