American Factfinder Pretesting Plan

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We plan to conduct a remote usability study on the new American FactFinder (AFF) Web site to identify issues that are problematic and frustrating to the user. AFF is a free online tool that allows users to find, customize and download Census information.

Testing will begin in February 2013 and go through June of 2013 on the live working site. A maximum of 100 interviews will be conducted. We will be conducting these interviews remotely using Loop 11, a remote testing tool. All sessions will be conducted with remote asynchronous technology, conducted without a test administrator. This will broaden the scope of participants used to test the website.

The Loop 11 software that we will be using for this trial evaluation will store the data in the cloud, therefore we cannot collect any PII or Title 13 data. The data that will be stored, are the responses to the task and qualitative questions which are attached and described below. Loop 11 also stores whether the user abandoned the task, completed the task, and how long it took the user to either click complete or abandon buttons on the screen. Finally, the data also contain whether the user got to the web page which we identify as a "success" so we have whether the user "failed" the task or "successfully completed" the task. Loop 11 stores the web site path data, so we can track how users traveled through the site. Loop 11 also stores click data in order to create heat maps of the clicks.

For the past year members of CSM have been evaluating different remote testing software packages. Loop 11 seems to be the most promising since it is reasonably priced, it works on mobile devices, and we can purchase a version to install on our servers such that in the future, we could actually collect Title 13 data. We are evaluating Loop 11 for this project to see how the demo version of Loop 11 works (e.g., setting up the test, using the email feature, creating tasks, evaluating the results, etc.) and whether we want to purchase it. We are using a free trial version with all the bells and whistles, but we are not collecting PII or Title 13 data.

We have not used this tool before and are not aware of others at the Census Bureau who have. We worked with our CAT lab to evaluate about six remote testing products and had Webex-type presentations made by each vendor. A member from the Security office at the Census Bureau was invited to each presentation.

The participants will be recruited via an email, which will be sent to test participants who have worked with AFF before and come from the Census Information Centers (CICs), State Data Centers (SDCs) and Research Data Centers (RDCs). They have all previously given their email addresses to the AFF team and have said they were willing to be contacted by the Census Bureau via email. All participants will be informed in the email that their participation is voluntary, anonymous and any data collected will not be linked to them. The group of users that will be sent the email are primarily expert users. All participants will have a minimum of one year Internet experience and use the Internet at least three times a week to search for information.

The email that participants receive will contain a link to a url, which will open the participant to the AFF site but within Loop11’s remote testing capability so that the question will appear at the top of the Web site with a submit button for each answer.

The remote testing session will consist of three task questions, four follow-up questions to elicit feedback on the participant’s experience with the site during the previous task, and a final debriefing/satisfaction questionnaire. Copies of the tasks, follow-up questions, and debriefing/ satisfaction questions that will be asked via the remote technology are attached. Respondents will not be compensated for their participation in this study.

We estimate that users will spend 15 minutes on average taking the study. Thus, the total estimated respondent burden for this study is 25 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Erica Olmsted-Hawala

Center for Survey Methods Research

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-4893

Erica.L.Olmsted.Hawala@census.gov