Generic Clearance Collection Request:
Cognitive Testing for the 2018 Centurion Site Tutorial Videos
Request: The Census Bureau plans to conduct research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The Census Bureau plans to conduct a research project evaluating tutorial videos that will be added to the online Centurion instrument for the Economic Census.
The Economic Census is conducted every 5 years and collects information on U.S. businesses and the economy from more than 1000 industries. For the 2017 Economic Census, the Census Bureau updated its existing Centurion online data collection system. This Economic Census survey is now conducted via a self-administered web-only method, requiring respondents to interact with the Centurion site in order to submit the survey contained within the Centurion platform. Title 13 of the United States Code authorizes this survey and provides for mandatory responses.
Thousands of individuals will interact with the Centurion site that houses the Economic Census, as well as other business related surveys. Usability testing, debriefing phone calls, and discussions with subject matter experts revealed several confusing features on the updated Centurion site that may be effectively addressed with short how-to videos.
We are exploring the use of supplemental tutorial videos that will be located on the Centurion site. The proposed videos are designed to assist respondents in logging in, navigating and successfully utilizing the Centurion site for completing the Economic Census. The videos may help orient respondents to the overall site and key features. We are proposing preliminary testing to assess the effectiveness of these new tutorial videos and to identify the optimal location within the website for where to locate these videos.
Purpose: The goal of this study is to identify the best content for the videos that would address the most frequent user issues, as well as the best location for the videos. Effective and readily accessible tutorial videos may reduce call center burden, and decrease the likelihood of respondent breakoff. The short video topics proposed include a walkthrough of the login process, a general overview of the site’s main dashboard, as well as specific directions to complete different tasks that were problematic for respondents. The login process takes place on a separate platform called “eCorr” that the respondent needs to navigate through before entering the Centurion site. The problem areas of the Centurion site were determined through usability testing, respondent debriefings and through analysis of the records of the calls received by Centurion customer service. The proposed pretesting research will give valuable feedback on refining the content of the videos and determining the best location within the website.
Population of Interest:
The Centurion website was updated for the 2016 Annual Survey of Manufactures and the 2016 Report of Organization. Because these surveys share similar content and design characteristics with the Economic Census, and because most of these respondents will be receiving the Economic Census, recent respondents to these surveys will be contacted and asked to participate in this research.
Timeline: Testing will be conducted from November through December 2017.
Language: Testing will be conducted in English only.
Method: Researchers will conduct cognitive interviews with up to 30 respondents in order to evaluate and refine the new tutorial videos for the Centurion site. During the interviews, respondents will complete several tasks in line with the actions they would take while completing the actual survey. They will view 9 how-to videos each about a minute or two in length. We will get the participant’s feedback on the content of the videos, and their opinions regarding the ideal location for them on the website. Local travel will be required; observers may attend many of these meetings.
Sample: We plan to conduct up to 30 interviews. We will contact potential participants via phone, explain the nature of our research, and ask them to participate in our study. Efforts will be made to sample respondents from a variety of sizes and types (i.e., industries) of businesses, though the sample will ultimately be dependent on who chooses to participate in the research. Participants will be informed that their response is voluntary and that the information they provide is confidential. Once interviews are scheduled, researchers will send respondents a confirmation via email, and may conduct reminder calls a few days before the meetings.
Recruitment: Participants in these cognitive interviews will come from a pool of respondents from the Annual Survey of Manufactures and the Report of Organization. We intend to meet with participants from companies covering a variety of industries.
Protocol: Participants will first complete initial paperwork including a consent form, standard demographic questionnaire and a modified internet-experience questionnaire. The interviews will be recorded (with consent) to facilitate post-hoc analysis. The internet-experience questionnaire will include several questions intended to ascertain the likelihood of participants watching how-to videos online.
Participants will interact with a PowerPoint designed to function like the Centurion instrument, as well as a mockup of the eCorr login platform. Participants will use a think-aloud procedure as they complete tasks in line with the actions they would need to take on the Centurion site in order to successfully fill out and submit their survey. After completing several tasks intended to get them familiar with the Centurion tool, they will watch 9 tutorial videos and be asked for their feedback. To ascertain the participant’s ideal location for the videos, participants will view a simple mock-up of different options, and select the one they most prefer. Participants will be asked debriefing questions before the end of the testing session.
Use of Incentive: Monetary incentives for participation will not be offered.
Length of interview: We expect that each interview will last no more than 60 minutes (30 cases x 60 minutes per case = 30 hours). Additionally, to recruit respondents we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 30 cases x 3 minute per call = 7.5 hours). Thus, the estimated burden for this project is 37.5 hours (30 hours for interviews + 7.5 hours for recruiting).
Enclosures: Below is a list of enclosed materials to be used in the current study:
Attachment A: Protocol
Attachment B: Demographic questionnaire to obtain participant characteristics
Attachment C: Consent form to obtain participant consent for participation and recording of the session
Attachment D: Modified Computer Use and Internet experience questionnaire to get an understanding of participants' experience using computers, the Internet, and how-to videos
Attachment E: Satisfaction questionnaire to allow for a subjective satisfaction measure to be gathered based on the participants’ experience with each online video
Attachment F: User tasks to allow for an assessment of the website’s features/design and assess its ease of use
Attachment G: Video script outline including screenshots for a general overview of the how-to video topics and visual guidance related to the how-to videos
Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:
Rebecca Keegan
Survey Statistician
Data Collection Methodology and Research Branch (DCMRB)
Economic Statistical Methods Division (ESMD)
U.S. Census Bureau
Office: 301-763-6003
Rebecca.Keegan@census.gov
Enclosures:
Cc:
Nick Orsini (DIR) with enclosures
Carol Caldwell
(ESMD) “
”
Diane Willimack
(ESMD) “
”
Amy Anderson Riemer
(ESMD) “
”
Kristin Stettler (ESMD) “ ”
Carma Hogue (ESMD) “ ”
William Davies (ESMD) “ ”
Lisa Donaldson (EMD) “ ”
Michelle Karlsson (EMD) “ ”
James Jamski (EMD) “ ”
Danielle Norman (PCO) “ ”
Mary Lenaiyasa (PCO) “ “
Jennifer Hunter Childs (ADRM) “ “
Jasmine Luck (ADRM) ” “
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Rebecca Keegan (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-21 |