Instrument Development and Design Research and Testing Highlights

Attachment J

Department of Commerce

United States Census Bureau

OMB Information Collection Request

2022 Economic Census

OMB Control Number 0607-0998

Instrument Development and Design Research and Testing Highlights

Component	Activity	Dates	Number of Cases	OMB Generic Clearance Date	Description, Selected Results and/or Recommendations
Single Unit Businesses	Task analysis	July 2014	8	June 2014	Response process is relatively straightforward for SUs; however, they rely heavily on paper questionnaires for gathering the data.
	Usability testing Round 1	October –November 2014	10	October 2014	Tested a low-fidelity prototype and provided recommendations for a response-driven design for implementation in 2014 ASM. Respondents preferred less information on the screen and skipping questions that were not relevant to them.
	Usability testing Round 2	October 2015	9	October 2014	Tested the 2015 ASM response-driven instrument prior to release in December 2015
	Respondent debriefings for 2015 ASM full-scale implementation of response-driven design	February – June 2016	23	January 2016	Respondents liked response-driven design, answering questions “one at a time.” Respondents wanted a worksheet preview, but did not see this feature prior to beginning.
	Analysis of Web paradata from 2015 ASM full-scale implementation of response-driven design	April 2016	NA	NA	Identified troublesome screens/questions; reviewed use of features/functions; evaluated indicators of burden. Prioritized changes for 2016 ASM.
Multi-Unit (MU) Businesses	Respondent debriefings for 2012 Economic Census	February 2014	11	January 2014	Debriefed respondents about their use of the Census Surveyor instrument in order to identify features that should be maintained and/or improved for the 2017 Economic Census electronic data collection instrument. Major findings included: Desire for online reporting because downloading Surveyor software is troublesome Use of spreadsheets is critical to ease burden Questionnaire-like reporting guide and instructions are needed to aid data-gathering if paper forms are not available Tools are needed to aid complex response process that requires gathering data from multiple data sources / areas in the company Many respondents were unaware of existing burden-reducing functionality in Surveyor, and these tools need to be made more evident, along with clear instructions on how to use them Enhance automated features/functions and streamline review of edit failures
Multi-Unit (MU) Businesses, continued	Usability testing Round 1	November 2014	10	November 2014	Tested a low-fidelity prototype and gathered requirements for a functioning prototype, and tested initial spreadsheet designs.
	Usability testing Round 2	May - June 2015	24	November 2014	Performance of SharePoint prototype inadequate and solution abandoned. Feedback on proposed MU instrument interface generally positive for main menu design, introductory screens, and step-by-step instructions for instrument features. Feedback on alternative spreadsheet designs generally positive for incorporating questions’ instructions and improved visual design.
	Focus group with NPC Register Analysis Branch analysts	April 2015		NA	Discussed analysts’ experience assisting respondents with Census Surveyor and obtained feedback on plans for revising the MU instrument.
Multi-Unit (MU) Businesses, continued	Debriefing interviews with 2014 COS/ASM respondents	April – June 2015	30	March 2015	Evaluated use of Census Surveyor features / functionality with respondents at small and mid-size multi-unit companies. Corroborated past research: Small MUs preferred within-instrument functions; Mid-size / larger MUs preferred Excel spreadsheet functions; Full capabilities of Census Surveyor overlooked by Rs; Downloaded / printed documents critical for gathering data from other company personnel
	Usability testing Round 3	September 2015	17	June 2015	Evaluated basic functionality of MU instrument prototype in Centurion - spreadsheet download/upload, form view, and database that communicates to spreadsheet and form view; navigation and usability of pages; effectiveness of content, especially instructions about reporting procedures; navigation and usability of spreadsheet template; gathered requirements for worksheet/reporting guide
Multi-Unit (MU) Businesses, continued	Analysis of Census Surveyor paradata from 2015 ASM	July – September 2016	NA	NA	Although the legacy Census Surveyor instrument was a powerful tool with useful features, Rs were not aware of / did not use quite a few of them.
North American Product Classification System (NAPCS)	Record-keeping study	November 2014	9	August 2014	Explored availability of product line data in respondents’ business records.
	Usability testing Round 1	October 2015	9	June 2015	Tested alternative presentations of product lists for SUs; recommended 2-screen design where Rs first select relevant products from a list, then second screen displays only the selected products and requests receipts for each one.
	Usability testing Round 2	October 2015	7	June 2015	Tested with Construction SUs and MUs: Prefer 2-screen design over spreadsheets; “Building” and “Non-building” categories helped Rs focus on their construction type
	Usability testing Round 3	April 2015	12	August 2014	Investigated the utility of a search feature for looking up products. Rs preferred “write-in” over search functionality, as tested.
	Usability testing Round 4	November 2015	9	August 2014	Tested orientation of MU NAPCS spreadsheets; decision to replicate Census Surveyor version with products in columns and locations in rows.
North American Product Classification System (NAPCS), continued	Field test of alternative NAPCS question designs and write-ins	January – March 2017	892	Submitted: December 2016; Approval expected: January 2017	This will be a field test using an experimental design to examine the effectiveness of two alternative designs and lay-out options for obtaining descriptions of products not explicitly provided in the product lists. Results are expected in late March, and will be used to recommend a design strategy for “write-ins” in the final Economic Census instrument.
MU/SU Centurion Instrument	Usability testing for 2016 ASM/COS Round 1	July 2016	18	June 2016	Tested a limited-functioning prototype. The redesigned instrument performed well overall, but additional instructions and clearer labeling of buttons would be helpful to respondents
	Usability testing for 2016 ASM/COS Round 2	October 2016	28	June 2016	Tested a more fully functioning prototype. There were some features and functionality that were not clear to participants, e.g. how to update their locations.
	Usability testing for 2017 Economic Census	TBD	TBD	TBD	This will be the first usability testing of complete fully functioning Centurion data collection instrument for 2017 EC.
Web Respondent Portal Design and Functionality	Usability testing	July 2016	18	June 2016	Joint with ASM/COS instrument usability testing. Early results did not show major difficulties with using the new Portal.
	Usability testing	October 2016	19	September 2016	Overall, the account creation process and linking a survey to that account was not problematic.
Worksheet / Question Preview	2015 COS/ASM Worksheet Debriefings	February – March 2016	25	January 2016	Evaluated use of form/PDFs in current systems, and obtained feedback on worksheet prototype: Form/PDFs play critical role in data collection Most Rs used them (or would have, had they known about them) Positive feedback on prototype worksheet
New Content: Industry-specific Special Inquiries	Cognitive testing	August – December 2016	159	July-November 2016	21 sets of questions were tested. Much of the new content was not problematic. The majority of the questions required only minor changes. Only one set of questions required revisions that were re-tested.
New Content: Factoryless Goods Production (FGP)	2012 Economic Census respondent debriefings	January – June 2014	97	April 2014	Evaluated effectiveness of “Special Inquiry” questions re: contract manufacturing services (CMS) to aid in identifying FGPs at the establishment level. Results found wide variation in Rs’ interpretation and responses to these questions demonstrated that CMS alone was inadequate for identifying FGP.
	2016 COS Enterprise Statistics Program cognitive testing – Rounds 1&2	June – October 2015	37	May 2015	Cognitive testing of enterprise level questions about FGP activities. Found that respondents had inconsistent interpretation or confusion about some key terminology / characteristics associated with FGP.
New Content: Factoryless Goods Production (FGP), continued	Exploratory early stage scoping exploratory interviews Round 1	September 2015	12	October 2014	Interviews with suspected FGPs and Manufacturing Service Providers (MSPs) demonstrated complexity: Misunderstanding and inconsistent interpretation of several key concepts and terms central to FGP concept. Typical Econ Census Rs in HR or Finance depts indicated someone else in headquarters, e.g. the Controller’s or Operations office, would be more knowledgeable about this activity. MSPs suggested alternative types of agreements. They were found to be less knowledgeable about several key attributes and record-keeping practices for FGPs, and so will not be included in subsequent testing.
	Exploratory early stage scoping exploratory interviews Round 2	December 2015	9	July 2015	Interviews with suspected FGPs gleaned additional information about FGP practices, concepts, and terminology. FGPs considered this to be an enterprise-level concept rather than estab-level. Also revealed additional difficulties for MUs associated with complexity of structure / ownership of the company, particularly those with foreign affiliates / ownership.
New Content: Factoryless Goods Production (FGP), continued	Cognitive testing Rounds 1-3	June – September 2016	40	April 2016	Multiple iterative rounds of cognitive testing of questions drafted following early stage scoping, in consultation with BLS collaborators. Draft questions were written with the expectation that “typical” respondents would answer, not necessarily the “most knowledgeable,” and this was found to work adequately. Findings for SUs were generally good, and questions required minor revisions between successive rounds. Findings for MUs were more problematic, generally related to complexities in structures and ownership of companies, particularly those with foreign affiliates or owners.
	Cognitive testing Round 4	November- December 2016	Up to 35	April 2016	Questions were modified based on Round 3 results, and the final round of cognitive testing with revised questions will focus on MUs. Will also include non-FGPs, in order to ensure that they can answer appropriately, that is, to guard against propensity for “false positive” responses.