Download:
pdf |
pdfAttachment G
Department of Commerce
United States Census Bureau
OMB Information Collection Request
2017‐2019 Report of Organization
OMB Control Number 0607‐0444
Instrument Development and Design Research and Testing Highlights
OMB
Number Generic
of Cases Clearance
Date
8
June 2014
Component
Activity
Dates
Single Unit
Businesses
Task analysis
July 2014
Usability testing
Round 1
October –
November
2014
10
October
2014
Usability testing
Round 2
October
2015
9
October
2014
Respondent
debriefings for
2015 ASM fullscale
implementation of
response-driven
design
February –
June 2016
23
January
2016
Analysis of Web
paradata from
2015 ASM fullscale
implementation of
response-driven
design
April 2016
NA
NA
Description, Selected
Results and/or
Recommendations
Response process is
relatively straightforward
for SUs; however, they
rely heavily on paper
questionnaires for
gathering the data.
Tested a low-fidelity
prototype and provided
recommendations for a
response-driven design
for implementation in
2014 ASM. Respondents
preferred less
information on the screen
and skipping questions
that were not relevant to
them.
Tested the 2015 ASM
response-driven
instrument prior to
release in December
2015
Respondents liked
response-driven
design, answering
questions “one at a
time.”
Respondents wanted a
worksheet preview,
but did not see this
feature prior to
beginning.
Identified
troublesome
screens/questions;
reviewed use of
features/functions;
evaluated indicators
of burden.
Prioritized changes
for 2016 ASM.
Component
Activity
Dates
OMB
Number Generic
of Cases Clearance
Date
Description, Selected
Results and/or
Recommendations
Multi-Unit (MU)
Businesses
Respondent
debriefings for
2012 Economic
Census
February
2014
11
January
2014
Multi-Unit (MU)
Businesses,
continued
Usability testing
Round 1
November
2014
10
November
2014
Debriefed respondents
about their use of the
Census Surveyor
instrument in order to
identify features that
should be maintained
and/or improved for the
2017 Economic Census
electronic data collection
instrument. Major
findings included:
Desire for online
reporting because
downloading Surveyor
software is
troublesome
Use of spreadsheets is
critical to ease burden
Questionnaire-like
reporting guide and
instructions are needed
to aid data-gathering if
paper forms are not
available
Tools are needed to aid
complex response
process that requires
gathering data from
multiple data sources /
areas in the company
Many respondents
were unaware of
existing burdenreducing functionality
in Surveyor, and these
tools need to be made
more evident, along
with clear instructions
on how to use them
Enhance automated
features/functions and
streamline review of
edit failures
Tested a low-fidelity
prototype and gathered
requirements for a
Component
Activity
Dates
OMB
Number Generic
of Cases Clearance
Date
Usability testing
Round 2
May - June 24
2015
November
2014
Focus group with
NPC Register
Analysis Branch
analysts
April 2015
NA
Description, Selected
Results and/or
Recommendations
functioning prototype,
and tested initial
spreadsheet designs.
Performance of
SharePoint prototype
inadequate and
solution abandoned.
Feedback on proposed
MU instrument
interface generally
positive for main menu
design, introductory
screens, and step-bystep instructions for
instrument features.
Feedback on
alternative spreadsheet
designs generally
positive for
incorporating
questions’ instructions
and improved visual
design.
Discussed analysts’
experience assisting
respondents with Census
Surveyor and obtained
feedback on plans for
revising the MU
instrument.
Component
Activity
Multi-Unit (MU)
Businesses,
continued
Debriefing
April –
interviews with
June 2015
2014 Report of
Organization/ASM
respondents
Usability testing
Round 3
Dates
September
2015
OMB
Number Generic
of Cases Clearance
Date
30
March 2015
17
June 2015
Description, Selected
Results and/or
Recommendations
Evaluated use of Census
Surveyor features /
functionality with
respondents at small and
mid-size multi-unit
companies.
Corroborated past
research:
Small MUs preferred
within-instrument
functions;
Mid-size / larger MUs
preferred Excel
spreadsheet functions;
Full capabilities of
Census Surveyor
overlooked by Rs;
Downloaded / printed
documents critical for
gathering data from
other company
personnel
Evaluated basic
functionality of MU
instrument prototype in
Centurion - spreadsheet
download/upload, form
view, and database that
communicates to
spreadsheet and form
view; navigation and
usability of pages;
effectiveness of content,
especially instructions
about reporting
procedures; navigation
and usability of
spreadsheet template;
gathered requirements for
worksheet/reporting
guide
OMB
Number Generic
of Cases Clearance
Date
NA
NA
Component
Activity
Dates
Multi-Unit (MU)
Businesses,
continued
Analysis of
Census Surveyor
paradata from
2015 ASM
July –
September
2016
North American
Product
Classification
System (NAPCS)
Record-keeping
study
November
2014
9
August
2014
Usability testing
Round 1
October
2015
9
June 2015
Usability testing
Round 2
October
2015
7
June 2015
Usability testing
Round 3
April 2015
12
August
2014
Description, Selected
Results and/or
Recommendations
Although the legacy
Census Surveyor
instrument was a
powerful tool with useful
features, Rs were not
aware of / did not use
quite a few of them.
Explored availability of
product line data in
respondents’ business
records.
Tested alternative
presentations of product
lists for SUs;
recommended 2-screen
design where Rs first
select relevant products
from a list, then second
screen displays only the
selected products and
requests receipts for each
one.
Tested with Construction
SUs and MUs:
Prefer 2-screen design
over spreadsheets;
“Building” and “Nonbuilding” categories
helped Rs focus on
their construction type
Investigated the utility of
a search feature for
looking up products. Rs
preferred “write-in” over
search functionality, as
tested.
Component
Activity
Dates
Usability testing
Round 4
November
2015
North American
Product
Classification
System
(NAPCS),
continued
Field test of
alternative
NAPCS question
designs and writeins
January –
March
2017
MU/SU
Centurion
Instrument
OMB
Number Generic
of Cases Clearance
Date
9
August
2014
Description, Selected
Results and/or
Recommendations
Tested orientation of MU
NAPCS spreadsheets;
decision to replicate
Census Surveyor version
with products in columns
and locations in rows.
This will be a field test
using an experimental
design to examine the
effectiveness of two
alternative designs and
lay-out options for
obtaining descriptions of
products not explicitly
provided in the product
lists. Results are
expected in late March,
and will be used to
recommend a design
strategy for “write-ins” in
the final Economic
Census instrument.
892
Submitted:
December
2016;
Approval
expected:
January
2017
Usability testing
July 2016
for 2016 Report of
Organization/ASM
Round 1
18
June 2016
Tested a limitedfunctioning prototype.
The redesigned
instrument performed
well overall, but
additional instructions
and clearer labeling of
buttons would be helpful
to respondents
Usability testing
October
for 2016 Report of 2016
Organization/ASM
Round 2
28
June 2016
Tested a more fully
functioning prototype.
There were some features
and functionality that
were not clear to
participants, e.g. how to
update their locations.
Component
Web Respondent
Portal Design
and
Functionality
Activity
Dates
Usability testing
for 2017
Economic Census
TBD
OMB
Number Generic
of Cases Clearance
Date
TBD
TBD
Usability testing
July 2016
18
June 2016
Usability testing
October
2016
19
September
2016
25
January
2016
2015 Report of
February –
Worksheet /
Question Preview Organization/ASM March
Worksheet
2016
Debriefings
Description, Selected
Results and/or
Recommendations
This will be the first
usability testing of
complete fully
functioning Centurion
data collection
instrument for 2017 EC.
Joint with Report of
Organization/ASM
instrument usability
testing. Early results did
not show major
difficulties with using the
new Portal.
Overall, the account
creation process and
linking a survey to that
account was not
problematic.
Evaluated use of
form/PDFs in current
systems, and obtained
feedback on worksheet
prototype:
Form/PDFs play
critical role in data
collection
Most Rs used them (or
would have, had they
known about them)
Positive feedback on
prototype worksheet
OMB
Number Generic
of Cases Clearance
Date
159
JulyNovember
2016
Component
Activity
Dates
New Content:
Industry-specific
Special Inquiries
Cognitive testing
August –
December
2016
New Content:
Factoryless
Goods
Production
(FGP)
2012 Economic
Census respondent
debriefings
January –
June 2014
97
April 2014
2016 Report of
Organization
Enterprise
Statistics Program
cognitive testing –
Rounds 1&2
June –
October
2015
37
May 2015
Description, Selected
Results and/or
Recommendations
21 sets of questions were
tested. Much of the new
content was not
problematic. The
majority of the questions
required only minor
changes. Only one set of
questions required
revisions that were retested.
Evaluated effectiveness
of “Special Inquiry”
questions re: contract
manufacturing services
(CMS) to aid in
identifying FGPs at the
establishment level.
Results found wide
variation in Rs’
interpretation and
responses to these
questions demonstrated
that CMS alone was
inadequate for
identifying FGP.
Cognitive testing of
enterprise level questions
about FGP activities.
Found that respondents
had inconsistent
interpretation or
confusion about some
key terminology /
characteristics associated
with FGP.
Component
Activity
Dates
New Content:
Factoryless
Goods
Production
(FGP),
continued
Exploratory early
stage scoping
exploratory
interviews
Round 1
September
2015
OMB
Number Generic
of Cases Clearance
Date
12
October
2014
Description, Selected
Results and/or
Recommendations
Interviews with
suspected FGPs and
Manufacturing Service
Providers (MSPs)
demonstrated
complexity:
Misunderstanding and
inconsistent
interpretation of
several key concepts
and terms central to
FGP concept.
Typical Econ Census
Rs in HR or Finance
depts indicated
someone else in
headquarters, e.g. the
Controller’s or
Operations office,
would be more
knowledgeable about
this activity.
MSPs suggested
alternative types of
agreements. They
were found to be less
knowledgeable about
several key attributes
and record-keeping
practices for FGPs, and
so will not be included
in subsequent testing.
Component
Activity
Dates
Exploratory early
stage scoping
exploratory
interviews
Round 2
December
2015
OMB
Number Generic
of Cases Clearance
Date
9
July 2015
Description, Selected
Results and/or
Recommendations
Interviews with
suspected FGPs gleaned
additional information
about FGP practices,
concepts, and
terminology. FGPs
considered this to be an
enterprise-level concept
rather than estab-level.
Also revealed additional
difficulties for MUs
associated with
complexity of structure /
ownership of the
company, particularly
those with foreign
affiliates / ownership.
OMB
Number Generic
of Cases Clearance
Date
40
April 2016
Component
Activity
Dates
New Content:
Factoryless
Goods
Production
(FGP),
continued
Cognitive testing
Rounds 1-3
June –
September
2016
Cognitive testing
Round 4
November- Up to
December 35
2016
April 2016
Description, Selected
Results and/or
Recommendations
Multiple iterative rounds
of cognitive testing of
questions drafted
following early stage
scoping, in consultation
with BLS collaborators.
Draft questions were
written with the
expectation that “typical”
respondents would
answer, not necessarily
the “most
knowledgeable,” and this
was found to work
adequately. Findings for
SUs were generally good,
and questions required
minor revisions between
successive rounds.
Findings for MUs were
more problematic,
generally related to
complexities in structures
and ownership of
companies, particularly
those with foreign
affiliates or owners.
Questions were modified
based on Round 3 results,
and the final round of
cognitive testing with
revised questions will
focus on MUs. Will also
include non-FGPs, in
order to ensure that they
can answer appropriately,
that is, to guard against
propensity for “false
positive” responses.
File Type | application/pdf |
File Title | Microsoft Word - Attachment G |
Author | hill0032 |
File Modified | 2017-08-28 |
File Created | 2017-08-16 |