AIES Dress Rehearsal Preliminary Findings and Recommendations

Attachment N
Department of Commerce
United States Census Bureau
OMB Information Collection Request
Annual Integrated Economic Survey
OMB Control Number 0607-1024
AIES Dress Rehearsal Preliminary Findings and Recommendations

Annual Integrated Economic
Survey Dress Rehearsal:
Preliminary Findings and
Recommendations
Melissa A. Cidade, EWD
Heidi St.Onge, ADEP
January 26, 2024

The Census Bureau has reviewed this data product to ensure appropriate access, use, and disclosure avoidance protection of the
confidential source data (Project No. P-7529180, Disclosure Review Board (DRB) approval number: CBDRB-FY24-ESMD010-004).

AIES Research Overview
Phase I: “The 78” Pilot 2022
• Goal: Understand response processes and further instrument refining
• Qualtrics instrument, 78 companies
Phase II: Response Spreadsheet Pilot 2023
• Goal: Induce independent response
• Response spreadsheet, about 900 companies
2022 AIES (Dress Rehearsal)
• Goal: Troubleshooting and infrastructure building
• New Centurion instrument, about 8,000 companies

CBDRB-FY24-ESMD010-004

2

Survey Response Data and
Paradata (N = 4,860)

Research
Modalities

Response Analysis Survey (RAS)
(N = 465)
Respondent Debriefing Interviews
(N = 44)
Respondent Usability Interviews
(N = 28)
Inbound Call Log (N = 924)
CBDRB-FY24-ESMD010-004

3

Positive feedback, too…
• “One of the things that I like about it because our company has grown
significantly by acquisition was being able to answer all the questions at one
time for all of the entities. I really appreciated that. Versus having multiple
surveys that I have had to do historically." – Debriefing Interview
• “When I got to the download spreadsheet, it was easy… We actually have 3 to 4
people working on [the survey], compiling it together, and submitting it as one.
It’s a little bit easier when we have the spreadsheet to work with as opposed to
manually entering something into a screen.” – Debriefing Interview
• “Very easy in our system. The questions on the survey were straightforward. It’s
easy to match the questions from the report that I run so easy to populate.” –
Debriefing Interview
CBDRB-FY24-ESMD010-004

Positive feedback 4

Usability
Testing

Dress
Rehearsal

Goal

Recommendation

Get feedback on the new
flow

Test new survey flow in AIES Dress Rehearsal collection.

x

x

Test key elements of the
spreadsheet design

Include functionality to clean up establishment lists

x

x

Consider functionality to orient respondents within the
spreadsheet

x

Continue to explore ways of communicating optionality at the unit
level.

x

Gain additional
information about
response burden

Explore other ways of collecting non-numeric responses.

x

Develop respondent
communications

Continue to develop response support materials.

Future
Research

x
x
x

Prime respondents for the change

x

Consider cutting content

x

x

Consider response support training.

x

Update letters to retain resonant messaging and drop
discouraging messaging.

x

Consider additional research into communications materials.

x

CBDRB-FY24-ESMD010-004

x

5

Get feedback on the new flow
Three-step structure
Linear design

CBDRB-FY24-ESMD010-004

6

CBDRB-FY24-ESMD010-004

Goal 1: Get feedback on the new flow 7

Get feedback on the new structure
• Finding 1: The three-step design needs additional support.

• “I didn’t know how many steps there were. I crossed my fingers, is this the end of it?
You didn’t know what they were going to ask on the next step. There was no way to
get a full view of what are all the questions going to be, what are all the steps? It
was all unknown until you tried to do it.” – Debriefing Interview

• Finding 2: Linear design suppressed response.

• “[I] would like the ability to be able to review questionnaire responses before
submitting. There should be an option at the end to preview survey responses so
someone other than the preparer could review for accuracy before submission.
The inability to go back and edit after answering a question is risky.” – Debriefing
Interview
CBDRB-FY24-ESMD010-004

Goal 1: Get feedback on the new flow 8

Recommendations: Get feedback on the new
structure
Dress Rehearsal Goal

Finding

Recommendation

Three-step design

The three-step design
Include flexible
needs additional support. navigation and a more
robust means of survey
previewing.

Further develop survey
preview and content
selection tool.

Linear design

Linear design suppressed
response.

Explore the ability to
move more freely
throughout the survey.

Develop ability to move
forward and backwards
through the survey.

CBDRB-FY24-ESMD010-004

Next Steps

Goal 1: Get feedback on the new flow 9

Test key elements of the
spreadsheet design
Include functionality to clean up establishment lists
Consider functionality to orient respondents within the spreadsheet
Continue to explore ways of communicating optionality at the unit level
CBDRB-FY24-ESMD010-004

10

Test key elements of the spreadsheet design
• Finding 3: Larger companies struggled
with Step 1.

Percentage of triggered warning/error
messages by type of message
(N = 68,693)*

• Finding 4: Respondents get lost in the
spreadsheet.

• “You had to scroll horizonal[ly] instead of
vertical[ly], and as you scrolled right, I had
to keep going back to see what line I am
on, am I entering in the right
information? A freeze pane view would be
good, or vertical presentation so you can
scroll that way.” – Debriefing Interview
CBDRB-FY24-ESMD010-004

Other
warnings
54.7%

Operational
status
warning
45.3%

*Total

number of warning/error messages triggered. Respondents
could trigger the same error more than once.
Goal 2: Test the spreadsheet 11

Test key elements of the spreadsheet design
• Finding 5: Respondents struggled with units.

• “First two rows are a glitch. I would just skip them and ignore them if it’s just the
two rows, that doesn’t give me information of which location you’re referring to.” –
Usability Interview

• Finding 6: Rounding functionality surprised respondents.
• Finding 7: NAPCS reporting needs additional attention.
• Missed
• Cumbersome
• Mismatched

CBDRB-FY24-ESMD010-004

Goal 2: Test the spreadsheet 12

Recommendations: Test key elements of the
spreadsheet design
Dress Rehearsal Goal

Finding

Recommendation

Next Steps

Include functionality to clean
up establishment lists

Large companies struggled
with Step 1.

Develop download/upload
functionality for Step 1.

Provide download/upload
functionality for Step 1.

Consider functionality to
orient respondents within
the spreadsheet

Respondents get lost in the
spreadsheet

Explore ways to freeze panes. Freeze left columns.

Continue to explore ways of
communicating optionality at
the unit level

Respondents struggled with
units.

Update ways of displaying
KAUs to cue respondents

Update KAU display.

Test other key elements of
the spreadsheet design

Rounding functionality
surprised respondents.

Include instruction that
entries will be rounded.

Include instruction on
instructions tab.

NAPCS reporting needs
additional attention.

Investigate better performing
ways of collecting NAPCS.

NAPCS investigations

CBDRB-FY24-ESMD010-004

Goal 2: Test the spreadsheet 13

Gain additional information
about response burden
Explore other ways of collecting non-numeric responses.
Prime respondents for the change.

CBDRB-FY24-ESMD010-004

14

Gain additional information about response
burden
• Finding 8: Error checking needs additional development.

• “[My response generated] an error. The error report was really weird. It said
it passed and then it had errors. I don’t understand that.” – Debriefing
Interview

• Finding 9: Respondents rely on a survey preview.

• “Not being able to get the entire survey prior to entering the data in caused
significant challenge.” – RAS
CBDRB-FY24-ESMD010-004

Goal 3: Response Burden 15

Gain additional
information
about
response
burden

Which of the following, if any, were challenges to completing the AIES
questionnaire?
(N = 465*)

400
350

334

300
243

250

227
189

200

172

150

Finding 10: Content
continues to be a
challenge.

155

108

100

89
32

50
0

153

Had to collect
information
from more
than one
database or
other source

Had to wait
to rely on
others within
my company
for the
requested
data

Had to add,
allocate, or
otherwise
manipulate
data to fit
questions

Too many
questions

Unclear or Some other
inadequately challenge
defined terms

I had no
The online Already had Questions
challenges
survey was other surveys were too
difficult to to complete complicated completing
the AIES
at the same
use
questionnaire
time

*Note: Respondents could select all that apply.

CBDRB-FY24-ESMD010-004

Goal 3: Response Burden 16

Recommendations: Gain additional
information about response burden
Dress Rehearsal Goal

Finding

Recommendation

Next Steps

Explore other ways of
collecting non-numeric
responses.

Error checking needs
additional development.

Consider automatically
checking submissions.

Update error labeling and
implement integrated
error checking.

Update labels on error
checking report.

Prime respondents for the Respondents rely on a
change
survey preview.
Content continues to be a
challenge.

Make content previews
more accessible and
easier-to-use.

Further develop survey
preview and content
selection tool.

Cognitively test
misreported content and
consider additional cuts.

Cognitive testing and
content cuts.

CBDRB-FY24-ESMD010-004

Goal 3: Response Burden 17

Cognitively test misreported content and
consider additional cuts.

Cognitive Testing

Cuts in Content:

48 total response fields, including:
Instrument Section

Number
of Fields Universe

Step 1: Pre-listed establishments

5

All

Step 1: Additional establishments

5

All who added estabs

Step 2: Company-level

14

All

Step 3: More granular data

7

Manufacturing companies

10

All

6

Retail, Wholesale, and Service
companies

1

One specific NAICS classification
CBDRB-FY24-ESMD010-004

Robotics

E-Commerce
Goal 3: Response Burden 18

Approximately how long did it take to complete
this survey?

Estimate of Burden
Source

Type

N

Median Burden

80%

443

4.5

397

4.5

Multi-Units

46

4

4,784

4

Single Units 2,161

2

Multi-Units

2,623

6

Overall

4,860

1.1

Same-Day 3,159
Completion

0.5

Multi-Day
1,701
Completion

1.6

Paradata*

Overall

100%
90%

Response
Overall
Analysis Survey Single Units
Instrument

Percentage responding by source and type of unit

*Disclosure Review Board (DRB) approval number: CDRB-FY24-ESMD001-003

6.5%

10.8%

13.0%

13.9%

70%

23.9%

27.5%

2.9%
9.4%

8.0%
23.5%

17.5%
70.3%

60%

25.7%

56.6%

50%

47.9%

40%

42.8%

30%
20%
10%
0%

RAS
Single Units
(N = 397)

RAS
Multi-Units
(N = 46)

Instrument
Single Units
(N = 2,161)

Instrument
Multi-Units
(N = 2,623)

4 hours or fewer

5 to 10 hours

11 to 30 hours

31 or more hours

CBDRB-FY24-ESMD010-004

19

Develop respondent
communications
Continue to develop response support materials.
Update letters to retain resonant messaging and drop discouraging messaging.

CBDRB-FY24-ESMD010-004

20

Develop respondent communications
Which of the following, if any, did you use to complete the AIES?
(N = 465*)
300
250

246

200
134

150

106

100

65

50
0

156

• Finding 11:
Respondents report
using the materials
when they are
relevant and
accessible.

58

49
21

AIES
website

AIES
AIES How-to Calling the Interactive AIES Videos Some other I did not use
content
PDFs
AIES
content tool
material or any of these
summary
Helpdesk
help
to complete
the AIES
*Note: Respondents could select all that apply.

CBDRB-FY24-ESMD010-004

• Finding 12: Each
communication piece
serves a specific
function.
Goal 4: Communications 21

Recommendations: Develop respondent
communications
Dress Rehearsal Goal

Finding

Recommendation

Next Steps

Continue to develop
response support
materials.

Respondents report using
the materials when they
are relevant and
accessible.

Review the ecosystem of
communications
materials to ensure that
all pieces are accessible.

Review website for
accessibility and ease of
use.

Update letters to retain
resonant messaging and
drop discouraging
messaging.

Each communication
piece serves a specific
function.

Additional
communications-based
research.

Conduct additional
communications-focused
research.

CBDRB-FY24-ESMD010-004

Goal 4: Communications 22

Conduct additional communications-focused
research.
Environmental Scan

• Best practices for contacting
businesses:
• Letters
•
•
•
•

Messaging
Mail type and envelop features
Sequencing and timing
Length of field period

Secondary Analyses

• Pilot and Dress Rehearsal response
• Legacy response
• 2022 Economic Census

• Support materials

• Videos
• Survey previews
• Website
CBDRB-FY24-ESMD010-004

Goal 4: Communications 23

Goal

Finding

Recommendation

Get feedback on
the new flow.

Three-step design

The three-step design needs additional
supports.

Linear design

Linear design suppressed response.

Include functionality to clean up establishment lists

Large companies struggled with Step 1.

Consider functionality to orient respondents within the spreadsheet

Respondents get lost in the spreadsheet

Continue to explore ways of communicating optionality at the unit
level

Respondents struggled with units.

Test other key elements of the spreadsheet design

Rounding functionality surprised
respondents.

Test key elements
of the
spreadsheet
design.

NAPCS reporting needs additional attention.
Gain additional
information
about response
burden.
Develop
respondent
communications

Explore other ways of collecting non-numeric responses.

Error checking needs additional
development.

Prime respondents for the change

Respondents rely on a survey preview.

Content continues to be a challenge.

Cognitively test misreported content and
consider additional cuts.

Continue to develop response support materials.

Respondents report using the materials
when they are relevant and accessible.

Update letters to retain resonant messaging and drop discouraging
messaging.

Each communication piece serves a specific
function.

CBDRB-FY24-ESMD010-004

Findings and Recommendations

24

Goal

Recommendation

Next Steps

Get feedback on the new
flow.

The three-step design needs additional
supports.

Further develop survey preview and content selection tool.

Linear design suppressed response.

Develop ability to move forward and backwards through the
survey.

Large companies struggled with Step 1.

Provide download/upload functionality for Step 1.

Respondents get lost in the spreadsheet

Freeze left columns.

Respondents struggled with units.

Update KAU display.

Rounding functionality surprised
respondents.

Include instruction on instructions tab.

NAPCS reporting needs additional
attention.

Additional research into NAPCS capture.

Error checking needs additional
development.

Update error labeling and implement integrated error
checking.

Respondents rely on a survey preview.

Further develop survey preview and content selection tool.

Cognitively test misreported content and
consider additional cuts.

Conduct cognitive testing on misreported content.

Respondents report using the materials
when they are relevant and accessible.

Review website for accessibility and ease of use.

Test key elements of the
spreadsheet design.

Gain additional
information about
response burden.

Develop respondent
communications.

Each communication piece serves a specific Conduct additional communications-focused research.
CBDRB-FY24-ESMD010-004
25
function.

Goal

Recommendation

Next Steps

Get feedback on the new
flow.

The three-step design needs additional
supports.

Further develop survey preview and content selection tool.

Linear design suppressed response.

Develop ability to move forward and backwards through the
survey.

Large companies struggled with Step 1.

Provide download/upload functionality for Step 1.

Respondents get lost in the spreadsheet

Freeze left columns.

Respondents struggled with units.

Update KAU display.

Rounding functionality surprised
respondents.

Include instruction on instructions tab.

NAPCS reporting needs additional
attention.

Additional research into NAPCS capture.

Error checking needs additional
development.

Update error labeling and implement integrated error
checking.

Respondents rely on a survey preview.

Further develop survey preview and content selection tool.

Cognitively test misreported content and
consider additional cuts.

Conduct cognitive testing on misreported content.

Respondents report using the materials
when they are relevant and accessible.

Review website for accessibility and ease of use.

Test key elements of the
spreadsheet design.

Gain additional
information about
response burden.

Develop respondent
communications.

Each communication piece serves a specific Conduct additional communications-focused research.
CBDRB-FY24-ESMD010-004
26
function.

Key Takeaways:
Testing is worth it!
• Found the showstoppers
• First practice in collection

Additional areas of investigation
• Production Implementation Research:
• Respondent debriefing interviews
• Additional Usability testing

• Understand sources of burden • Outstanding Recommendations Research:
• Piloting is a proven method

• Cognitive interviewing
• Communications-based research

• Survey Advancement Research:

• Large firm response
• Non-respondents and historically undercounted
businesses

CBDRB-FY24-ESMD010-004

Key Takeaways

27

Goal

Recommendation

Next Steps

Get feedback on
the new flow.

The three-step design needs additional supports.

Further develop survey preview and content selection tool.

Linear design suppressed response.

Develop ability to move forward and backwards through the survey.

Test key elements Large companies struggled with Step 1.
of the spreadsheet
Respondents get lost in the spreadsheet
design.
Respondents struggled with units.

Provide download/upload functionality for Step 1.
Freeze left columns.
Update KAU display.

Rounding functionality surprised respondents.

Include instruction on instructions tab.

NAPCS reporting needs additional attention.

Additional research into NAPCS capture.

Gain additional
information about
response burden.

Error checking needs additional development.

Update error labeling and implement integrated error checking.

Respondents rely on a survey preview.

Further develop survey preview and content selection tool.

Cognitively test misreported content and consider
additional cuts.

Conduct cognitive testing on misreported content.

Develop
respondent
communications.

Respondents report using the materials when
they are relevant and accessible.

Review website for accessibility and ease of use.

Each communication piece serves a specific
function.

Conduct additional communications-focused research.

Bold – in progress, probable inclusion in 2024
CBDRB-FY24-ESMD010-004

Recommendations and Next Steps 28