Dress Rehearsal De Dress Rehearsal Debriefing Protocol

Attachment C

Department of Commerce

United States Census Bureau

OMB Information Collection Request

Annual Integrated Economic Survey

OMB Control Number 0607-XXXX

AIES Dress Rehearsal Respondent Debriefing Interview Protocol

Summer/Fall 2023

This protocol is a guide – the questions here will not necessarily be asked exactly as worded in the protocol or in this order. Not all questions will be asked in every interview. As much information about interview participants and enterprises/establishments should be obtained prior to the interview as possible. In some cases, probing may need to be adjusted based on the background research that has been conducted and/or in response to participants’ insights.

These interviews will be guided by four research questions, each with sub-questions:

Research Question 1: How are respondents completing the survey?

• What is the process of gathering the data necessary to complete the survey?

• Does the holistic approach lead to better data quality?

Research Question 2: What are the patterns that shape the ways respondents answer the survey?

• What aspects of a company, such as size, staff, and records, dictate the data we receive?

• How do respondents make decisions about how to answer items or topics?

Research Question 3: Is the new survey overly burdensome?

• How, if at all, has the existing survey response procedures been the same and different using the new survey, as compared to the separate collection of the existing annual surveys?

• How many and what kind of staff are involved in gathering the information for this survey?

• What aspects of the survey instrument may be impeding the response process?

Materials: Protocol, consent form, survey responses, respondent instrument (overview of responses)

Method: We will conduct the interviews by phone, Microsoft Teams, or we will conduct the interviews in person, depending on availability and federal employee travel restrictions.

Expected length of interview: 1 hour (60 minutes) maximum

General probes that may be used throughout the interview:

• How did you arrive at this number/answer this question?

• Were these data easy to access?

• What else can you tell me about this?

• Can you tell me more about that?

• How confident are you in that response?

• What looked unclear or is confusing here?

1. Introduction

If necessary: I sent you a consent form in an email today – did you get a chance to sign that? If not, please do so now, and then we’ll get started.

Thank you so much for agreeing to talk with me today!

As part of the dress rehearsal for the new Annual Integrated Economic Survey, we are following up with some companies to learn more about the processes you may or may not have used to complete the survey.

I am part of a group within the Census Bureau that makes sure that our surveys are performing as expected and provides feedback to other parts of the Bureau about ways to improve the performance of our instruments. I’m talking with you today because of your unique role in testing out the new AIES survey instrument.

Remember, my job is to improve the surveys. Please be candid and frank in your responses. Our interview is being conducted under the authority of Title 13, which means that your responses are confidential, and neither your name nor the name or identifying information about your company will be included in any of our findings.

Do you have any questions before we get started?

I’d like to record our session today so that when I go to analyze the results of these interviews, I can use the recording to pick up on anything I may have missed in my notes. Do I have your permission to record our session today?

<<Turn on recorder>>

2. Company Background

1. What is your role in the company? How long have you been with the company?

2. What is your role in the process for responding to Census Bureau surveys?

   • Examples: gathering data, entering data, consulting with data providers, etc.

3. Tell me a little bit about your business. What types of goods or services does this business provide?

   • How many locations does your business currently operate? In how many states? In how many countries?

   • Is this a foreign or domestic company?

   • Are there related companies?

   • Did you have any confusion about which parts of the company to include when answering the survey?

4. In what ways was your approach the same or different for this survey versus other annual survey(s)?

3. Interaction with Instrument

General AIES approach:

5. How did you feel about all the information being accessible in one survey?

   • Was it easier or harder to answer all in one place?

6. How long do you think it took you to complete the whole survey? How long for the 3-step survey? How long for the spreadsheet?

7. Did some topics take more or less time than others? Which ones?

   • Which topics were the easiest to answer about? Which were the hardest?

   • Did certain items take more or less time to answer about?

8. Overall, which question(s) would you say was the most time-consuming or difficult to complete?

9. Had you heard about the change from the annual surveys to the AIES?

   • Where had you heard about it?

   • Did anyone help you prepare?

   • Did you do anything to change your approach here?

10. Are there ways you might prepare for the AIES in the future?

Three step survey:

11. Let’s talk about the 3-step survey. What did you think of that process?

    • What information were you supposed to provide at those screens?

12. Could you provide all the information for the verification questions or did you need to reach out to anyone else?

    • If you reached out to someone else, who?

    • What would you do if there were locations you couldn’t verify information for?

13. Did you feel breaking out into three steps was a useful way of organizing this information?

14. How easy or difficult was it to answer these questions?

15. How long did these steps take you?

    • Did any steps take more or less time than the others?

Spreadsheet – General reaction:

16. Let’s talk about the spreadsheet now. What were your first impressions?

17. What was your general approach to the spreadsheet like?

    • Did you have a plan for how to fill it out?

    • What questions did you answer first? What did you leave for last?

    • What was your process like? Did it change for the topics?

18. Do you think there were characteristics about your company that impacted how you answered?

    • Do you think company size or amount of establishments made it easier or harder to answer?

Spreadsheet - Usability

19. What did you think of being able to answer by industry or establishment?

    • For SUs: How did you choose which way to fill out the spreadsheet?

    • Was the ability to answer by either industry or establishment made clear to you?

    • Did you use this option?

    • If you did, did you see how the establishments summed on the industry level? Was this helpful to you?

20. Did you feel that the color code was clear? (not for SUs)

21. Did you notice some cells were not necessary or optional? (not for SUs)

    • I see you answered by establishment/industry for the spreadsheet/certain items or topics. How did you decide that?

22. Did you see the company totals at the bottom? Did these factor into your approach?

23. When you entered the spreadsheet, you would have seen a walkthrough tutorial outlining how to use it. Do you recall going through that tutorial? Was it useful to you?

    • Would you have known how to get back to this tutorial, if needed?

    • Did you use any other help document provided? (help pages; census.gov etc)

24. What did you think of the error notification?

25. Overall, were there any features you felt were missing?

    • Is there anything that would have made reporting easier if it had been included or formatted differently?

Nonresponse:

26. Some of your cells/items/topics/locations were left blank. Can you tell me more about why?

27. [skip if they completed any of the survey] I noticed that you did not complete the AIES dress rehearsal survey. Can you tell me about that?

    • What, if anything, could we at the Census Bureau do to support you in responding to this survey?

    • What are the reasons why you did not fully complete (or did not complete at all) the AIES dress rehearsal survey?

    • Do you have any suggestions for ways the Census Bureau could encourage companies to respond to this survey?

28. Were there any questions you thought weren’t applicable to your company?

    • If yes, how did you answer them?

Reaching out:

29. Which topics or items did you have readily available in your records?

30. Did you have to contact anyone else to find this information?

    • Who did you contact?

    • Which topics required you to reach out to others?

31. [If applicable] I remember you reached out to us about [a specific topic]. What ended up happening?

    • How did you decide when it was time to contact us?

4. Wrap-up

• That’s all the questions I have for you today! Do you have any other comments, questions, or suggestions for us?

Thank you so much for your time today.