Protocol

2024 AIES Participant Debriefing Protocol

Purpose:

Researchers in the Census Bureau’s Economic Statistical Methods Division (ESMD) will conduct participant debriefing interviews in support of the 2024 Annual Integrated Economic Survey (AIES). These interviews will cover three main topics: response process, content/question wording, and instrument functionality.

Not all questions in this protocol will pertain to all participants, and some modules are optional depending on the length of the interview, firm characteristics, and pre-screener responses.

Research Questions: The research will be guided by the following two overarching research questions:

1. How are respondents reporting to the AIES?

   1. What are respondents' overall impressions of the survey?

   2. What are the unique reporting needs?

   3. What are the barriers to reporting?

2. What is respondents’ feedback on instrument performance and response burden?

   1. What feedback do respondents have on updates made to the instrument since the last iteration of the survey?

   2. What are the respondent-reported reasons for item nonresponse on the AIES?

   3. What are respondents’ impressions of screen layout, font, and other features of the online instrument?

Informed Consent: Respondents will be asked to complete a consent form electronically before the time of the interview. Additionally, they will be asked to voluntarily complete a pre-screener prior to their scheduled meeting time to identify any issues they had with reporting. The pre screener will be used to target debriefing topics and optimize the interview time.

Method: Debriefing interviews will be conducted virtually via Microsoft Teams.

Expected length of interview: 1 hour (60 minutes) maximum per interview

General probes that may be used throughout the interview:

• How easy or difficult was it to report?

• Did you have to consult records/other sources? If so, describe.

• Can you tell me more about that?

• How confident are you in your response?

• Was there anything unclear or confusing?

Introduction (5 minutes)

If necessary: You should have received a link to a consent form and pre screener/questions about your recent experience via email. Did you have a chance to review and sign the consent form and/or report to that pre screener?

[IF YES] Did you have any questions about any of the information presented there?

[IF NO] Please open the link now and review the form, then sign it. Take your time and let me know if you have any questions.

Thank you so much for agreeing to talk with me today!

The purpose of this meeting is to make sure that the Annual Integrated Economic Survey is performing as expected and to get feedback about ways to improve the survey. I’m talking with you today because you are identified as the person who provided response to the AIES this year. When I refer to “the AIES survey” during our conversation I’m talking about the Annual Integrated Economic Survey that you responded to about your business recently.

I didn’t write the survey questions, so you do not have to hold back when telling me your feedback. Please be candid and frank in your responses. Our interview is being conducted under the authority of Title 13, which means that your responses are confidential, and neither your name nor the name or identifying information about your company will be included in any of our findings. Only members of the research team will have access to your data.

Do you have any questions before we get started?

I’d like to record our session today so that when I go to analyze the results of these interviews, I can use the recording to pick up on anything I may have missed in my notes. Do I have your permission to record our session today?

[Turn on recording]

Topic 1: Response Process (7 minutes)

Let’s get started with some general questions about your role at your company and your experience with Census Bureau surveys in general before we start talking about the Annual Integrated Economic Survey specifically that you recently submitted [provide date, if known].

1. Tell me a little bit about your business. What types of goods or services does this business provide?

2. What is your role or position title? How long have you been in this role?

3. Are you typically the person who responds to survey requests from the Census Bureau?

   • Do you work with anyone else to get the data you need? If so, who?

4. What was your overall impression of the Annual Integrated Economic Survey that you recently completed?

   • Any initial concerns about the legitimacy of the survey? Was there anything that you saw either online or in the request that prompted you to report?

   • Were there any concerns about the data you provided to the survey? [If yes] Tell me more about that

     1. Did these concerns impact how or what you reported to the survey?

5. [If reported to AIES SY23] Did you notice anything about the survey that was different from last year? Did anything stand out for you?

6. How easy or difficult was it for you to find all the data you needed to report to this survey? Why is that?

7. How easy or difficult was it for you to use the website to navigate and report to this survey? Why is that?

Topic 2: Content (15 minutes)

PROBLEMATIC ITEMS

[If respondent did not complete the pre screener (Appendix A):]

• Did some topics or sections take more or less time than others? Which ones? [Interviewer, mark all that apply] [If unknown, select Unknown, 2 items for discussion will be randomly chosen]

• [IF NEEDED] Was it difficult to answer questions for the company level?

• [IF NEEDED] Was it difficult to answer questions for the location/establishment level?

• [IF NEEDED] Was it difficult to answer questions for the industry level?

• Was there any confusion about which parts of the company to include when answering the survey?

[IF INDICATED PROBLEMATIC CONTENT- PRE-SCREENER or via Analyst case notes]

Based on the responses you provided on the pre-screener that was sent before our session, it appears that you experienced some difficulty reporting for items at the [establishment, company, industry] level on the survey for questions about [topic].

Let’s review some of these together. [Instruct respondent on which item number to select to view the appropriate survey question(s).

Let’s first talk about reporting about [topic].

• What was this question asking you to report, in your own words? Was there anything unclear or confusing?

• Can you tell me more about the difficulty you experienced when reporting for this?

• At what level were you reporting this information? Was that clear?

  • How easy or difficult was it to report at the [establishment, company, industry] level?

• How confident are you in the response you provided?

• [If applicable] Were the instructions helpful or not helpful?

• Did you have to consult with other sources or records to report? Describe.

NOTE: Qualtrics will randomly review items from 2 topics identified - items are available until quota is reached for each topic identified as problematic – prioritizing those identified from prior studies first.

ITEM MISSINGNESS:

[INTERVIEWER INSTRUCTION: Before interview, indicate topics from the response data that was missing and probe as appropriate] Note: Items reviewed during problematic items discussion, will be skipped.]

I took a look at your AIES submission and responses, thank you again for responding. I noticed that you did not provide data for [TOPIC] for your [establishment, company, industry]. Can you tell me more about that?

Topic 3: Instrument (15 minutes)

1. When it came to using the site to report you mentioned [Summarize response given in intro]. Tell us more about your experience reporting to this survey. Was it easy or difficult to navigate the site to report. Any issues? What worked well for you?

2. [Single units]

[If reported AIES SY23 and R did not reference difference in reporting in Intro] Did you notice anything different about the way you reported to this survey this year?

[If necessary] Was this helpful or not helpful?

3. [Multi units]

Think about when you entered your data into the AIES instrument. There were options where you could download an Excel file, or you could fill in the data using an online spreadsheet [show screen]. How did you decide how to respond?

• Did you use just one way to respond, or did you use both?

  1. Was there a reason you chose to report that way? How did it go?

[If reported AIES SY23] Did you notice a difference from your reporting last year? If yes, describe. What are your thoughts on this?

[If reported via Spreadsheet, show template]

Can you please describe your overall experience using the spreadsheet template to report?

What did you think about the shading and overall layout and display of the template?

Was the auto sum feature helpful or not helpful? Describe.

4. [Error check] Think about when you entered your data into the AIES instrument. Do you remember running the error check feature? How was that process?

• What do you recall about the process of running the error check?

• Did running the error check help you to submit your data?

• What feedback, if any, do you have on error checking on this survey?

5. [Question preview tool] Take a look at this page – do you recall if you used this page at all? If yes, describe how you used this page.

   • [If yes] Did you find it easy or difficult to select your industry/industries?

   • [If yes] How did you know what to select?

[If no] Please tell me what you think this page is all about.

Did you know about this page before our time together today?

What are your overall impressions of this page?

Wrap up/Debriefing (3 minutes)

Universe: All participants

1. Based on your experience this year, are there ways you might prepare for the AIES in the future?

2. That’s all the questions I have for you today! Do you have any other comments, questions, or suggestions for us that we didn’t get a chance to talk about?

Thank you so much for your time today.

Appendix A: Pre-screener

Note: If participant selects I had trouble reporting data for specific question, they will receive the following screen. To the extent possible this items listed will be based on the primary NAICS so that topics that are not applicable are not presented