OMB Memo

Census Bureau Participant Exploratory Research for the
2023 Annual Integrated Economic Survey

Submitted Under Generic Clearance for Questionnaire Pretesting Research

April 19, 2024

Request: The U.S. Census Bureau, through a contract with RTI International (RTI), plans to conduct research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725, expiration: 12/31/2025). We will be conducting exploratory interviews to document the record keeping practices and common verbiage for e-Commerce economic activity for future iterations of the Annual Integrated Economic Survey (AIES), a cross-economy annual survey that produces national and subnational annual economic estimates.

This exploratory research is one part of the larger respondent-centered research in support of the implementation of the AIES, including participant debriefing interviews and usability testing. This interviewing represents an additional research methodology to explore mis- and-under-reported content on the AIES specific to e-Commerce.

Background and Previous Research:

AIES: The Census Bureau’s Economic Directorate asked the National Academy of Sciences (NAS) to convene an expert panel to review their appropriated annual economic surveys and recommend improved methodologies for conducting and processing them. The panel started work in July 2015 and the final report, with recommendations, was released in May 2018.

To address these recommendations, the Economic Directorate has been conducting research into harmonizing and simplifying the design and production process for these economic surveys and the Economic Census. The results of several years of this research have culminated into a single harmonized survey instrument designed to be administered as one survey cycle throughout the economy, regardless of company size, industry, or other characteristics.

The full suite of research conducted in support of the AIES development is documented in the recent OMB submission, Census Bureau Participant Debriefing Research for the 2023 Annual Integrated Economic Survey (under the same OMB clearance number, 0607-0725).

E-Commerce: E-Commerce is a challenging phenomenon to measure accurately, an issue that has been well documented, especially in the last 20 years. Former Chief Economist for the Bureau of Economic Analysis, Barbara Fraumeni, outlined in 2001 the slippery nature of measuring e-Commerce, identifying several key issues to measuring e-Commerce, especially the ethereal nature of the products and services, the intangible benefits of e-Commerce, like time savings, speed and ease of transactions, and conveniences (319-321). Fayyaz (2018:3) outlines three aspects of e-Commerce important to measurement: the ordering and delivery process (the ‘how’); the nature of the products (the ‘what’); and the actors involved, including digital intermediaries and households (the ‘who’). From this framework, they develop a three pronged definition for e-Commerce, mainly (2018:4):

1. Digitally ordered transactions: sale or purchase of a good or service conducted over computer networks by methods specifically designed for the purpose of receiving or placing orders;

2. Digitally facilitated transactions: cross-border trade flows facilitated by online platforms (such as social media companies and credit card and bank systems); and

3. Digitally delivered transactions: services and data flows that are delivered digitally as downloadable products.

Barefoot et al. further mused on the importance of e-Commerce definitionally, finding that while the federal government has been trying to measure the ‘digital economy’ since the mid-1990s, one of the “most fundamental challenges” to this measurement is “the lack of a precise and universal definition that clarifies which activities should be included when measuring the digital economy” (2018:6). Further, the United Nations goes so far as to ask if e-Commerce is itself a separate economic phenomenon, able to be measured, saying “the growing prevalence of digitalization makes it increasingly difficult to distinguish the digital economy from economic activity in general, and therefore, difficult to quantify” (2021: 12).

In fact, the Census Bureau’s own investigations into e-Commerce collection have produced the same three consistent findings: respondents either did not understand what we were asking about, did not have access to e-Commerce data, or e-Commerce was not salient to their business. This includes testing on behalf of the legacy annual surveys that were integrated and harmonized into the AIES, including specific testing for the Services Annual Survey (SAS), the Annual Retail Trade Survey (ARTS), and the Annual Wholesale Trade Survey (AWTS); our review dated back to 2012.

In preparation for the 2023 AIES (launched in March 2024), the harmonized e-Commerce question incorporated wording and definitions from the legacy surveys. The current wording and definition of the e-Commerce question from the AIES survey is listed in Figure 1, with the survey of origin in parentheses bolded after the word or phrase of interest. It borrows from SAS, ARTS, AWTS, and the Annual Survey of Manufactures (ASM). See Attachment D for the question wording for e-Commerce questions from the legacy surveys.

Figure 1: Harmonized e-Commerce Question for the 2023 AIES, including legacy survey origin

Of the total $$00 Sales, Shipments, Receipts, or Revenue reported (SAS), what amount was from goods (ASM), services, or manufactured products that were ordered or whose movement was controlled or coordinated (ASM) electronically?

Please provide an estimate if exact figures are not available (SAS).

Electronic Commerce (e-commerce) or e-shipments:

The sale of goods, services, or manufactured products where the buyer/customer places an order, or the price and terms of the sale are negotiated, over the Internet, extranet, email, mobile devices (m-commerce), an Electronic Data Interchange (EDI) network, private networks, dedicated lines, or other comparable online systems/applications (ARTS/AWTS, harmonized).

Payment may or may not be made online (e.g., order online but pay at a location in person) (ARTS/AWTS).

Include:

• Revenues from customers entering orders directly on your own and/or third party websites or mobile applications (SAS)

• Shipments to other domestic plants of your own company for further manufacture, assembly, or fabrication (ASM)

• Buy or reserve online, pick up in store (BOPIS) (ARTS/AWTS)

• Buy or reserve online with home/business delivery (ARTS/AWTS)

• Buy online with curbside pickup (ARTS/AWTS)

• Buy online with kiosk/locker/tower pickup (ARTS/AWTS)

• Monetary donations received online as revenue from electronic sources (for tax-exempt firms) (ARTS/AWTS)

Purpose: While the 2023 AIES e-Commerce question has been harmonized across the in-scope legacy surveys, it has not undergone pre-testing with respondents for reporting accuracy or data accessibility. As noted, prior research has indicated that respondents either have not understood the e-Commerce questions, did not have access to e-Commerce data, or e-Commerce was not salient to their business. As such, we intend to focus this research less on definitional aspects of e-Commerce and more on the record keeping practices and data accessibility of e-Commerce as defined by the respondent. We will use the findings from these exploratory interviews to consider next steps in the capture of e-Commerce activity on AIES, including drafting revised question(s) for testing next year.

As such, we propose the following three research questions with sub-questions to guide this exploratory interviewing:

1. What are the current e-Commerce record-keeping practices of businesses?

• What is included or excluded in these records?

• What are the differences in record keeping by firm industry(ies)?

• What are the differences in record keeping by firm size and complexity?

• At what level of granularity are these records kept?

2. How accessible are records of e-Commerce activity?

• How easy or challenging is accessing this information?

• How many people at the business are involved in maintenance and pulling of e-Commerce records for reporting?

• Are there levels of granularity that are more or less accessible for respondents (e.g., establishment-level compared to industry- or company-wide level)?

3. How do current e-Commerce record keeping practices inform current reporting practices on the AIES?

• What are respondents currently including (and excluding) when reporting e-Commerce for AIES?

• How much data manipulation (aggregation, allocation, estimation, and others) do respondents engage in to report e-Commerce for AIES?

• What are the factors that influence respondents’ decisions to report e-Commerce on the AIES?

Population of Interest: The population of interest consists of a wide variety of businesses that are known to engage in at least some e-Commerce as part of their core business and meet the eligibility requirements for inclusion in the AIES sample¹. We anticipate targeted recruitment for the debriefings to include companies with at least one establishment classified in a six digit North American Industry Classification System (NAICS) code in the manufacturing sector (“manufacturing firms”). We will also pay special attention to complex firms, operationalized as companies with establishments classified in four or more NAICS codes.

Sample: For these exploratory interviews, we will be contacting respondents to the 2023 AIES who reported at least one e-Commerce variable this year. Census Bureau staff are already periodically providing RTI with a list of responding businesses to conduct debriefing interviews approved in an earlier request; to this list, we will append response data for the e-Commerce variables and from this list, RTI staff will recruit interview participants. That way, we will be able to minimize recruitment overlap between these two related efforts (debriefing interviews and exploratory e-Commerce interviews).

We plan to conduct a maximum of 45 interviews. We plan to conduct interviews with participants representing businesses of varying sizes and complexities, as well as response status. This number of interviews was selected because it is a manageable number of interviews for the timeframe available for this testing.

Recruitment: For these exploratory interviews, RTI will contact potential participants via email, explaining the nature of the research and asking them to participate in the study. They will recruit additional participants by telephone, as needed, to obtain wide participation by characteristics of interest, including manufacturing status, complexity, response status, and others. The realized sample of participants will be those who are able to be contacted and who agree to participate in the study.

Participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by Census Bureau employees or those with Special Sworn Status who are involved in the research project. Once interviews are scheduled, researchers will send participants a confirmation via email.

See Attachment B for respondent recruitment materials.

Method: Researchers from the Census Bureau and from RTI will conduct all interviews using Microsoft Teams. Interviews will follow a semi-structured protocol that includes both required and optional sets of interview questions. See Attachment A for the interview protocol. Interviews will last no longer than 1 hour.

Interviews may be recorded with respondent consent. These interviews will be stored on a Census Bureau secure server, accessible only to researchers on the project. These recordings will capture audio, computer screen, and may include participant faces and setting.

Protocol: Some parts of the interview will be administered to all participants, regardless of response status or firm characteristics. Some parts of the interview will be in-scope for participants representing certain firms. Finally, some of the parts of the interview are optional for all participants, and will be administered based on timing, participant feedback, and interviewer purview. Subject area specialists from the Census Bureau may participate in some of the debriefing interviews for observational purposes.

Attachment A contains the protocol for the exploratory interviewing. Attachment E contains the current AIES harmonized E-Commerce question that will be shared with participants during the interview.

Consent: In addition to the required PRA and Privacy Act notices, the consent form will also indicate that the respondent agrees that the interview can be audio- and video-recorded to facilitate analysis of the results (see Attachment C). Respondents will sign the consent form via a Qualtrics form that allows for a digital signature. Respondents who do not consent to be recorded will still be allowed to participate.

Incentive: Monetary incentives for participation will not be offered.

Interview Length and Burden Estimate: Interviews will last no more than 60 minutes each, and we will conduct no more than 45 interviews across various firm types and locations (total burden: 45 hours). We estimate it will take respondents five minutes to read and review recruitment materials, and we anticipate sending no more than two recruitment emails (total burden: about 7.5 hours). If recruitment is sluggish for particular types of firms, we may conduct up to five phone attempts at 3 minutes per call to gain participation in interviewing (total burden: about 11.5 hours). We anticipate it will take respondents no more than five minutes to sign the consent form, schedule the session, and log on when the session begins (total burden: about 4 hours). This brings the total estimated response burden to 68 hours.

Table 3: Burden Hours Assumptions for Exploratory Interviews

Activity	Target N	Recruitment N	Burden in hours
Interviewing (60 minutes max)	45	500	45
Recruitment materials review (10 minutes max)	45	500	7.5
Recruitment phone calls (15 minutes max)	45	500	11.5
Scheduling, signing consent forms, logging in (5 mins max)	45	500	4
Total			68

Timeline: Testing will begin in May 2024. All interviewing will be completed no later than the close-out date of the collection, October 2024. Findings and recommendations from interviewing will be delivered to the survey team by October 2024.

Language: Testing will be conducted in English only.

Attachments: Please refer to the following attachments related to this request:

Attachment A: 2023 AIES e-Commerce Exploratory Interview Protocol

Attachment B: Recruitment Materials

Attachment C: Participant Informed Consent Form

Attachment D: 2023 AIES Harmonized E-Commerce Question and Legacy Annual E-Commerce Questions

Attachment E: 2023 AIES Harmonized E-Commerce Question for Interview Participants

Works Cited:

Barefoot, K., Curtis, D., Jolliff, W., Nicholson, J. R., & Omohundro, R. (2018). Defining and Measuring the Digital Economy [Working Paper]. https://www.bea.gov/sites/default/files/papers/defining-and-measuring-the-digital-economy.pdf

Fayyaz, S. (2018). A Review on Measuring Digital Trade and E-Commerce as New Economic Statistics Products. 1–13. https://www.oecd.org/iaos2018/programme/IAOS-OECD2018_Fayyaz.pdf

Fraumeni, B. M. (2001). E-Commerce:  Measurement and Measurement Issues. The American Economic Review, 91(2), 318–322.

United Nations Conference on Trade and Development. (2021). COVID-19 and E-Commerce:  A Global Review. United Nations. https://etradeforall.org/wp-content/uploads/2021/03/eT4a_Covid-19_and_E-Commerce_Global_Review.pdf

Contact: The contact person for questions regarding the design of this research is listed below:

Melissa A. Cidade, Ph.D.

Principal Methodologist, Annual Integrated Economic Survey

Economy-Wide Statistics Division, Office of the Division Chief

U.S. Census Bureau

Washington, D.C. 20233

Melissa.Cidade@Census.gov

Cc:

Nick Orsini (ADEP) with enclosure

Stephanie Studds (ADEP) “ “

Lisa Donaldson (EWD) “ “

Edward Watkins (EWD) “ “

Tom Smith (EMD) “ “

Temika Holland (ESMD) “ “

Rebecca Keegan (ESMD) “ “

Aleia Clark Fobia (ADRM) “ “

Jasmine Luck (ADRM) “ “

Danielle Norman (PCO) “ “

Mary Lenaiyasa (PCO) “ “

1 For more information on sample considerations and sample unit eligibility for AIES, see: https://www.census.gov/programs-surveys/aies/technical-documentation/methodology.html