OMB Memo

Generic Information Collection Request

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting cognitive interviewing for the National Science Foundation’s (NSF) Business Research and Development Survey (BRDS). BRDS is the primary source of information on research and development performed or funded by businesses within the United States. The survey is conducted for the National Center for Science and Engineering Statistics by the Census Bureau in accordance with an agreement between the two agencies. Results are used to assess trends in the performance and funding of business research and development (R&D). The annual survey examines a nationally representative sample of companies in manufacturing and non-manufacturing industries.

Further information regarding BRDS can be found at this website: https://www.census.gov/programs-surveys/brds.html

Purpose: There is a desire to publish statistics on the amount of research and development that businesses are investing in artificial intelligence (AI) paid by their company and paid by others. NSF is very interested in only learning about research and development on artificial intelligence as opposed to research and development using/applying artificial intelligence. Cognitive interviews will focus on this difference.

Additionally, NSF is interested in publishing statistics on the complete research and development workforce. The survey currently collects and publishes information about the employees of the company. Some companies have leased or temporary employees, including consultants, that work on research and development. NSF would like to assess whether companies can provide a count of these leased or temporary employees.

Respondents will be asked these three additional questions along with follow-up probes about how they track this information, what records are available, potential burden, a discussion about the question terminology, and best location for these questions within the form.

Staff from the Data Collection Methodology & Research Branch within the Economic Statistical Methods Division (ESMD) of the Census Bureau will be conducting cognitive interviews regarding this testing, with support from staff from the Economic Reimbursable Division (ERD).

Population of Interest: The new questions are proposed to be included in BRDS beginning next survey cycle. Based on the subject matter of the questions, we intend to interview companies with a range of total R&D (with an emphasis, but not an exclusive focus, on companies that have larger amounts of total R&D expenditures). Companies known to report R&D costs for software development and leased or temporary employees will also be targeted.

To assist in the recruiting efforts, staff from ERD will provide DCMRB staff with a list of businesses on the population of interest with respondent contact information.

Timeline: Testing will be conducted in August, 2018.

Language: Testing will be conducted in English only.

Method:

The method of research will be cognitive interviews, which are interviews aimed at understanding the way that a person interprets the items that are being presented to them, in order to understand and reduce potential measurement error before these questions are included in the survey. We will also be asking early stage scoping questions, which have a broader focus on the respondents’ understanding and familiarity with the concepts being addressed in the question, the way their company’s records are organized, and the terminology that they use in reference to these concepts. The interviews will follow a semi-structured interview protocol (Enclosure 1). The draft questions and a separate definition for AI are also enclosed (Enclosures 2, 3, and 4).

We plan to conduct one hour in-person or telephone cognitive interviews with respondents in the Washington, DC metropolitan area and one additional large metropolitan area based on the availability of a sufficient amount of the population of interest once recruiting begins. Interviews will be conducted by researchers from the Data Collection Methodology and Research Branch. Observers from ERD’s BRDS collections operation staff may be present at the interviews. Their primary purpose will be for observation, but they may also assist in cases where additional clarification of the subject matter is required, and to provide information on the purpose and uses of the survey.

We will request that the cognitive interviews be audio recorded with the participants’ permission (see Enclosure 5), to aid researchers in accurately summarizing key findings from the interviews.

Sample: We plan to conduct a total of 20 cognitive interviews. The sample size necessary for this test was determined by several factors: standards in cognitive interviewing practice about the number of interviews that can be used to draw conclusions; the need to meet with companies that cover varying industries, varying sizes, and varying R&D capabilities and expenditures; and the constraints of timing and resources. We plan to conduct interviews with respondents from the current BRDS sample.

Recruitment: We will contact potential participants via phone, explain the nature of our research, and ask them to participate in our study. The sample of participants will be those who are able to be contacted and who agree to participate in the study. Respondents will be informed that their participation is voluntary. After an interview is scheduled, respondents will receive a confirmation email as well as a reminder email or phone call prior to the interview.

Protocol: Participants will first be asked a series of exploratory questions about their research and development activities on artificial intelligence and their use of leased or temporary employees. Next, participants will be asked follow-up probes regarding how they track this information, what records are available, potential burden, terminology, and best location for these questions within the form. Then, participants will review the new proposed questions. Participants will then be asked follow-up probes related to these questions.

Use of Incentive: Monetary incentives for participation will not be offered.

Length of interview: We expect that each interview will last no more than 60 minutes (20 cases x 60 minutes per case = 20 hours). Additionally, to recruit respondents we expect to make up to 10 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (10 attempted phone calls per completed case x 20 cases x 3 minute per case = 10 hours). Thus, the estimated burden for this project is 30 hours (20 hours for interviews + 10 hours for recruiting).

Enclosures: Below is a list of materials to be used in the current study:

1. Draft protocol

2. Draft Artificial Intelligence questions (5-13 and 5-27)

3. Draft leased or temporary employees question (6-8)

4. Draft Artificial Intelligence definition

5. Consent form

Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Amy Anderson Riemer

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-7544

Amy.E.Anderson.Riemer@census.gov

Enclosures

Cc:

Ron Jarmin (DIR) with enclosure

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Carma Hogue (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Kristin Stettler (ESMD) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

Kevin Deardorff (ERD) with enclosure

Richard Hough (ERD) with enclosure

Michael Flaherty (ERD) with enclosure