OMB memo

National Telecommunications and Information Administration (NTIA) Internet Use Survey Cognitive Interviews

Request:

The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We propose to conduct an iterative cognitive evaluation of the National Telecommunications and Information Administration (NTIA) Internet Use Survey, which is part of the Current Population Survey (CPS), and is sponsored by NTIA in the Department of Commerce. As part of this current submission, we are seeking approval for cognitive interviews conducted by Census Bureau staff from the Center for Behavioral Science Methods (CBSM).

Purpose:

Most of the current NTIA Internet Use Survey items have already been cognitively evaluated because the most recent iterations of the survey underwent cognitive testing before being implemented. For this iteration, NTIA is interested in shortening the questionnaire, which will next be fielded in November 2023. The purpose of this cognitive testing research is to test a shortened questionnaire where select items have been removed and existing items have been revised to decrease the amount of time to administer the NTIA Internet Use Survey.

In data collection, the survey is currently administered via computer assisted personal interviewing (CAPI) and computer assisted telephone interviewing (CATI). However, the NTIA Internet Use Survey will be adapted to a self-administered web questionnaire later this decade. Therefore, we will evaluating both the interviewer-administered content for this next data collection cycle in addition to self-administered content for future cycles.

The full supplemental questionnaire is attached and includes core questions from the CPS used to roster the household members. Rostering household members is done to determine household member eligibility for the NTIA Internet Use Survey (see Enclosure 1: CPS Roster; Enclosure 2: NTIA Internet Use Survey).

We plan to cognitively test all items in the NTIA Internet Use Survey using specific probes designed as part of the cognitive test (see Enclosure 3: NTIA Internet Use Survey Protocol). We will also evaluate how easy it is for participants to answer questions on behalf of other people in their household.

The results of the cognitive test will inform the final revised NTIA Internet Use Survey CAPI/CATI instrument content that will be used to collect data in November 2023. We will also collect exploratory data on the feasibility of answering questions on a self-administered NTIA Internet Use Survey web instrument to inform future data collection cycles.

Populations of Interest:

The planned cognitive pretesting evaluation will focus on assessing and improving the questionnaire content for the general population.

Timeline:

Round 1 of testing focusing on the interviewer-administered content will be conducted from approximately March 2023 through May 2023, with recruitment beginning in March. Round 2 of testing focusing on the self-administered web content will be conducted from approximately June 2023 through September 2023, with recruitment beginning in June.

Language:

Testing will be conducted in English only.

Method:

We will conduct the interviews with a maximum of 40 participants across the two rounds: 20 participants for each of the interviewer-administered and self-administered content. For interviewer-administered testing, we will use either a Microsoft Excel, Word, or Qualtrics template to administer the questionnaire. For web testing, we will program a self-administered questionnaire in Qualtrics.

The interviews will be conducted remotely via Microsoft Teams. We will allow interviews to be conducted via telephone for participants who are not able to use Microsoft Teams. Each interview will be about 65 minutes long. Before the start of the interview, we will conduct a five minute “tech check” for those participants who will be using Microsoft Teams. We will ensure they can join the session and share the screen.

In both self-administered and interviewer-administered interviews conducted on Microsoft Teams, the interviewer may share their screen with participants to gather feedback on some of the question wording during retrospective probing. In the self-administered interviews, participants will share their screen while completing the survey so interviewers can follow along. There will be no screen sharing in interviewer-administered interviews when the questionnaire is being administered.

Sample:

Our recruiting efforts will target participants with the following characteristics:

• Ages 18 years old or older,

• Living in households with related household members or living in households with unrelated adult household members,

• Computer, tablet, and/or smartphone ownership, and

• Ability to complete a survey online

We will target participants who live in single member households, households with members who are closely related (e.g., parent/child, siblings, spouses) or more distantly related (e.g., grandparent/grandkid, cousins) and households with unrelated members. Single person household interviews will focus exclusively on comprehension. Other interviews will focus on comprehension as well as the feasibility of participants answering the questions for themselves and other household members. We will also target participants who own a computer, tablet, or smartphone and report being able to complete a survey online. See Table 1 below for the recruitment goals of each interview type by the composition of the household:

Table 1: Number of interviews by interview type and composition of the household (n = 40).

	Single-member household	Related household	Unrelated household
Interviewer-Administered	3	12	5
Self-administered	3	12	5

Recruitment:

Participants will be recruited using several methods: CBSM’s database pool of participants, word-of-mouth, advertisements on different web sites (e.g., Craigslist.com, NextDoor), and broadcast messages (see Enclosure 4: Communication Materials) distributed through the Census Bureau. All interviews will be conducted via Microsoft Teams. We will allow interviews to be conducted via telephone for participants who are not able to use Microsoft Teams. The advertisements on web sites and broadcast messages will have a national outreach, and as a result, we will recruit participants from the entire U.S.

People who respond to the advertisements will be screened with the universal screener and other questions attached (see Enclosure 4). They will be selected based on meeting the demographic characteristics described above and having diversity in gender, race/ethnicity, age, geography, and other demographic characteristics to the maximum extent possible.

Participants will be screened either by telephone or will complete a self-administered web form. They will receive a reminder about the interview appointment after they agree to participate.

Protocol:

We will inform participants that their response is voluntary, confidential, and will be accessed only by employees involved in the research project. The consent form (see Enclosure 4) will also indicate that the participant agrees that the interview can be video and audio-recorded to facilitate analysis of the results. Interviewers will ask participants to give consent orally to be recorded. Participants who do not consent to be audio-recorded will still be allowed to participate.

Participants will first be asked a list of roster questions to obtain information about their demographics and household size. They will then be asked the complete list of questions from the NTIA Internet Use Survey. We will conduct the cognitive interviews using retrospective methods which allow the participant to complete the questionnaire in its entirety and respond for all eligible household members before being asked probes about particular questions of interest. During probing, participants will be asked about the feasibility of answering the questions for both themselves and on behalf of other household members and their comprehension of specific terminology in the questionnaire.

The interview protocol includes probes regarding whether: (a) the wording of the revised questions allows participants to understand the meaning and purpose of the questions, (b) participants can answer the revised questions in the NTIA Internet Use Survey accurately and with little to no difficulty (c) participants can answer the revised questions without needing any additional information or clarification for time purposes and (d) participants can answer the revised questions on behalf of other household members.

Incentive:

We are requesting a $40 incentive for each remote participant. Payment will be mailed in cash to an address the participant provides on the payment voucher after the interview is complete.

Length of Data Collections:

We estimate that each interview will take approximately 65 minutes. The screening questionnaire will take approximately 15 minutes per person.

We expect to screen five people for each successful recruit. In total, we will screen about 200 people to arrive at 40 successful recruits. Thus, we estimate the burden for this research is 94 hours: 44 hours for interviewing (1.1 hours X 40 interview participants), and 50 hours (0.25 hours X 200 people) for screening.

Table 2: Estimated Burden

Research Activity	Number of Participants	Participation Time (minutes)	Burden Time (hours)
Screening for Interviews	200	15	50 hours
Interviews	40	65	44 hours
Total			94 hours

The materials to be used in this study are enclosed with this letter and include:

• CPS Roster (Enclosure 1)

• NTIA Internet Use Survey (Enclosure 2)

• Cognitive interview protocol (Enclosure 3)

• Communication Materials (Enclosure 4)

The contact person for questions regarding data collection and technical aspects of the design of this research is listed below:

Jonathan Katz

Center for Behavioral Science Methods

U.S. Census Bureau

Washington, D.C. 20233

301-763-5956

Jonathan.M.Katz@census.gov

3