B-2 Round 1 Cognitive Testing Findings (5/25/2023)

2023 NTIA Internet Use Survey Cognitive Testing Round 1

Conducted by CBSM at U.S. Census Bureau

Document prepared by Jonathan Katz, Alexandra Piccirillo, Caitlyn Keeve, and Jessica Holzberg

5/25/2023

Introduction

This document contains the results and recommendations from cognitive testing of Round 1 of the 2023 National Telecommunications and Information Administration (NTIA) Internet Use Survey; it has been reviewed for disclosure avoidance and approved under CBDRB-FY23-CBSM003-005. For questions or comments, please contact jonathan.m.katz@census.gov. In addition to the authors, we want to acknowledge Kevin Younes for his help recruiting on this project.

Methods

The main goal of testing was to evaluate a series of revised items that were streamlined for clarity but also length to reduce time administered in the interview, at the request of NTIA and ADDP (Associate Directorate for Demographic Programs).

We interviewed 20 participants for Round 1 of 2023 NTIA Internet Use Survey testing. Participants came from existing Center for Behavioral Science Methods (CBSM) databases where they had either completed a screener for a prior testing project or completed a cognitive or usability interview. Participants were re-contacted for this study and asked to complete either a Qualtrics self-administered screener or an interviewer-administered screener specific to this study prior to interviewing.

Interviews were conducted either remotely via Microsoft Teams or by telephone. Interviewers completed an interviewer-administered Qualtrics programmed instrument with which they could follow along. Participants were asked about internet use of the household and of a randomly selected household member. A section of Table 1 (below) shows the relationship between the participant and the individual they served as a proxy for, if any. Following the completion of the survey, participants were asked retrospective probes about their answers to the survey questions.

Participant characteristics

The participant characteristics are listed in the table below. We attempted to recruit a variety of participants included those from both related and unrelated households and also those that may have had limited Internet experience. Based on the interview data we collected and retrospective probing, we were able to recruit three participants who appeared to be a limited Internet user or didn’t use the Internet across many devices. One of these three participants was visually impaired and the other two participants were 65 and older. We did not have any participants who did not use the Internet as this was not the main focus of testing. Participants were from a variety of states across the country: California, Colorado, Connecticut, District of Columbia, Florida, Illinois, Maryland, Massachusetts, New Jersey, South Carolina, Texas, Virginia, and Washington.

Table 1. Participant Characteristics
Household	Single member	2
	Related	13
	Unrelated	5
Education	Less than high school	0
	Completed high school	2
	Some college, no degree	4
	Associate’s degree (AA/AS)	5
	Bachelor’s degree (BA/BS)	5
	Post-Bachelor’s degree (for example, MA, MS, Ph. D, JD)	4
	Other degree or studies	0
Sex	Female	13
	Male	7
Hispanic	Yes	3
	No	17
Race*	White	14
	Black or African American	5
	American Indian or Alaska Natïve	1
	Asian	2
	Native Hawaiian or other Pacific Islander	0
Age	18-34	4
	34-50	9
	51-64	3
	65 and over	4
Income	Less than $15,000	1
	$15,000 to $24,999	1
	$25,000 to $49,999	8
	$50,000 to $100,000	8
	More than $100,000	2
Interview Type	Microsoft Teams	17
	Telephone	3
Relationship to Participant for Proxy Questions1 (EMAIL to HOMIOT)	Self	2
	Husband/wife/spouse	4
	Unmarried partner	3
	Parent(s)	3
	Children	1
	Siblings	2
	Roommates	5

*A participant could select one or more races so total sum may not equal to the number of participants interviewed.

Tested Wording	Findings	Recommendations
CMPINT Production Wording This month we are asking some additional questions about the Internet, as well as computers, mobile phones, tablets, and other devices that connect to the Internet. First, we will ask what types of computers and other devices [you/members of your household] currently use. Please focus on devices [you/members of your household] currently have access to, and have used at least occasionally during the past six months. We are interested in devices used at any location, whether at home, work, school, a library, or anywhere else. Round 1 Wording This month we are asking some additional questions about Internet use. First, we will ask what types of computers and other devices [you/members of your household] currently use. Please focus on devices [you/members of your household] currently have access to, and have used at least occasionally during the past six months. We are interested in devices used at home, work, school, a library, or any other place.	Participants did not have any questions when read this intro. Although we did not probe for this, participants generally seemed to be thinking of the correct time frame for these questions. Based on think aloud feedback during the interview questions, one participant may have been thinking about which household members own specific devices rather than use them. This seemed to be the only case where this happened. Another participant was correctly thinking of “use” rather than “ownership” when answering. For example, when asked the tablet question, the participant mentioned their husband owned a tablet but hadn’t used it, so they reported ‘No’.	Since the revised wording is shorter and seemed to test well, we recommend this updated wording for production: “This month we are asking some additional questions about Internet use. First, we will ask what types of computers and other devices [you/members of your household] currently use. Please focus on devices [you/members of your household] currently have access to, and have used at least occasionally during the past six months. We are interested in devices used at home, work, school, a library, or any other place.”
Production Wording DESKTP [Do you/Does anyone in this household, including you,] use a desktop computer at least occasionally? Include any desktop computers [you/members of your household] currently have access to and have used during the past six months. (If needed) A desktop is a personal computer that is too large or heavy to be frequently moved from place to place. (If yes & is multi-person household) Who is that? LAPTOP What about a laptop? [Do you/Does anyone in this household] use a laptop computer? (If needed) A laptop is a personal computer that is portable and includes a built-in keyboard and screen. (If yes & is multi-person household) Who is that? Round 1 Wording (First 16 interviews) DESKTP (Main question wording the same). (If needed) A desktop is a personal computer that must be plugged into a power outlet and is typically at a single location. (If yes & is multi-person household) Who is that? LAPTOP Full question wording the same. Round 1 Wording (Last 4 interviews)* LAPTOP [Do you/Does anyone in this household, including you] use a laptop computer at least occasionally? Include any laptop computers [you/members of your household] currently have access to and have used during the past 6 months (If needed) A laptop is a personal computer that is portable and includes a built-in keyboard and screen. (If yes & is multi-person household) Who is that? DESKTP What about a desktop? [Do you/Does anyone in this household] use a desktop computer? (If needed) A desktop is a personal computer that must be plugged into a power outlet and is typically at a single location. (If yes & is multi-person household) Who is that? *Order was flipped for last 4 interviews	Prior to the first round of testing, we updated the “if needed” wording for desktop as we thought this would be a more accurate definition given changes in construction of desktops. In the first 16 interviews, three participants had trouble answering whether they had a desktop computer. One participant who reported “Yes” to desktop computer seemed to have a laptop as they mentioned it was a “desk laptop” during probing. A second participant asked, “is a desktop the same as a laptop?” and the interviewer read the ‘if needed’ text. This participant then said “Yes”. This participant later said during probing that their laptop was work-issued and is essentially used as a desktop. They then added it could be a considered a desktop if it’s not moved around. A third participant who answered “Yes” to desktop originally later said during probing that they realized they don’t have a desktop, just a laptop. One other participant who correctly answered this question mentioned during probing that her elderly parents use their laptop as a desktop computer as they do not move it. Because a few participants struggled answering the desktop question, we suggested re-ordering the desktop and laptop questions with laptop coming first for the last 4 interviews. We thought if participants heard the laptop computer first, they would be more likely to say “Yes” to that question and then they would properly answer “No” to the desktop question after hearing that second. For the last four interviews, no participants appeared to incorrectly categorize a desktop computer as a laptop computer. Two of the four participants reported not having a desktop computer and were able to adequately describe what a desktop was.	We recommend having the laptop question precede the desktop question for production to lessen the chances a participant categorizes a laptop computer as a desktop computer. LAPTOP [Do you/Does anyone in this household, including you] use a laptop computer at least occasionally? Include any laptop computers [you/members of your household] currently have access to and have used during the past 6 months (If needed) A laptop is a personal computer that is portable and includes a built-in keyboard and screen. (If yes & is multi-person household) Who is that? DESKTOP What about a desktop? [Do you/Does anyone in this household] use a desktop computer? (If needed) A desktop is a personal computer that must be plugged into a power outlet and is typically at a single location. (If yes & is multi-person household) Who is that?
TABLET Production and Round 1 Wording What about a tablet or e-book reader, such as an iPad or Kindle? (If needed) A tablet or e-book reader is a personal computer that is portable and has a built-in screen, but does not have a built-in physical keyboard. (If yes & is multi-person household) Who is that?	All participants were able to answer this question without issue. During the interview, one participant mentioned their spouse using a Microsoft Surface and initially answered ‘Yes’ to the laptop question. However, they were unsure if it was actually a laptop, and they changed their answer to ‘No’ after further thinking. They then answered ‘Yes’ to the tablet question. These were the correct answers to those two questions given their situation. During probing, participants were asked about the difference between a laptop and tablet. Participants generally were able to explain the difference between the two. Some of the responses included that laptop had a keyboard: “Laptop has a built-in keyboard where tablet does not” Some participants also thought about how a laptop uses a mouse and has more memory storage compared to a tablet. Another couple of participants thought of a laptop similar to a desktop but made to be portable: "Tablets are limited to certain information or capabilities whereas laptop to me sounds kind of similar to a desktop except that it's made to be portable". Only one participant did not seem to be able to understand the difference between a laptop and tablet as this participant said they were the same thing which was incorrect.	We recommend the current wording for production.
MPHONE Production and Round 1 Wording What about a smartphone, or a cell phone that connects to the Internet? (If needed) [Do you/Does anyone in this household] use a smartphone? (If yes & is multi-person household) Who is that?	All participants said ‘Yes’ to this question as at least one person in their household uses a smartphone. One participant incorrectly reported that one of the people in the household uses a smartphone when they did not. The reason this participant erroneously said another household member was a user is because they mistakenly thought we were asking simply about “cell phone” without the Internet so they included the other household member’s flip phone. This participant struggled with other questions in the interview, so we think what happened for this question is a unique issue.	We recommend the current wording for production.
WEARAB Production Wording [Do you/Does anyone in this household] use a wearable device that is connected to the Internet, such as a smart watch or fitness band? Examples include an Apple Watch or Fitbit. (If yes & is multi-person household) Who is that? Round 1 Wording [Do you/Does anyone in this household] use a wearable device that is connected to the Internet, such as a smart watch or fitness band? (If needed) Examples include an Apple Watch or Fitbit. (If yes & is multi-person household) Who is that?	Prior to testing, we modified the examples of a smart watch and fitness band to be read only if the respondent needed clarification. This was done to reduce interview time. The question was tested to ensure that removing the examples would not affect comprehension. Fifteen of the twenty participants reported that a person in their household uses a wearable device to connect to the Internet. None of the participants were read the ‘if needed’ text as it appeared that no participants had trouble answering. Participants were later probed if they found the question easy or difficult to answer and participants generally found this easy. When participants were asked of other examples of wearable devices, multiple participants thought about Apple Watches, Fitbits, pedometers, Samsung watches, blood and heart monitors, Virtual Reality headsets, and wearable devices in your sleep (a ring for sleep). One participant also thought about Garmin and another participant mentioned using an Amazon Halo. Four participants were asked if the examples in the ‘if needed’ text would have been helpful. Two participants said they would have been helpful in answering the question, but the other participants thought the question itself without the ‘if needed’ text was easy to understand. For participants who said the examples would have been helpful, they still seemed to answer the question accurately based on their responses to the interview question and probes.	Question for NTIA: Are Virtual Reality headsets out of scope for this question? If so, we don’t recommend additional wording changes at this time for the final Blaise wording but may recommend some additional testing on this issue in the future. Otherwise, the tested wording worked well, and we recommend this for production.
TVBOX Production Wording [Do you/Does anyone in this household] use a smart TV, a game or video system, or another device that connects to the Internet and plays through a TV? Examples include an Xbox, Apple TV, PlayStation, Roku, or a Blu-Ray player that can access the Internet. (If yes & is multi-person household) Who is that? Round 1 Wording [Do you/Does anyone in this household] use a smart TV, a game system, or another device that connects to the Internet and plays through a TV? Examples include an Apple TV, Xbox, or Roku that can access the Internet. (If yes & is multi-person household) Who is that?	Prior to testing, we made a slight change to remove “PlayStation” from the question wording. This was done to reduce interview length and we thought the word “PlayStation” was redundant since Xbox and other examples were already mentioned. Thirteen of the twenty participants reported that their household uses a Smart TV or a game or video system that connects to the Internet and plays through a TV. The majority of participants seemed to answer this question without much issue. The omission of the word PlayStation did not seem to affect participant’s understanding that they should include PlayStation as an example. Seven participants mentioned “PlayStation” as an example when responding to the probing questions. However, a few participants had trouble answering this question. Two participants answered this incorrectly as they reported ‘No’ but they should have answered ‘Yes’. During the administration of the survey, one participant mentioned having an Apple TV but did not consider it a smart TV, so they answered ‘No’: Participant: "We do have an Apple TV box but not a smart TV. Does that answer your question?" (Interviewer repeated survey question) Participant: “No” It was unclear why they did not consider an Apple TV as a smart TV. A second participant who also said ‘No’ but should have said ‘Yes’ reported that originally, she was thinking of “touch screen TVs”. During probing, she realized she in fact had a smart TV. This participant mentioned having a clearer definition or separating the questions by device type would be helpful. One participant who answered correctly also had trouble identifying what constituted a device that plays through a TV during probing. One participant said that if someone downloaded a movie and used an HDMI cable to connect the laptop to a smart TV, they would consider that in scope (this participant said ‘Yes’ to the survey question because they have a smart TV that is connected to an Xbox). Another participant mentioned that this question could be difficult because of “potential add-ons” to a regular television (Firestick, Wii-U remote). One participant also was not sure of the word “use” in this context as the participant had a brother with a disability that watches the smart TV but isn’t able to use or interact with the television capabilities.	Although a few participants had trouble answering this question, we still think the tested wording is most optimal. We should carefully test this question in self-administered testing to see if similar problems occur.
DEVQUA Production Wording Thinking about all the different devices we just discussed, how well do the computers and other Internet-connected devices [you use/used by this household] work overall? Please do not include home Wi-Fi routers or similar equipment. Read and select best match; if respondent is unsure, ask for an average They do not work at all They rarely work well They somewhat work They work well They work perfectly Round 1 Wording Thinking about all the different devices we just discussed, how well do the computers and other Internet-connected devices [you use/used by this household] work overall? Please do not include home Wi-Fi routers or similar equipment. Read and select best match; if respondent is unsure, ask for an average They do not work at all They work poorly They somewhat work They work well They work perfectly	In previous production cycles, DEVQUA had taken a long time to be administered in interviews. As a result, we modified “They rarely work well” to “They work poorly”. All of the participants said the devices at least “somewhat work”, with two participants saying, “they somewhat work”, thirteen participants saying, “they work well”, and five participants saying, “they work perfectly”. The majority of participants seemed to be able to answer this question without much issue. One participant did take a long time to answer the question and had to be read the categories multiple times. Initially, the participant was discussing how each of the household member’s devices worked, rather than on an overall basis. During probing, only two participants reported that the question was difficult. One of these participants said it was difficult because they would lose programming for a minute and would not be sure if it was due to the router or network. However, three participants did seem to consider external factors like weather and outages when probed about this question. There is a possibility they were conflating their thinking with DEVSTA when being probed about DEVQUA. One participant said the following: “Other than whole system going down in [the] community, it works well for us” (this participant answered “they work well”) Another participant said the following: “Can’t control weather, can’t control wire” (this participant answered “they somewhat work”) One of the participants that mentioned the question was difficult also said there had recently been connectivity issues that sometimes were a result of power outages. However, they also mentioned their old laptop had broken and they were without one for a period of time (this participant said “they work well”). Participants were also asked what does “they work poorly” mean and generally were able to interpret this phrase. Participants said this means that the devices don’t meet expectations, cannot accomplish a goal, or they do not work: “They work but they don’t give you exactly what you want, and they aren’t very reliable”. “Computer crashes a lot, maybe it doesn’t charge well, maybe doesn’t have the capacity to do the work you need to do on it”. A couple participants were also thinking of specific thresholds when asked what “they work poorly” mean: “It works less than 30% of the time”. A couple participants did think about the Internet connection when thinking about what “they work poorly” meant.	There is concern that participants are possibly thinking of events out of scope (Internet connection, power outages, and weather events). However, we recommend the tested wording for production because adding in events not to include could potentially make the question more confusing. If NTIA wants to, we can add examples not to include for self-administered testing to be used in future production cycles. Question for NTIA: What is the distribution of responses for this question from previous cycles? If most participants are already providing at least “they somewhat work”, the problems found in testing may not be as worrisome.
DEVSTA Production Wording During the last six months, how often were [you/members of your household] unable to use a computer or other Internet-connected device because it was in use by someone else, located elsewhere, or otherwise inaccessible? Read and select best match Never During one or two days During three or four days During five or six days During seven or more days Round 1 Wording At any point during the past six months, were [you/members of your household] unable to use a computer or other Internet-connected device because it was in use by someone else, located elsewhere, or otherwise inaccessible? Yes No	Similar to DEVQUA, this question had taken a long time to be administered in previous production cycles. As a result, we modified this question to be a Yes/No question. The distribution was split as ten participants answered, ‘Yes’ and ten participants answered ‘No’. Participants generally seemed to be able to answer this question without much issue. A couple of participants didn’t provide a codable response when read the question initially. One participant took a while to answer as they mentioned having outages and was wondering if we’re including outages. This participant eventually said “Yes” which was the correct response. Another participant said during the survey administration they have had computers not functioning the way they want to, so they had a friend look at it. The participant added that they had occasional difficulties and then it got fixed. After probing for a ‘Yes’ or ‘No’ response, the participant said ‘Yes’. Later in retrospective probing, they said the screen was frozen and the computer would not follow the commands as additional examples. During retrospective probing, all participants except one said that the question was easy to answer. The one participant that said it was difficult seemed to be confused by the “someone else, located elsewhere” phrasing. They said “Yes” to this question as they seemed to be thinking of instances where they let someone in their household use their devices, thus making the devices temporarily inaccessible. This participant also mentioned an instance where someone else in their household damaged their device and they were no longer able to use it. This interpretation of the question seemed to be in scope. Participants who reported “Yes” to this question were thinking of instances where they had to share devices or if there were any outages (power, weather). One participant also mentioned an example of leaving phones/laptops or chargers in other places that affected device use. Many participants who reported “No” mentioned examples of everyone having their own devices or having enough devices in the house. One participant wasn’t sure what “located elsewhere” meant but still answered this question correctly.	Although some participants didn’t provide an initial codable response which increased interview time, we think this revised question tested well and participants were able to answer their situation accurately. Therefore, we recommend the following: “At any point during the past six months, were [you/members of your household] unable to use a computer or other Internet-connected device because it was in use by someone else, located elsewhere, or otherwise inaccessible?”
INTINT Production Wording Next, we will ask where [you/members of your household] currently use the Internet. Please focus on places where [you/members of your household] have used the Internet at least occasionally during the past six months. Internet use includes a wide variety of activities, from checking email or browsing the Web to watching videos or using mobile apps. We are interested in Internet use on all devices, including the ones we just discussed. Round 1 Wording Next, we will ask where [you/members of your household] currently use the Internet. Please focus on places where [you/members of your household] have used the Internet at least occasionally during the past six months. Internet use includes a wide variety of activities, from checking email or browsing the Web to watching videos or using mobile apps.	All participants understood this question as written. When asked which devices they were thinking of when answering this series of questions, a majority of respondents said that they were thinking of their cellphones and laptops. Some mentioned only thinking of one device. Several said the device that came to mind was dependent upon the question being asked. One participant, who was visually impaired, was thinking about when they used “Alexa” when answering this series of question.	Given that some respondents only thought of a single device, we recommend instructing all respondents to think about all devices they use. Consider using the following wording: “Next, we will ask where [you/members of your household] currently use the Internet. Please focus on places where [you/members of your household] have used the Internet at least occasionally during the past six months. Internet use includes a wide variety of activities on ALL devices, from checking email or browsing the Web to watching videos or using mobile apps.” Question for NTIA: Would Alexa be considered in scope for using Internet? We can see this being relevant for visually impaired individuals who use the Internet via voice commands.
INHOME Production and Round 1 Wording [Do you/Does anyone in this household, including you,] use the Internet at home? This includes accessing the Internet with a cell phone, computer, tablet, or other device. (If yes & is multi-person household) Who is that?	All participants understood this question as written.	We recommend the current wording for production.
INWORK Production Wording What about at work? [Do you/Does anyone in this household] use the Internet at work? Do not include work from home. (If yes & is multi-person household) Who is that? Round 1 Wording [Do you/Does anyone in this household] use the Internet at work? Do not include work from home. (If yes & is multi-person household) Who is that?	We removed the sentence “What about at work?” prior to testing because we thought it was redundant and we wanted to reduce time in the interview. One participant incorrectly reported they use the Internet at work even though they are not currently working for pay or profit (note, the question was asked since another household member works). The participant interpreted the question incorrectly and mentioned using the Internet at the library. Otherwise, there were no problems with understanding this question.	We recommend using the tested wording of this question for production: “[Do you/Does anyone in this household] use the Internet at work? Do not include work from home. (If yes & is multi-person household) Who is that?”
INSCHL Production Wording What about at school? [Do you/Does anyone in this household] use the Internet at school? Please [only include use at school as a student/count students only], and do not include taking online classes from a location other than a school campus. (If yes & is multi-person household) Who is that? Round 1 Wording [Do you/Does anyone in this household] use the Internet at school? Please [only include use at school as a student/count students only]. Do not include taking online classes from a location other than a school campus. (If yes & is multi-person household) Who is that?	We removed the sentence “What about at school?” prior to testing because we thought it was redundant and we wanted to reduce time in the interview. We also streamlined the question text. All participants understood the question as written.	We recommend using the tested wording of this question for production: “[Do you/Does anyone in this household] use the Internet at school? Please [only include use at school as a student/count students only]. Do not include taking online classes from a location other than a school campus. (If yes & is multi-person household) Who is that?”
INCAFE Production and Round 1 Wording What about at a coffee shop or other business that offers Internet access? (If needed) [Do you/Does anyone in this household] use the Internet while at a coffee shop or other business that offers Internet access? (If yes & is multi-person household) Who is that?	Most participants understood this question as written. One participant (who was not a student) said the “library” when answering this question originally. This participant should have included “library” in INLICO. Otherwise, there were no other issues participants had when answering.	We recommend the current wording for production.
INTRAV Production Wording What about while going from one place to another? Examples include using the Internet while riding public transit, or using the GPS on your cell phone while driving. (If needed) [Do you/Does anyone in this household] use the Internet while going from one place to another? (If yes & is multi-person household) Who is that? Round 1 Wording What about while going from place to place? Examples include using the Internet while riding public transit, or using Google Maps or Waze on your cell phone while driving. (If needed) [Do you/Does anyone in this household] use the Internet while going from one place to another? (If yes & is multi-person household) Who is that?	Prior to testing, we updated the question wording to include “Google Maps” and “Waze” to assist the participant in comprehension. This question had been found to take longer to administer than other questions in the series in prior production cycles. All participants found this question to be easy to answer and no participants were read the ‘if needed’ text. A majority of participants noted that they used a service such as Google Maps or Waze and seemed to understand these examples as GPS apps. We asked participants in probing if there were other ways they might use the internet while traveling from place to place, and those mentioned included: fitness apps to track distance when using a bicycle to move from place to place; using music streaming services (such as Spotify); listening to podcasts; using messaging services. One participant mentioned that they were also thinking of using technology while on an airplane.	We recommend using the revised wording that was tested: “What about while going from place to place? Examples include using the Internet while riding public transit, or using Google Maps or Waze on your cell phone while driving. (If needed) [Do you/Does anyone in this household] use the Internet while going from one place to another? (If yes & is multi-person household) Who is that?”
INLICO Production and Round 1 Wording What about at a library, community center, park, or other public place? (If needed) [Do you/Does anyone in this household] use the Internet at a library, community center, park, or other public place? (If yes & is multi-person household) Who is that?	All participants seemed to understand the question as written. One did ask if a library on a school campus counts as a public place or if it should be counted as a school; he ultimately said that he or somebody in his household used the internet at a library which seemed to be the correct response.	We recommend the current wording for production. Question for NTIA: If a participant answers ‘Yes’ for a member using the Internet on a school campus library, that is in scope, correct?
INELHO Production and Round 1 Wording What about at someone else’s home? (If needed) [Do you/Does anyone in this household] use the Internet at someone else’s home? (If yes & is multi-person household) Who is that?	Sixteen of the twenty participants answered ‘Yes’ to this question. Most participants understood and answered this question without much issue. One participant said ‘No’ to this question originally but then once they were read INOTHR, they asked what the prior question was. After the interviewer read the intro text to the section, the participant changed their answer to ‘Yes’ for INELHO. This may have been a unique case where the participant didn’t encode the question correctly at first. Otherwise, there were no other issues answering this question.	We recommend the current wording for production.
INOTHR Production and Round 1 Wording [Do you/Does anyone in this household] use the Internet at some other location we haven’t covered? (If yes & is multi-person household) Who is that?	Seven of the twenty participants answered ‘Yes’ to this question. Although participants were not probed about this other location, some of the participants who answered ‘Yes’ voluntarily mentioned the other place they were thinking about. Two participants mentioned hotels and two other participants mentioned airports. One participant mentioned the social services building.	We recommend the current wording for production.
NETCHK Production and Round 1 Wording Based on your responses, it sounds like (list names where all IN* != 1) [does/do] not use the Internet. [Does/Do] [he/she/these people in your household] use the Internet from any location, and for any purpose? (If additional users) Who uses the Internet from any location, and for any purpose?	This question was only asked of household members who did not use the Internet at any location in the past 6 months. Only two participants were asked this question. One participant answered as a proxy for their parent and they said “No”. The other participant, who is visually impaired, answered for themselves and said they did not use the Internet at any location. This participant said “No” but also said the following while answering: Participant: “No unless it’s considering calling on the iPhone. I don’t know if that’s considered using the Internet or Alexa. That’s the only things I can think of.” Interviewer: “If I were not here, how would you answer?” (paraphrased) Participant: “No” This was the same participant mentioned earlier that uses Alexa to access the Internet. Question for NTIA: Is there a reason why “you” isn’t a fill with “he”, “she”, and “these people in your household”? We have the “you” fill for other questions in the questionnaire	We recommend adding a “you” fill to be consistent with the rest of the questionnaire.
MOBDAT Production and Round 1 Wording [Do you/Does anyone in this household] access the Internet using a data plan for a cell phone, smartphone, tablet, mobile hotspot, or other device? This type of Internet service is provided by a wireless carrier, and may be part of a package that also includes voice calls from a cell phone or smartphone. Yes No	All participants said ‘Yes’ to this question except one participant who said “Don’t Know” as they were not sure. Almost all participants found this question easy to answer. A few participants mentioned being on family plans or being able to use the Internet on a hot spot or off Wi-Fi. We probed on the second sentence of the item, “This type of Internet service is…. from a cell phone or smartphone”, to see if participants thought this was necessary wording to include to help them answer the question. Most of the participants found the second sentence helpful when answering. About six participants seemed to indicate that the second sentence was helpful but not necessary in answering the question. When probed, one participant did find the “voice calls” phrase confusing and wasn’t sure what that meant (they answered the survey question correctly). Another participant who was a limited Internet user found the second sentence a little confusing, but he mentioned that he was not up to date with the technology vocabulary. The one participant who said “Don’t Know” seemed to have a data plan but her husband handles the plan. When we showed the question on screen, she indicated that it was easier reading it than hearing it (this person was an older adult). She thought maybe the “hotspot” wording confused her when first hearing it. This participant struggled with other questions in the questionnaire. One other participant also mentioned the question was a little more difficult since they’re on a family plan but someone else controls the data. They did mention the “hotspot” wording could be confusing because they don’t use it and they believed it could be separate from the data plan. This participant correctly answered “Yes” during the interview.	Although one participant reported “Don’t Know”, we think this is a unique case and the current question wording is most optimal.
HOMTE Production Wording I am going to read a list of ways that people access the Internet from their homes, other than a mobile data plan. At home, [do you/does anyone in this household] access the Internet using: Read and select all that apply High-speed Internet service installed at home, such as cable, DSL, or fiber-optic service? (If needed) This type of Internet service is often provided by a cable company or phone company. Satellite Internet service? (If needed) This type of Internet service is received through a satellite dish installed at home. Dial-up service? Some other service? (Do not read) None Round 1 Wording (First 10 interviews) Same question wording except removed Dial-up service from the list of options Round 1 Wording (Last 10 interviews) Same question wording as first 10 interviews except revised the first option to the following: High-speed Internet service. This type of service is often provided by a cable company or phone company. (If needed) This can include cable, DSL, or fiber-optic service.	Both tested wordings for this question were problematic for participants. Eighteen participants said, “High-speed Internet service”, three participants said, “Satellite Internet service”, and four participants said, “Some other service”. In the first 10 interviews, one participant said “No” originally to “High-speed Internet service” but after hearing the other options, they mentioned that their answer to “High-speed Internet service” may have been ‘Yes’. After re-reading the question and the ‘if needed’ text multiple times, the participant finally reported ‘Yes’ which was the correct answer. In probing, the participant mentioned that he wasn’t sure what DSL was and he thought fiber-optic service was not laid everywhere (he didn’t have cable but had a router through Xfinity). Another participant reported that they had “Satellite” which was incorrect. This participant mentioned having a Xfinity subscription shared with a neighbor, but they don’t have a router and use a hotspot. In retrospective probing, this participant also mentioned they use a combination of the hotspot on their phone and a hotspot purchased separately to access the internet at home. (Question for NTIA: How should this participant have answered this question?) After the first 10 interviews, we modified the wording so the examples would be read only ‘if needed’. This did not seem to help participants in answering correctly. One participant said only “Some other service” when answering and said “Wi-Fi router” as the other service. They answered this way because they did not think their Verizon plan was high-speed Internet service, though it seems like they should have answered “High-speed Internet service.” Another participant answered both “High-speed Internet service” and “some other service” because they thought Xfinity and Internet were two different providers (it seemed like they should have answered just “High-speed Internet”). Another participant reported both “High speed Internet service” and “Satellite”. However, based on responses in retrospective probing, it seems like they should have just reported the former option as they incorrectly interpreted the “Satellite” question. Participants were probed on whether they liked the revised testing wording used in the last ten interviews or the production wording tested in the first 10 interviews. There seemed to be a slight preference for the wording used in the first interviews. A few of the participants who preferred the originally tested wording appeared to like the examples in the beginning of the question. Prior to cognitive testing, we removed “Dial-up service” as a response option because not many respondents answered “Yes” to that option in the previous production cycle and there were many more responses in “Some other service”. For the other participants not already mentioned who said, “some other service”, one participant reported a hot spot and the other participant mentioned Internet came through a landline because Internet and landline were on the same bill. To ensure that “Dial-up service” is not a common Internet service anymore, we asked participants during retrospective probing if there were any other types of ways that people can access the Internet. Only three participants mentioned “dial-up service” but none of these participants reported having “dial-up service” or spontaneously said they know someone who has it. Eight participants mentioned a hot spot as an “other service” while a couple of participants mentioned Ethernet and one participant mentioned a digital SIM card. The other participants who were asked could not come up with another service.	Since the revised wording did not perform well, we recommend additional CAPI testing of this item in the next testing cycle. We recommend testing revised versions of this question in self-administered testing (mentioned later in the recommendations). For CAPI production, we recommend the wording that was used in the first 10 interviews: I am going to read a list of ways that people access the Internet from their homes, other than a mobile data plan. At home, [do you/does anyone in this household] access the Internet using: Read and select all that apply High-speed Internet service installed at home, such as cable, DSL, or fiber-optic service? (If needed) This type of Internet service is often provided by a cable company or phone company. Satellite Internet service? (If needed) This type of Internet service is received through a satellite dish installed at home. Some other service? (Do not read) None We also are comfortable removing ‘Dial-up service’ since this is not as common to Internet users anymore. For self-administered testing, we recommend doing one of the two below (or a combination): Use something similar that is in the American Community Survey (ACS) instrument (note, dial-up Internet service option is removed here): Do you or any member of this household have access to the Internet using a – Cellular data plan for a smartphone or other mobile device? Broadband (high speed) Internet service such as cable, fiber optic, or DSL installed in this household? Satellite Internet service installed in this household? Some other service? Specify service Revise the production wording to include examples: I am going to read a list of ways that people access the Internet from their homes, other than a mobile data plan. At home, [do you/does anyone in this household] access the Internet using: Read and select all that apply High-speed Internet service installed at home, such as cable, DSL, or fiber-optic service? (If needed) This type of Internet service is often provided by a cable company or phone company, such as Xfinity or Verizon. Satellite Internet service? (If needed) This type of Internet service is received through a satellite dish installed at home, such as DirectTV. Some other service? (Do not read) None
HNETQL Production and Round 1 Wording In [your/your household’s] experience over the past six months, does your [household’s] Internet connection at home fill [your/the household’s] needs, including for speed, reliability, and, if applicable, data caps? Yes No	Seventeen of the twenty participants answered “Yes” to this question. Participants did not have difficulty answering this question. One participant who said “Yes” mentioned the Internet only somewhat meets their needs. Another participant who said “Yes” mentioned that the Internet goes in and out, but their service provider is not the most reliable.	We recommend the current wording for production.
HNETST Production and Round 1 Wording At any point during the past six months, did [you/this household] temporarily lose a home Internet connection due to difficulty paying? Yes No	Only two of the twenty participants answered “No” to this question. Two participants needed the question re-read to them but both participants still answered correctly. One of these participants seemed to have trouble understanding the question because it was a question they had difficulty encoding (even though they had the right interpretation): “Question is confusing to me because it’s asking me did you lose your connection because you did not pay your bill”. The participant said they only lost Internet because it was down in the community, so they correctly answered ‘No’.	We recommend the current wording for production.
USEINT Production Wording We are interested in learning about the applications and services people commonly use with on the Internet. [(If is multi-person household) We have randomly selected one member of your household for this section.] Consider how [you personally/NAME] currently use[s] the technologies we’ve been discussing, whether at home or any other location. Please focus on activities [you have/NAME has] done online at least occasionally during the past six months, using an Internet-connected device such as a computer, tablet, or cell phone. Round 1 Wording We are interested in learning about the applications and services people use on the Internet. [(If is multi-person household) We have randomly selected one member of your household for this section.] Consider how [you personally/NAME] currently use[s] the technologies we’ve been discussing, whether at home or any other location. Please focus on activities [you have/NAME has] done online at least occasionally during the past six months.	Participants generally did not have any problems understanding this instruction although we did not explicitly probe on this. A couple of participants needed clarification we were asking about the past 6 months during these series of questions. Once the interviewer clarified the time frame, there were no issues understanding the time frame. During retrospective probing, participants were asked how easy or difficult it was to answer this series of questions as a proxy for the other household member. Most participants said these questions were easy to answer. One participant did mention these questions were a little more difficult because she and her roommate did not discuss everything; she did mention being confident but not 100% in her answers about the roommate’s Internet use. Another participant mentioned the questions about Internet use outside of home were particularly difficult. A couple of participants mentioned some questions being difficult as they weren’t able to observe some activities. For example, one participant said that those activities social in nature (Facetime, anything with audio component) were easier to notice compared to paying bills online.	We recommend the tested wording for production.
EMAIL Production and Round 1 Wording [Do you/Does NAME] use email? Yes No	This question was clear to all participants, and all answered with no difficulty.	We recommend the current wording for production.
TEXTIM Production Wording What about texting or instant messaging? [Do you/Does NAME] text or use instant messaging? Yes No Round 1 Wording What about texting or instant messaging? (If needed) [Do you/Does NAME] text or use instant messaging? Yes No	Prior to testing, we modified the second sentence to be read ‘if needed’. We thought this second sentence was unnecessary to be read to everyone and we wanted to reduce time in the interview. Generally, there was no issue with this question. One participant said 'No' originally but then when they heard 'instant message', they changed their answer to 'Yes’. They were thinking about how their spouse sends an "instant message" through the computer when they can’t leave voicemails to their friends. This participant, who was visually impaired, mentioned that their spouse can read and respond to texts better than he can but does not text very often.	We recommend the tested wording for production: What about texting or instant messaging? (If needed) [Do you/Does NAME] text or use instant messaging? Yes No
SOCIAL Production and Round 1 Wording What about using social networks, such as Facebook, Twitter, or Instagram? (If needed) [Do you/Does NAME] use social networks? Yes No	This question was clear to all participants and the ‘if needed’ text was not read. One participant said their “housemate rarely uses social media” so they were confident in their answer of ‘No’. Another participant mentioned that their spouse only uses Facebook out of these options. During retrospective probing about proxy answers, the social media question was found to be the easiest for participants to answer. For example, several participants mentioned that they knew for sure that the person they were reporting for only used one social media network (Facebook), which made it easy to answer.	We recommend the current wording for production.
GAMING Production and Round 1 Wording What about playing video games online, whether on a smartphone, console, PC, or any other computing device? (If needed) [Do you/Does NAME] use the Internet to play video games? Yes No	Most participants did not hesitate when thinking about their answers to this question. However, one participant expressed uncertainty and said: “Gaming question is pretty broad, there are very different kinds of gaming. Playing an app on the phone like Candy Crush is different than playing Call of Duty on the Xbox via the TV. Gaming on a TV is more intense and [you] can be connected to, calling in friends to play at the same time”. This person answered ‘Yes’ to this question which appeared to be the correct answer but they were unsure why these types of gaming were all included into one category when they seem like different activities. One participant said that this was the most difficult question out of the series to answer on behalf of their spouse because they wouldn’t know if the spouse plays games on their phone, but they would know if they played video games on the TV in the living room. Their final answer was ‘Yes’ to this question.	We recommend the current wording for production.
CONFER Production Wording In the past six months, [have you/has NAME] participated in video or voice calls or conferencing over the Internet, such as with Skype, FaceTime, or Zoom? (If needed) [Do you/Does NAME] participate in video or voice calls or conferencing? Yes No Round 1 Wording In the past six months, [have you/has NAME] participated in video or voice calls or conferencing over the Internet, such as with FaceTime or Zoom? (If needed) [Do you/Does NAME] participate in video or voice calls or conferencing? Yes No	Prior to testing, we removed “Skype” from the list of examples as we thought that example was redundant, and the application is much less commonly used nowadays. All participants found this question easy to answer. During retrospective probing, participants were asked if there were any video call applications they were thinking about that weren’t mentioned in the question wording. Teams and Google Meet were common voice call apps mentioned and one participant said WhatsApp would be helpful since anyone who has family overseas uses that for voice calls. One participant suggested we consider asking if people use Zoom and Teams for work meetings and video calls and other platforms like Facetime and Facebook video for personal use. In this case, using conferencing apps for work versus personal use seemed like different concepts for her. This participant answered ‘Yes’. One participant also mentioned that for certain apps she doesn’t turn her camera on, but for others she does. This did not stop them from answering the question though, and they answered, ‘Yes’, which seemed to be the correct answer.	We recommend the revised testing wording for production: In the past six months, [have you/has NAME] participated in video or voice calls or conferencing over the Internet, such as with FaceTime or Zoom? (If needed) [Do you/Does NAME] participate in video or voice calls or conferencing? Yes No Future testing should continue to evaluate including other video or voice call applications like WhatsApp.
VIDEO Production and Round 1 Wording What about streaming or downloading videos? [Do you/Does NAME] watch videos over the Internet? Examples include YouTube and Netflix. Yes No	All participants found this question easy to answer and had no issues with the terms “streaming or downloading”. All seemed familiar with the terms YouTube and Netflix, but we did not ask any probes about this.	We recommend the current wording for production.
AUDIO Production and Round 1 Wording What about streaming or downloading music, radio programs, or podcasts? (If needed) [Do you/Does NAME] listen to music, radio programs, or podcasts using the Internet? Yes No	All participants found this question easy to answer and had no issues with the terms “streaming or downloading” in reference to audio, but we did not ask any probes about this.	We recommend the current wording for production.
PUBLISH Production and Round 1 Wording What about posting or uploading blog posts, original videos, or other content that [you/NAME] created? Do not include photos or videos taken for personal use and shared to social media. (If needed) [Do you/Does NAME] publish or upload original content that [you/NAME] created to the Internet? Yes No	When probing about proxy responses, one participant mentioned that having an example of what it means to ‘upload content’ would be helpful for the average person. Despite this comment, they were able to answer the question confidently as a proxy. Another participant mentioned that this one required much more thought than the other proxy question, “to make sure he understood question”. However, after hearing the phrase "Do not include" the participant had all they needed to answer correctly.	We recommend the current wording for production.
TELEWK Production and Round 1 Wording [Do you/Does NAME] use the Internet to telecommute or work while away from [your/his/her] usual workplace, such as working from home? Yes No	Generally, there was no difficulty with the question. However, one participant mentioned that their mother (target person) is a teacher so during the pandemic she created lesson plans online from home. However, the participant answered ‘No’ to this question because it had not happened in the last 6 months.	We recommend the current wording for production.
JOBSCH Production and Round 1 Wording In the past six months, [have you/has NAME] used the Internet to search or apply for a job? (If needed) [Do you/Does NAME] use the Internet to search or apply for a job? Yes No	Most participants were able to answer this question without issue. One participant wanted it to be clear that they only “searched” for a job using the internet but did not apply online. This participant said ‘No’ but it seems like they should have answered ‘Yes’ to this question based on the current question wording. Another participant said their roommate searched and found a job online but not in the past 6 months. This participant had trouble calculating back to six months and said they may have been off with the date her roommate found the job. Ultimately, they answered ‘No’ to this question.	Although one participant incorrectly interpreted the question, we still think the current wording for JOBSCH is the most optimal.
EDTRAI Production and Round 1 Wording What about online classes or job training? (If needed) [Do you/Does NAME] use the Internet for educational classes or job training? Yes No	All participants found this question easy to answer and had no issues with the terms “online classes” or “job training”.	We recommend the current wording for production.
EGOVTS Production and Round 1 Wording What about accessing government services, such as registering to vote or renewing your driver’s license? (If needed) [Do you/Does NAME] use the Internet to access government services? Yes No	Most participants did not have trouble answering this question. This question needed to be repeated for one participant due to them needing clarity that we were still referring to the past 6 months. Their answer was, “no, not in the last 6 months”. Another participant mentioned that their roommate had looked up this kind of information but didn't actually do the service online (voting), so they answered ’No’ which appeared to be the correct response.	We recommend the current wording for production.
USESVC Production Wording What about requesting services provided by other people, for example hailing a ride using Uber, ordering food using DoorDash or Instacart, reserving a room through Airbnb, or requesting a home repair through Angie’s List? (If needed) [Do you/Does NAME] use the Internet to request services provided by other people? Yes No Round 1 Wording What about requesting services provided by other people? Examples include hailing a ride using Uber, ordering food using Door Dash or Instacart, reserving a room through Airbnb, or requesting a home repair through Angi? (If needed) [Do you/Does NAME] use the Internet to request services provided by other people? Yes No	Prior to testing, we modified the wording slightly. Changes were made to streamline the text and because Angie’s List had been renamed to Angi. This question had no issues and participants were familiar with the examples used.	We recommend the tested wording for production.
ESRVCS Production and Round 1 Wording What about offering [your/his/her] own services for sale via the Internet? Examples include offering rentals on Airbnb and driving for Uber or Lyft. Do not include any goods or possessions sold online, such as clothing, shoes, or crafts. (If needed) [Do you/Does NAME] offer [your/his/her] own services for sale via the Internet? Yes No	One participant hesitated on this question and expressed some difficulty answering. This participant found this question difficult because her and her housemate listed their spare room online for a new permanent housemate, which they do not consider an Airbnb. This participant was unsure if that counted as a "offering a service". The interviewer repeated the question, and their final answer was ’No’ which was the correct answer. All other participants had no difficulty answering this question.	We recommend the current wording for production.
ECOMME Production and Round 1 Wording In the past six months, [have you/has NAME] used the Internet for online shopping, travel reservations, or other consumer services on the Internet? (If needed) [Do you/Does NAME] use the Internet for online shopping, travel reservations, or other consumer services? Yes No	Participants had no issues answering this question and were able to answer within the time frame of the past 6 months. Participants seemed to understand the terms used. However, we did not probe on this wording.	We recommend the current wording for production.
EGOODS Production and Round 1 Wording [Do you/Does NAME] use the Internet to sell goods? Examples include selling items on Etsy or eBay. Yes No	Participants had no issues answering this question and seemed familiar with the examples used. However, we did not probe on this wording.	We recommend the current wording for production.
FINANC Production Wording [Do you/Does NAME] use the Internet for financial services such as banking, investing, paying bills online, or sending money to other people? Examples include using your bank’s website or app, paying a credit card or utility bill online, or sending money using Venmo or PayPal. Round 1 Wording [Do you/Does NAME] use the Internet for financial services such as online banking, investing, paying bills, or sending money to other people? (If needed) Examples include using your bank’s website or app, paying a credit card or utility bill online, or sending money using Venmo or PayPal. Yes No	Most participants did not have any difficulty answering this question. When the 'if needed' text was read during probing, several participants thought the examples were helpful, however, most were able to think of examples on their own that were in scope. One participant thought "Financial services” is a broad term, but they did not have difficulty answering this question. Similarly, after showing the question wording, one participant who said ‘Yes’ during the interview mentioned that the examples of PayPal and Venmo would have "definitely been helpful" in answering the question. They added they would have thought of more financial services besides just their personal banking apps. Another participant said “No” to the survey question but should have said “Yes” since they didn’t think of the other person in the household sending money to other people on the Internet. During retrospective probing, a couple of participants mentioned CashApp and Zelle as other applications where individuals send each other money.	Since one participant erroneously did not include applications where users send money to each other, we recommend the following revised wording for FINANC: [Do you/Does NAME] use the Internet for financial services such as online banking, investing, paying bills, or sending money to other people using services like CashApp, Venmo, or PayPal? (If needed) Examples include using your bank’s website or app, paying a credit card or utility bill online?
VOICEA Production and Round 1 Wording What about communicating with a digital assistant, such as Alexa, Siri, or Google Assistant? Yes No	Participants seemed familiar with the examples of digital assistants in the question wording. However, we did not probe on these examples. One participant wanted to clarify that this question was referring to behaviors "in the past 6 months". When the interviewer confirmed this timeframe, the participant answered ‘No’. Another participant was unsure whether a Ring doorbell was in scope but chose to include it in their answer and said ‘Yes’. It seems this participant should have answered ’No’ to this question; this participant did correctly answer ‘Yes’ in HOMIOT where the Ring doorbell was in scope.	Although one participant answered this question incorrectly, we still think this current wording and the question order of VOICEA and HOMIOT is most optimal.
HOMIOT Production Wording What about interacting with household equipment or appliances that are connected to the Internet, such as a connected thermostat, light bulb, or security system? (If needed) [Do you/Does NAME] use the Internet to interact with household equipment or appliances? Round 1 Wording What about interacting with household equipment or appliances that are connected to the Internet, such as a smart thermostat, light bulb, or security system? (If needed) [Do you/Does NAME] use the Internet to interact with household equipment or appliances? Yes No	Although most participants appeared to answer this question correctly, several participants seemed to have some difficulty in answering. During the interview, a couple of participants expressed some hesitation. After the ‘if needed’ text was read, one participant said they were unsure whether they had any of these devices and ultimately answered ‘No’ to the question. Other respondents who expressed some hesitation and were read the ‘if needed’ text also requested the question to be repeated so they could think about their answer. During retrospective probing, a few participants mentioned being unsure whether their devices fell into this category (e.g., Amazon echo dot, Ring camera). The one participant who had an Amazon echo dot answered ‘Yes’ to the survey question and the other participant who had a Ring camera answered ‘Yes’; both of these participants appeared to answer the question correctly. One additional participant who said ‘No’ during the interview mentioned in probing that she has a “smart garage” where she gets phone app notifications about the garage; this participant should have answered ‘Yes’ it seems based on this response. One participant was unsure whether to think about devices that their roommate (target person) only accesses in their home when they (the participant) are out of town, but decided to answer ‘Yes’ to this question as a proxy. Another participant thought about a Smart TV when answering this question but ended up deciding not to count it under this category which was correct. When presented with the first question wording (“smart”) and alternative question (“connected”) wording, most respondents preferred the use of ‘smart’ because it seemed to be a “more modern term” and made more sense in this context compared to the term ‘connected’. Several mentioned that ‘smart’ is commonly used, a familiar term, and “the way most devices are marketed in today’s world”. One respondent who answered ‘Yes’ to this question said they only use a home security system. They suggested that we should consider asking which smart devices they use in a list format so people would be able to choose which one they use rather than having to provide a blanket yes to this question.	Although several participants had difficulty answering this question, we still think the tested wording is the most optimal: “What about interacting with household equipment or appliances that are connected to the Internet, such as a smart thermostat, light bulb, or security system? (If needed) [Do you/Does NAME] use the Internet to interact with household equipment or appliances?” Yes No The wording “smart” also seemed to resonate more with participants so we recommend using that phrasing. One alternative is to also include examples in the ‘if needed’ text like Amazon Echo and Google Nest: “What about interacting with household equipment or appliances that are connected to the Internet, such as a smart thermostat, light bulb, or security system? (If needed) [Do you/Does NAME] use the Internet to interact with household equipment or appliances, such as Amazon Echo and Google Nest?” Yes No
MEDINT Production and Round 1 Wording (If multi-person household) Our remaining questions are about your household as a whole. (All) I’m going to ask a couple of questions about how [you have/your household has] used the Internet for health-related activities during the past six months.	Participants were largely answering this question thinking of their entire household’s experiences with online health related activities. Twelve participants were thinking of their household while four participants, who were from multimember households, were thinking of just themselves. The participants who were answering only for themselves were not technically incorrect when answering ‘Yes’, but in instances where the answer was ‘No’, there is a risk that there were inaccuracies in reporting. One participant who works in healthcare mentioned that she was answering the medical questions in the context of her work. This is a unique situation and does not present issues with the question wording as tested.	We recommend the current wording for production.
MEDREC Production and Round 1 Wording [Do you/Does anyone in this household, including you,] access health records or health insurance records online? Yes No	Participants had no difficulties in answering this question.	We recommend the current wording for production.
MEDDOC Production Wording What about communicating with a doctor or other health professional using the Internet? (If needed) [Do you/Does anyone in this household] communicate with a doctor or other health professional using the Internet? Yes No Round 1 Wording [Do you/Does anyone in this household] communicate with a doctor or other health professional using the Internet? (If needed) Examples include telehealth appointments and communicating over a patient portal or email. Yes No	Prior to testing, this question was revised for clarity. We removed the first sentence and modified the ‘if needed’ text to be read as the main question stem. We added a new ‘if needed’ text to include examples of communicating with a doctor and health professional. Participants responded well to the new wording and there were no difficulties in answering. Most participants appreciated the ‘if needed’ text, which was shown to them in probing. With the exception of one participant who was read the if needed text during the interview, the remaining participants were able to answer this question accurately without the ‘if needed’ text. Those who were not read the ‘if needed’ text did seem to be thinking of telehealth services. One participant noted that they felt that some examples of telehealth services (ex. Teladoc) might be helpful.	We recommend the revised wording that was tested: “[Do you/Does anyone in this household] communicate with a doctor or other health professional using the Internet? (If needed) Examples include telehealth appointments and communicating over a patient portal or email."
MEDINF Production and Round 1 Wording [Do you/Does anyone in this household] research health information online, such as with WebMD or similar services? Yes No	Most participants had no difficulties in responding to this question. One participant seemed a little unsure about his response during the interview. He said the following: “I’m a little unsure about [MEDINF] as the example of WebMD…I guess I feel like there’s a distinction between health information and health care information. I’m going to say Yes to that one.” This participant seemed to answer correctly. This was a unique case so we don’t think there are any issues with the question wording.	We recommend the current wording for production.
MEDMON Production and Round 1 Wording [Do you/Does anyone in this household] use an electronic health monitoring service that collects and sends data to your doctor or health care provider through the Internet? Examples include connected devices that monitor vital statistics, blood glucose levels, or blood pressure. Yes No	Participants had no difficulties in answering this question. One participant did say ‘Yes’ when the interviewer was reading the question but then switched to ‘No’ after the question was fully read. During probing, the participant mentioned that her parent used to have a heart monitor where the information would go directly to the doctor, but it hadn’t happened in the last 6 months, so the participant was thinking of the correct time frame.	We recommend the current wording for production.
UHHINT Production and Round 1 Wording (If multi-person household) We are interested in learning generally, without identifying specific people, what challenges your household has had while using the Internet. (If single-person household) We are interested in learning what challenges you have had while using the Internet.	Participants had no difficulties in understanding the direction as written.	We do not recommend changes to this direction at this time.
PSPRE Production Wording During the past year, have concerns about privacy or security stopped [you/anyone in this household] from doing any of these activities online: Read and select all that apply Conducting financial transactions such as banking, investing, or paying bills online? Buying goods or services online? Posting photos, updates, or other information on social networks? Expressing an opinion on a controversial or political issue, such as on a blog or social network, or in a forum, email or any other venue? Searching for information using a platform such as Google Search, Yahoo Search, Microsoft Bing, or another web search engine? (Do not read) None of the above Round 1 Wording (1st 10 interviews) During the past year, have concerns about privacy or security stopped [you/anyone in this household] from doing any of these activities online: Read and select all that apply Conducting financial transactions such as banking, investing, or paying bills online? Buying goods or services online? Posting photos or other information on social networks? Expressing an opinion on a controversial or political issue, such as on a blog or social network, or in an email? Finding information using Google or another search engine? (Do not read) None of the above Round 1 Wording (2nd 10 interviews) During the past year, have concerns about privacy or security stopped [you/anyone in this household] from: Read and select all that apply Conducting financial transactions such as banking, investing, or paying bills online? Buying goods or services online? Posting photos or other information on social networks? Expressing an opinion on a controversial or political issue, such as on a blog or social network, or in an email? Finding information using Google or another search engine? (Do not read) None of the above	Prior to testing, we streamlined the response options to reduce time administering the question while also maintaining clarity. In the first 10 interviews, at least two participants had to be re-read some part of the question. One participant when answering (2) mentioned that they were reluctant about purchasing goods online so the question was re-read; they eventually said “Yes”. Another participant lost track of what the question was asking so the main question stem had to be re-read. Because of these issues, we modified the main question wording slightly for the last 10 interviews to improve comprehension. In the last 10 interviews, one participant needed the question re-read because they were asking whether it was in the past 6 months (this participant struggled with other questions in the survey). When answering this question, one participant mentioned that he was not privy to his roommates’ security concerns, so he answered on his behalf. When talking about (5), another participant asked if we were just asking about Google or all search engines. The participant mentioned earlier who was unsure about the time frame also at first interpreted the question “have we used Google? Yes”. However, they then corrected themselves and said “oh, concerns about security” and then switched the response to ‘No’ which seemed to be the correct answer. When asked what time frame they were thinking of, most said they were thinking about the last year. One mentioned that she was thinking specifically of an incident that occurred a month prior to her interview, but later said that these were concerns that she had had prior to the incident. Another said that they were thinking about the past five to six years when thinking of privacy concerns. A third said that they were not thinking about any time frame in particular and instead was thinking “in general”, they did say that they would rather see the question than hear it asked aloud. This respondent did not struggle to answer the question, however. Several other participants mentioned they were thinking specifically about the past six months.	We recommend removing “doing any of these activities online” from the original production wording as that change may have helped participants better understand the question. We also recommend the tested wording for each of the five sub-items. Regarding the reference periods, we recommend one of the following below: Use the wording “past 12 months” in the question stem as previous questions use months as a unit of measure Repeat the reference period “past year” before the 3rd and 4th sub-item so participants don’t lose track of the reference period.
PSCON Production and Round 1 Wording Overall, what concerns [you/members of this household] the most when it comes to online privacy and security risks? Do not read; select all that apply and/or enter verbatim response if other Identity theft Credit card or banking fraud Data collection or tracking by online services Data collection or tracking by government Loss of control over personal data such as email or social network profiles Threats to personal safety, such as online harassment, stalking, or cyber-bullying No concerns Other: ___________________________	Participants generally did not have issues answering this question. One participant seemed confused when first read this question as it appeared they did not realize this was an open-ended question. After interviewer told them it was open-ended and re-read the question, the participant was able to answer the question. When another participant was first read the question, they asked the interviewer “do you mean of the areas mentioned previously”? The participant was then informed to answer however they interpreted it. The participant responded, “I guess financial issues” which was coded as “credit card or banking fraud”. Interviewers coded verbatim answers into appropriate categories and below is the breakdown of answers: Identity theft-10 Credit card or banking fraud-4 Data collection or tracking by online services-5 Data collection or tracking my government-2 Loss of control over personal data such as email or social network profiles-3 Threats to personal safety, such as online harassment, stalking, or cyber-bullying-0 No concerns-1 Other-7 The ‘Other’ responses varied across the participants. One participant who’s a parent mentioned that her underage child has access to the Internet at any time. A second participant mentioned “scams”. A third participant mentioned concerns about net neutrality. A fourth participant mentioned thinking that something is secure when making a purchase when in fact it is not. Another participant mentioned they considered not streaming movies from random web sites due to privacy and online safety concerns.	We recommend the current wording for production.
PSYCBA Production and Round 1 Wording During the past year, [have you/has any member of your household] experienced an online security breach, identity theft, or a similar crime? Yes No	Participants were able to answer this question without much issue. Some of the participants that answered ‘Yes’ voluntarily provided examples during the interview about how they experienced a breach or identity theft. One participant said they did not experience an actual incident but had seen some of the scams that try to get you to go to their web site. After the interviewer re-read the question, this participant answered ‘No’ which seemed to be the correct answer.	We recommend the current wording for production.
NHMINT, NONINT, EVRHOM, NOHM, PRINOH, and LOPRCE	These questions were not asked in the first round of testing since we did not have any households who did not use the Internet at home.	We recommend the current wording for these group of questions for production.
TVINT Production and Round 1 Wording Our last few questions are about the use of cable TV and satellite TV services not accessed through the Internet.	Participants understood this wording with no issue. However, they were not probed on this.	We recommend the current wording for production.
TRADTV Production and Round 1 Wording [Do you/Does anyone in this household] subscribe to cable TV or satellite TV? This often requires a cable box or satellite dish. Do not include TV streaming subscriptions, such as Netflix, Sling TV, or Hulu. Yes No	Nine of the twenty participants said ‘Yes’ to this question. All participants were able to answer this question and none of them reported any issues.	We recommend the current wording for production.
PREVTV Production and Round 1 Wording [Have you/Has anyone in this household] ever subscribed to cable TV or satellite TV? Do not include TV streaming subscriptions, such as Netflix, Sling TV, or Hulu. Yes No	The eleven participants who said ‘No’ to TRADTV were administered this question. When administered, one participant asked, “did you say anyone ever?” and asked to repeat the question (this participant needed many questions repeated throughout the interview). After re-reading the question, this participant said ‘Yes’. Another participant mentioned that their parents’ households have cable but no one in their current household has ever individually subscribed to cable TV or satellite TV so they reported ‘No’, which was the correct answer.	We recommend the current wording for production.

Time per section

We calculated the time elapsed between the time the interviewer read the first word of a question to the time the last question was answered for the following sets of questions:

• CMPINT (“This month we are asking some additional questions about Internet use...”) to HNETST (“Did this household temporarily lose a home Internet connection…”)

• USEINT (“We are interested in learning about the applications and services people use…”) to HOMIOT (“What about interacting with household equipment or appliances…”)

• MEDINT (“how your household has used the Internet for health-related activities…”) to PREVTV (“Has anyone in this household ever subscribed to cable TV or satellite…”)

CMPINT to HNETST

Minimum	5:52
Maximum	12:06
Mean	8:51
Median	9:13
Range	6:14

n = 20

USEINT to HOMIOT

Minimum	3:00
Maximum	6:23
Mean	4:29
Median	4:29
Range	3:23

n = 20

MEDINT to PREVTV

Minimum	2:35
Maximum	7:02
Mean	4:07
Median	3:46
Range	4:27

n = 20

Note, these section times reported are not meant to reflect how long these sections would take in production. Interviewers used a Qualtrics instrument that was programmed to administer the questions that differs from Blaise. In addition, interviewers sometimes would take notes on specific questions as they were being administered.

1 Two participants did not report as a proxy for someone else.

19