BRS Cognitive Interviewing Results

FROM: Struther Van Horn and Jennifer Edgar

Office of Survey Methods Research

SUBJECT: Cognitive Testing of Business Survey on the Impacts of the Coronavirus

DATE: May 8, 2020

Introduction

The Bureau of Labor Statistics (BLS) is designing a QCEW Business Survey (QBS) survey about the impact of COVID-19 on establishments, and what changes they’ve made in reaction to the COVID-19 situation. Prior to submitting the survey for OMB approval, the Office of Survey Methods Research (OSMR) conducted nine cognitive interviews to evaluate the effectiveness of the questions.

Method

The cognitive testing protocol was developed to explore topics of potential concern that arose during the development of the survey and during the multiple rounds of expert reviews. After the study was explained, participants completed the survey verbally and then answered a series of debriefing questions. Non-standardized interviewing was used, interviewers generally followed the protocol but also asked follow up probes based on participant answers or explanations as needed. An observer attended each interview taking notes. Each interview was audio recorded per participant consent.

Following each interview, the interviewer and observer debriefed, discussing the participant’s responses and explanations. These discussions served allowed for potential issues (and confirmation of no issues) to be identified almost in real time. This approach also allowed for a potential issue to be identified mid-testing, a change to be developed and approved by stakeholders, and updating wording to be tested with the remaining interviews.

Participants

We recruited participants from personal contacts of BLS staff, supplemented by one interview with a former cognitive interviewing participant. Efforts were made to seek variety in terms of: industry (targeting those likely to have experienced differing COVID impacts), number of employees, location, and single or multiple locations.

Industry	# of Employees	Location	Single or Multi Unit
Technology	3,500	Boston	MU
Hospital	2,200	Illinois	MU
Construction	150	Michigan	SU
Manufacturing	140	Washington	SU
Recreation	75	Maryland	MU
Motels	15-25	Nebraska	MU
Restaurant	25	Colorado	SU
Retail	>5	Virginia	SU
Health Care	>5	Colorado	SU

Results

Overall, cognitive testing results of the QBS COVID19 survey were positive. The questions were able to capture the intended information, and there was almost no indication of participants experiencing confusion, burden or sensitivity in reaction to the questions. Additionally, only one participant provided a ‘don’t know’ response, suggesting that the requested information is readily known without having to consult records. The cognitive testing results suggest that the survey would be able to collect accurate information.

Most participants completed the survey within 4 to 5 minutes. The one participant who took longer did so because she ‘thought aloud’ as she answered, explaining her thought process as she went.

Several participants commented that the survey was “easy to understand and answer” and that the survey was “clear” and “concise and to the point”. Additionally, several participants commented on the content of the questions, stating that the survey is asking about important topics that are close to businesses priority topics. When participants were probed on whether or not they felt businesses would be likely to respond to a survey request with these questions, the general perception was that given the survey is only 7 questions, many people would be likely to respond to such a survey request. Participants made comments that the survey was “not overly burdensome” and “many people would be willing to answer these questions.”

Question by Question Results

Instructions:

Instructions: The Bureau of Labor Statistics (BLS) is looking to understand the impact of the coronavirus (also known as COVID-19) on your business and the U.S. economy. We know your business is likely facing challenges, and we appreciate your response to this important survey.

This survey is concerned with how the Coronavirus pandemic has affected this business location. Please answer the following questions about this business location’s experience beginning March 1, 2020.

Please answer the following questions for this business location only.

Participant feedback on the instructions were favorable. Many participants commented that these instructions were “clear and easy to understand”. When participants were probed on the date when changes to their businesses occurred due to COVID, the March 1^st, 2020 predated any dates given by participants. Dates given by participants tended to be on or around March 15^th, 2020.

For participants who had multiple locations, they understood that the questions should be answered in thinking about a specific location.

Question 1:

As a result of the Coronavirus pandemic, which of the following did this business location experience? (Mark all that apply)

☐Shortage of supplies or inputs

☐Decrease in demand for products or services

☐Increase in demand for products or services

☐Inability to move or ship goods

☐Government-mandated closure of this business location

☐Other, specify:

☐None

☐Don’t know

Question #1 tested well. The majority of participants felt that the question made sense and that the response options were comprehensive enough to address the variety of changes that had happened to their business. Almost all participants selected more than one response option, with one explicitly saying they appreciated that the question was a ‘Mark all that apply’ as it allowed for them to represent the multiple changes that had occurred within their business. Some selected seeming contradictory response options (e.g., ‘decrease in demand’ and ‘increase in demand’), and the debriefing revealed their responses were accurate.

One participant misinterpreted ‘demand for products or services,’ thinking about revenue resulting from those products or services, rather than customer demand. During the debriefing, it was revealed that while his rationale for selecting that answer was incorrect, there was a decrease in customer demand tied to the decrease in revenue, so the answer did accurately reflect the company’s situation.

Only two participants indicated they would use the ‘other, specify’ response option, a positive finding as the goal is to have the close-ended response options capture the majority of the information. One indicated that they would write in that their business closed, as it was a decision they made corporately, not just due to a government mandate. The other provided a long explanation of the situation their company faced, but during the debriefing discussion recognized that most of the impacts were reflected in the other response options.

As a result of testing, no changes were made to this question, the question is expected to work as intended.

Question #2:

As a result of the Coronavirus pandemic, what changes, if any, were made to employment or payroll at this business location? (Mark all that apply)

☐Hired additional employees

☐Laid off or furloughed employees, employees worked no hours

☐Increased employees’ hours of work

☐Reduced employees’ hours of work, employees worked at least some hours

☐Increased salaries and wages

☐Reduced salaries and wages

☐Other, specify:

☐None

☐Don’t know

Overall, question #2 tested well. Participants understood the intent of the question and were able to successfully able to map their company’s situation to the response options. As with Question 1, almost all participants selected more than one response option, with some selecting some seemingly contradictory options.

The one potential problem identified during testing was with the term ‘laid off or furloughed.’ Three of the first six participants hesitated selecting that option as they weren’t sure their employees were furloughed (because they were part time or because they were still being paid). Based on those results, and in consultant with the economists who developed the survey, the language was changed in questions 2, 3 and 4.

Revised Question #2:

As a result of the Coronavirus pandemic, what changes, if any, were made to employment or payroll at this business location? (Mark all that apply)

☐Hired additional employees

☐Increased employees’ hours of work

☐Reduced employees’ hours of work, employees worked at least some hours

☐Told employees not to work, with or without pay (for example, laid off or furloughed)

☐Increased salaries and wages

☐Reduced salaries and wages

☐Other, specify:

☐None

☐Don’t know

The re-worded question was shown to two of the participants who had difficulty with the original language. Both participants said that the newly reworded question was preferable to the original wording. One participant commented that it “makes a lot more sense” as you “don’t have to worry about the legal definition of furlough.” The other commented that the question was “more easily understood” and was “articulated well.” Additionally, this participant mentioned that from their experience in speaking to other colleagues, ‘told employees not to work, with our without pay’ is a very common situation facing businesses. Both participants said that they would check this response option (which correctly reflected their businesses situation).

Two interviews were conducted after the wording change. One participant understood the wording as intended. When asked if he was familiar with the terms ‘laid off’ and ‘furloughed’ he said he had a basic understanding but wasn’t confident in the details and felt that the ‘told not to work’ language was much clearer.

The other participant who received the revised wording understood the wording, but didn’t want to select it as he was proud of not having ‘laid off or furloughed’ any employees; all his employees who were told not to work received full pay (to him those terms imply without pay). During the debriefing, he recognized that the response option did apply to him, due to the ‘with or without pay’ clause. He said if the parenthetical had not been in the response option, he would have selected it. He did write “employees told not to work, but all were paid” in the ‘other, specify’ response option, which would have led to his situation being accurately captured in the data.

Overall, the testing suggests that the question will be effective and is expected to work as intended. There is a possibility that some issues remain around ‘laid off or furloughed,’ but we expect overall the question will correctly represent the respondents’ situations.

Question #3:

Did this business location continue to pay some or all employees who were laid off or furloughed as a result of the Coronavirus pandemic while they were not working?

☐Yes

☐No

☐Not applicable, no employees were laid off or furloughed

☐Don’t know

Generally, question #3 tested well. Participants generally understood the intent of the question and were able to successfully able to map their company’s situation to the response options. Participants answered the range of responses correctly.

Even the participants who should have selected ‘Laid off or furloughed employees, employees worked no hours’ in question #2 but did not were able to understand this question and correctly answer this question, seeming to understand the intent of the question.

In order to make this question consistent with changes made to question #2, question #3 was also changed after 6 interviews:

Revised Question #3:

Did this business location continue to pay some or all employees who were told not to work as a result of the Coronavirus pandemic while they were not working?

☐Yes

☐No

☐ Not applicable, no employees were told not to work

☐Don’t know

The re-worded question was shown to two of the participants who had difficulty with the original language. Both participants said that the newly reworded question was preferable to the original wording. One participant commented that the edits “seemed reasonable” and that the question was “easy to answer”. The other participant mentioned that this was a “great way to word it” as “each organization has different nomenclature, and this takes that out of it”.

One interview was conducted after the wording change. That participant understood the intended meaning and had no difficulty answering the question.

Overall, the testing suggests that the question will be effective and is expected to work as intended.

Question #4:

Did this business location continue to pay a portion of health insurance premiums for some or all employees who were laid off or furloughed as a result of the Coronavirus pandemic?

☐Yes

☐No

☐Not applicable, no employees were laid off or furloughed

☐Not applicable, this business location does not offer health insurance

☐Don’t know

In general question #4 tested well. Participants understood the intent of the question and were able to successfully able to map their company’s situation to the response options. Participants answer the range of answers correctly. Companies who did not provide health insurance were able to correctly answer the question. One company had a somewhat complex situation, with employees being given the option to opt out of health insurance without penalty, and that participant was also able to respond correctly. Additionally, participants were probed on the difference between both ‘Not applicable’ response options and were able to differentiate between the two.

In order to make this question consistent with changes made to question #2, question #3 was also changed after 6 interviews:

Revised Question #4:

Did this business location continue to pay a portion of health insurance premiums for some or all employees who were told not to work as a result of the Coronavirus pandemic?

☐Yes

☐No

☐Not applicable, no employees were told not to work

☐Not applicable, this business location does not offer health insurance

☐Don’t know

The re-worded question was shown to two of the participants who had difficulty with the original language. Both participants said that the newly reworded question was preferable to the original wording. The other participant mentioned that this was “good wording” and from his perspective, is preferable as the edits broaden the question.

One interview was conducted after the wording change. That participant understood the intended meaning and had no difficulty answering the question.

Overall, the testing suggests that the question will be effective and is expected to work as intended.

Question #5:

Did this business location offer more opportunities for employees to telework (work remotely) as a result of the Coronavirus pandemic? (Mark all that apply)

☐Yes, offered telework to employees who could not telework prior to the Coronavirus pandemic.

☐Yes, increased number of telework hours for employees already permitted to telework.

☐No change in existing telework arrangements.

☐No telework at this location both before and after the Coronavirus pandemic.

☐Don’t know

Question #5 tested well. All participants indicated that they understood the question and the response options.

Participants were probed specifically on whether or not they understood the difference between the first response option, ‘Yes, offered telework to employees who could not telework prior to the Coronavirus pandemic’ and the second response option, ‘Yes, increased number of telework hours for employees already permitted to telework’. All participants indicated that they understood the difference, with the first response option relating to employees who did not previously telework were now being offered the option to telework and the second option relating to employees who were already telework eligible, had an increase in telework hours. Several participants selected both ‘yes’ options, providing evidence they were able to map their situation to the response options.

Only one participant expressed any potential difficulty with the question; the participant from the construction firm said he’d select the second ‘yes’ option, but didn’t feel that it completely represented his situation since employees were “forced to telework” when worksites closed. Those that typically do construction duties did get some new tasks that could be done from home, but the nature of work was different (and they were not required to telework).

As a result of testing, no changes were made to this question, the question is generally expected to work as intended.

Question #6:

As a result of the Coronavirus pandemic, did this business location increase the amount of paid sick leave provided to employees? (Mark all that apply)

☐Yes, provided paid sick leave to employees who did not have paid sick leave prior to the Coronavirus pandemic.

☐Yes, increased amount of paid sick leave for employees who already had sick leave prior to the Coronavirus pandemic.

☐No change to paid sick leave or no paid sick leave provided.

☐Don’t know

Question #6 tested well. Participants indicated that they understood the question and the response options. Participants were probed to see if these response categories covered all of the potential scenarios with paid sick leave. Participants indicated that the response options did account for a variety of potential options that would occur for paid sick leave. Participants were also probed on whether the response options allowed for them to account for any differences across employees. Participants indicated that the ‘mark all that apply’ feature allowed for them to account for differences across types of employees. Additionally, a few participants do not offer ‘sick leave,’ instead offer ‘paid time off’ which includes paid sick leave, annual leave, and personal leave. Those participants had no trouble understanding the question and answering accurately, nor did the participant from a company that offers unlimited, and untracked, paid time off.

As a result of testing, no changes were made to this question, the question is expected to work as intended.

Question #7:

Federal and state governments have created loans and grants to help businesses that have been hurt by the Coronavirus pandemic. Sources of these loans and grants, some of which were created under the CARES Act, include the Paycheck Protection Program, Main Street Lending Program, Economic Injury Disaster Loan, and Secondary Market Corporate Credit Facility.

Did this business location receive a Coronavirus-related loan or grant tied to re-hiring or maintaining employees on the payroll?

☐Yes

☐No

☐Don’t know

Generally, question #7 tested well; participants understood the intent of the question and were able to answer accurately for their company’s situation. Although none of the participants had yet received relevant payment, they all were aware of the types of programs that the question was referring to. The ‘paycheck protection program’ was the most recognizable of the program, though most participants recognized the rest of the programs on the list (with the exception of the ‘secondary market corporate credit facility’). Participants correctly associated the Paycheck Protection Program as being ‘tied to re-hiring or maintaining employees.’

This question was the only one where one participant replied ‘don’t know.’ She understood the intent of the questions, and knew that the selected location had applied for a relevant program, but was not informed if they had received it, noting that information would be housed in a different office. Two participants indicated they would know if a payment had been received, but not necessarily if an application had been submitted, confirming the focus of the question.

As a result of testing, no changes were made to this question, the question is expected to work as intended.

Question #8:

Would you like to provide any information about changes affecting employment that this business location made in response to the Coronavirus pandemic?

Question #8 was added to the survey after the majority of the interviews were completed, and so was not tested. It is an optional question at the end of the survey, and the data will not be tallied and published with the main tables.

10

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	VanHorn, Struther - BLS
File Modified	0000-00-00
File Created	2021-01-13