ASM Cognitive Report on Coronavirus Pandemic and revised Payroll and Employment Questions

Findings and Recommendations from Cognitive Testing for the Annual Survey of Manufactures

Prepared for:

Marlo Thornton, Economy-Wide Statistics Division

Prepared by:

Melissa A. Cidade, Response Improvement Research Staff

Rebecca Keegan, Response Improvement Research Staff

Office of Economic Planning & Innovation

Economic Programs Directorate

U.S. Census Bureau

October 15, 2020

The Census Bureau has reviewed the report for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices applied. (Approval ID: CDRB #CBDRB-FY21-ESMD001-001).

Table of Contents

Executive Summary 4

Research Objectives 7

Research Methodology 7

Findings and Recommendations 9

General Findings and Recommendations 9

Finding #1: Overall, respondents report that the new questions are easy to answer. 9

Finding #2: Most respondents cannot answer the ASM on their own. 10

Specific Findings and Recommendations 11

Questionnaire Component: Payroll Question Series 11

Finding #3: The revised order of the payroll questions works for respondents. 11

Finding #4: Respondents classify production workers differently. 11

Finding #5: Most respondents can provide quarterly production numbers. 12

Questionnaire Component: NAPCS Additions Question Series 13

Finding #6: Respondents understand the additional products and why they have been added. 13

Finding #7: Data on new products may be inconsistent. 14

Questionnaire Component: Donated Products Question Series 14

Finding #8: Respondents use disparate definitions of the words ‘donate’ and ‘product.’ 14

Finding #9: Data on donations may be inconsistent. 16

Questionnaire Component: Days Closed Question Series 17

Finding #10: Respondents use disparate definitions of the word ‘closed.’ 17

Finding #11: Data on closures may be inconsistent. 19

About the Response Improvement Research Staff (RIRS) 20

Appendix A: Cognitive Interviewing Methodology 21

Appendix B: Cognitive Interviewing Protocol 24

Appendix C: Test Instrument 36

Executive Summary

Research Objectives – This research project was designed to test on the Annual Survey of Manufactures (ASM) two new question series developed in response to the COVID-19 global pandemic, and reordered questions on payroll. Additionally, testing included the addition of six COVID-specific products included in industry-specific product listings.

Methodology – Researchers used cognitive interviewing, a qualitative data collection methodology that uncovers measurement error caused by mismatches in understanding the question and the meaning of the question to test these items. In total, we conducted 21 phone interviews with respondents familiar with the ASM. Respondents came from various positions within companies, and represent 10 broad industry classification codes.

Findings and Recommendations

General

Finding #1: Overall, respondents report that the new questions are easy to answer. While there are some changes to the questions that may enhance their performance, overall, respondents did not struggle with the new questions.

Finding #2: Most respondents cannot answer the ASM on their own. Over and over again, respondents mentioned that responding to the Annual Survey of Manufactures is a collaborative effort with various colleagues throughout their company.

Specific

Payroll Question Series

Finding #3: The revised order of the payroll questions works for respondents. Most respondents noted that the order of the questions does not impact their responses, nor does it necessarily add burden to responding.

Finding #4: Respondents classify production workers differently. Without a clear definition of production and non-production workers, respondents use disparate criteria to classify workers into these two categories.

Recommendation: Retain the ASM statement on how to define and parse production workers and non-production workers.

Finding #5: Most respondents can provide quarterly production numbers. In some instances, these data are already being tracked; in others, the burden of pulling these data is apparent but limited.

NAPCS Additions Question Series

Finding #6: Respondents understand the additional products and why they have been added. All respondents –regardless of shifts in production – knew that the six additional products were added to their listing as a result of the COVID-19 global pandemic.

Finding #7: Data on new products may be inconsistent. Some firms are closely tracking additional products manufactured in response to the global pandemic; others find it challenging to find the appropriate data to answer these questions.

Recommendation: While the question stem clearly indicates that estimates are acceptable, we may want to consider drawing more attention to this caveat, since it seems the burden of gathering these data could be higher for some manufacturers than for others.

Donated Products Question Series

Finding #8: Respondents used disparate definitions of the words ‘donate’ and ‘product.’ Some respondents were unclear if this question includes only the donation of items produced at their manufacturing plants, or if any donations should ‘count’ for this question. Likewise, a few were unsure if this question only wanted COVID related donations or if any should be included.

Recommendation: To ameliorate miscommunication on this question, we recommend specifying to respondents that this questions asks about any manufactured donations, not just COVID-related.

Finding #9: Data on donations may be inconsistent. Some companies are tracking donation data systematically; others are haphazardly tracking donations or are not tracking in the aggregate at all.

Recommendation: Since these data are less concrete for some establishments compared to others, we recommend retaining the explicit statement already on the ASM that respondents can provide their best estimate for these questions. That way, we can lower the risk of item non-response, while also recognizing that these data are a best estimate.

Days Closed Question Series

Finding #10: Respondents use disparate definitions of the word ‘closed.’ Some mentioned that even when manufacturing plants are closed, there is usually some staff – security, maintenance, and others – who are in the building, making it unclear if that ‘counts’ as closed. Others noted that even if a location is not productive, work is still happening remotely – which lead them to wonder if the company counts as ‘closed.’ Finally, closure relationship to COVID-19 was unclear to some respondents, noting that plants can be shut down by state mandate, or as a result of declining demand because of COVID-19.

Recommendation: Because of the ambiguity of the word closed, we recommend providing guidance for respondents in the form of a definition or by rewriting the question entirely.

Finding #11: Data on closures may be inconsistent. Some firms are closely tracking closures, some are informally tracking closures, and some are not tracking closures, leading to differences in data availability to respond to this question.

Recommendation: We recommend retaining the statement on the ASM alerting respondents again to estimate their responses, and explicitly stating that estimates are acceptable.

Research Objectives

On March 13, 2020, a national emergency was declared by the Executive Branch in response to the growing COVID-19 global pandemic. As a result of this unprecedented event, Census Bureau researchers determined that additional questions would need to be added to the Annual Survey of Manufactures (ASM) to measure the changing landscape of manufacturing in the United States. The ASM team developed the following changes to the survey instrument in response to COVID-19:

• Payroll Questions Series - the ASM team has reorganized how the Payroll and Employment questions are asked and have expanded the question to now ask establishments for quarterly payroll for production workers as opposed to just annual payroll collected previously.

• NAPCS Question Series - the ASM currently collects detailed sales values for the North American Product Classification System (NAPCS); the ASM team added six product descriptions covering products that were necessary during the coronavirus pandemic to all manufacturers.

• Donated Products Question Series – in response to a perceived increase in donated products, the ASM team added two questions to measure the value of any products donated.

• Days Closed Question Series – across the country, a patchwork of state and local regulations closing or limiting performance of businesses and other institutions caused some manufacturers to close for one or more days; in response, the ASM team added a question on the number of closed days per quarter for manufacturers.

In response to these newly developed questions, the Economic Statistical Methods Division (ESMD) Data Collection Methodology and Response Branch (DCMRB) concluded that cognitive interviewing is the most appropriate methodology for understanding the validity of the new items. To that end, the following research questions guided testing:

• Do respondents understand what information they are being asked to provide with these questions?

• Do respondents understand the terminology used in the questions?

• Are respondents able to answer the questions that are being asked?

• Is the requested information available in respondents’ records?

• Does the revised order of the Payroll and Employment questions make sense to respondents?

• Does the order of the questions about donations (Sales first then Special Inquiry) make sense to respondents?

Research Methodology

Cognitive interviewing is a method of pre-testing survey questions that involves in-depth interviewing, paying particular attention to the mental processes respondents use to navigate the platform¹. Cognitive interviewing uses a framework dependent on evaluating questions against their outcome objectives, including the research questions outlined above, and to what level of accuracy respondents can provide data in response. It is dependent on a “think aloud” technique that encourages respondents to “verbalize thoughts while engaged in a cognitive activity with little interjection by the interviewer other than” to keep the respondent thinking out loud². In engaging respondents in this exercise, their cognition is slowed while not impacting their task performance; that is, the act of asking respondents to think aloud while responding to survey questions does not necessarily change their interactions with the questions, even though it causes them to move through the survey more slowly³.

Cognitive interviewing depends on a semi-structured protocol. The protocol acts as a guide for the interview, emphasizing areas of interest to make sure that the research questions are being addressed, while also allowing maximum flexibility for the researcher. The protocol for this project is provided in Appendix B of this report. More information on the methodological underpinnings of cognitive interviewing is outlined in Appendix A.

For this cognitive testing, researchers conducted 21 phone interviews between September 14 and September 26, 2020. Prior to the interview, researchers emailed respondents an online consent form to document their voluntary participation in the study. Researchers also emailed an online version of the survey questions to be tested, with the instruction to refrain from clicking on the URL for the survey until the time of the interview. In this way, researchers were able to capture respondents’ “first impressions” of the questions.

All 21 respondents were familiar with, had previously completed, or were now responsible for completing the ASM. Interviews lasted no more than 30 minutes. Most respondents came from their company’s accounting department. See Table 1 for an overview of respondent professional titles.

Table 1: Respondent Titles

Title	N
Reporting	3
Accounting	9
Controller or other C-Suite	7
Other or uncodeable	2
Total	21

Respondents ultimately represent 18 different specific industry classifications codes, and 10 broader industry classification codes⁴. Table two lists collapsed classifications and the number of respondents from each.

Table 2: Combined NAICS Categories of Respondents

Industry*	N
Textile Mills and Apparel Manufacturing	7
Electrical, Metal, and Chemical Manufacturing	5
Paper Manufacturing	3
Other Miscellaneous Manufacturing	6
Total	21

*Industries have been collapsed to protect respondent anonymity.

Findings and Recommendations

Below are the findings and recommendations from the cognitive testing. The first section – labeled General Findings and Recommendations – outlines overall issues with the ASM questions that were tested, including comprehension of the questions overall and the collaborative approach to response companies use to complete the ASM. The second section – labeled Specific Findings and Recommendations – provides feedback on the specific question series tested with respondents.

General Findings and Recommendations

Finding #1: Overall, respondents report that the new questions are easy to answer.

At the conclusion of the interview, we asked respondents about their impression overall with regard to the new questions we had tested. Most respondents said that the questions were not challenging. Said one, the new questions are “pretty much easy,” while another said that they were “not difficult to answer.” Note that a few respondents mentioned that while the questions are not themselves difficult to answer, they may require additional time or research. Said one, the questions are “not too bad,” but that “it just takes some time and having to do research for some of them.” Finally, another noted that additional questions add some time for her, but also for those from whom she asks for data, saying that “on things that we can gather centrally there will be very little impact. We are still going to have to reach out to individual facilities. For these kinds of questions, we'll just tack it on to the data requests we already put out.”

Response Error(s): None.

Recommendation: Because a number of respondents independently acknowledged the potential for an increased investment in time to complete the ASM, we recommend alerting respondents as soon as possible that new questions will be embedded into the survey. This could be through a pre-notice, included in the contact strategies already in place, or some other means.

Resolution: Subject area experts will review their contact strategies to see if additional information prior to launch is warranted.

Finding #2: Most respondents cannot answer the ASM on their own.

Similarly to other Census Bureau data requests, the ASM often requires respondents to coordinate pulling together the requisite data from several areas of the company. One laid out the steps that he takes in order to complete the ASM, saying:

This is a questionnaire that will definitely require more than one person to answer, based on how our company is set up. HR will need to be involved. [For the question asking for the] number of days closed, I will need to talk to the groups that are overseeing that. [For the question asking about] donations - same thing, I'd need to talk to the groups and their heads of financing. I will have to put someone in charge of gathering these data.

At the same time, survey requests are often not the main focus of respondents’ work. Said one, “Surveys, to be honest, fall to the bottom of the list, like it or not. Nobody has time to do these things. [Responsibility for responding falls within] the tax department, but I have to rely on other people to supply the information. There's not a lot we can complete on our own without input from the business units.” A few respondents pointed out that the larger the company, the more challenging it is to gather the requested data. One noted that “because we are a big operation, often folks like us who complete these things aren't privy to the particulars of these questions,” and that researching the appropriate contact within the company can itself be time consuming and challenging.

Response Error(s): None

Specific Findings and Recommendations

Questionnaire Component: Payroll Question Series

For this testing, the payroll question series for the ASM was reordered. Below are the findings specific to this reorganized set of payroll questions.

Finding #3: The revised order of the payroll questions works for respondents.

Very few respondents had issues with the reordering of the payroll questions. Many were indifferent – it makes no difference to them the order of the questions. One highlighted this sentiment saying “I [have] never really thought of the order [of the questions]…it doesn’t matter what order they come in.” One respondent colorfully noted that the reordering of the question is “really six in one, half dozen in the other. I get the raw data files and [get] pluggin' and chuggin from there” regardless of the order. Another more simply stated that she “think[s] the flow works.”

Some respondents even mentioned that they prefer the updated question flow. One simply pointed out that there is “Not a lot of difference [between 2019 and 2020] - but I like the way 2020 flows.” Another noted that the new flow might be more intuitive, saying that starting by asking the total “is correct… I'm assuming in various sub-questions, production [workers] plus [non-production workers] should add up to various totals.”

Note that one respondent suggested flipping questions B2 and B3, suggesting, “I think [question] number 2…might be a little out of order. Going from head count to cost of headcount to production worker quarterly payroll - I would put quarterly [questions] near annual and then the cost [question last].” This respondent did not indicate any confusion with the question or issue obtaining the data.

Response Error(s): None

Finding #4: Respondents classify production workers differently.

Although this is outside of the research questions for this project, many respondents talked about the ambiguity in asking about production vs. non-production employees. On the production ASM instrument, this definition is clearly laid out for respondents; however, it was not specified on the testing instrument. This gives the opportunity to see how respondents themselves think about production/non-production employees.

Working through this question series, one respondent parsed the word production and all of the different ways this could be conceptualized:

When you mean production, we have to be specific. Do you mean anyone that works in the factory? Or people working as direct labor? We have direct and indirect labor and supervisory staff, too. They can be salaried, but they support production. For us, we see them as production. Regardless, if it is the director of the plant, or someone on-the-line, we have the information. But we need to know the level - direct labor, indirect labor, of the plant itself. For us it is all production, just different layers. From a company perspective, anyone who works in a plant is production labor. But, if I’m looking from the perspective of a plant, I'm not sure they would include everyone. Looking at it now, I would include anyone at a plant as production and anyone not at a plant is nonproductive.

Other respondents talked about the ambiguity around production and non-production workers. Asked one, “by production workers....do you mean laborers? And then other workers would be analysts, managers?” noting that in their records, they don’t track production vs. non-production, but rather “it's [tracked] more like titles.” This respondent noted that she can break it out, “but it takes some manipulation.”

Still others outlined their company’s “rule of thumb” for classifying production and non-production workers. Some are based on location, like one respondent who said “We assign all hourly workers at a location to production, and all of the salary go to non-production.” Another echoed this sentiment, noting “Looking at it now, I would include anyone at a plant as production and anyone not at a plant is nonproductive.” Other companies are tracking by job title or cost center, like one respondent who said he would start with “a filter- first by date then what cost centers are included in production. There's a number for each department” that she would use to determine production or non-production.

Some respondents noted that there may be an issue in conceptualizing ‘employee.’ One respondent noted that differing categories of employment are tracked differently, asking “Do you include temporary replacements, trainees, or others who are in production but not fully employed? For some plants, material seasonal workers are employed – do they count? We have programs with universities for training, do I include them? Do I disclose somewhere that they are not full-time workers?”

Response Error(s): Respondents may inappropriately include or exclude production workers based on their own definition.

Recommendation: Retain the ASM statement on how to define and parse production workers and non-production workers.

Resolution: No changes.

Finding #5: Most respondents can provide quarterly production numbers.

Most respondents are able to provide quarterly production workers numbers – both the number employed and the payroll and benefits. Many respondents noted that their company is already tracking this information. Said one respondent, “Quarterly payroll figures… we do have this available. We have [a monthly report] by…hourly and salary, and [additional information] beyond that.” Of those that said they do not currently track it, there was an indication that it would not be difficult to begin tracking. Said one respondent, “[I would] have to do a little bit of massage the data to get quarterly by production; [that’s] not difficult - on the first [reporting, I] just make the query that makes the report for you every quarter.”

Note that even in cases where the data are readily available, some respondents pointed to the increased burden of getting quarterly production worker figures. Said one, “Overall, it's more a question of the time to find the data… just the time to get it together.” In this case, it is not the questions that are challenging, but rather the data handling time. Some noted that they need to reach out to other administrative offices within their company to get the data. One noted, “I would have to go to the payroll accountant and get the data for [quarterly production workers]. [I have to] reach out to another part of the company.” Others said that payroll is tracked and aggregated using disparate systems within one company. This can make it more challenging to aggregate the data, with one saying “Because we have so many different businesses, not everyone is working on the same type of system for their record keeping. [We] may have different software across the company. Not everyone would be going in and using the same central database [for payroll].”

Response Error(s): None

Questionnaire Component: NAPCS Additions Question Series

In response to the COVID-19 global pandemic, some manufacturers retrofit their operations to produce additional products to support public health efforts. In order to capture these changes, six additional products were added to the respondents’ tailored listing of products, based on the North American Product Classification System (NAPCS). In cognitive testing, we paid close attention to whether or not these additions captured newly produced products and whether or not including them on the listing distracted or enhanced response.

Finding #6: Respondents understand the additional products and why they have been added.

First, we asked respondents about the additional six products added to the list. All respondents indicated that they understood the items on the list, and understood why their firm was being asked about the items. Said one, “the [additional] terms are familiar - I know those [products], but we just don't do them.” Note here that the respondent recognizes the products, recognizes that his company does not produce them, and is able to answer the overall question. The additional product codes are also useful for companies that have begun new production. Said one respondent, we “did manufacture hand sanitizer…all [categories] make sense.” Respondents are including products that were manufactured in response to COVID-19 whether or not the product was sold; some additional manufacturing was for internal use.

Note that some firms suggested additional products not listed that may be of interest because of the pandemic, including:

• Toilet paper

• Hand sanitizer pumps

• Air purifiers and air filters

• Specialty glass

Response Error(s): None

Finding #7: Data on new products may be inconsistent.

Knowing that respondents understand the additional NAPCS products is only one piece of the puzzle; we also needed to ascertain whether or not respondents know that new products are being manufactured, and how those data are being tracked. For those producing additional new products, we asked about the new ones specifically; for those not producing new products, we asked more broadly about how they might learn about new products in production.

Some firms indicated that they are already tracking new products manufactured. One said simply that “when I pull the reports, I can see new products and new operations, and [then I can] reach out to that group and find out more about it.” Another noted that the company’s “internal system generates all of our sales and we can get the data [for new products] from there.”

A few said that they only get the aggregated product data – so, to report data by product by establishment, it would take some manipulation to break down the totals. One noted that “in terms of tracking, I would have to ask the specific facility. We don't have a costing system built up. [We have] revenue estimates, [and then I] look at inventory for each facility. Would be hard to get how much did we produce, [and there is] no price on it; it was for internal use.”

Response Error(s): In having to manipulate the available data to report at the level that the ASM requests, respondents may introduce computational error in their response.

Recommendation: While the question stem clearly states that estimates are acceptable, we may want to consider drawing more attention to this caveat, since it seems that the burden of gathering these data could be higher for some manufacturers than others.

Resolution: Subject area staff pointed out that the question on the production instrument already indicates permissibility of estimates; no change.

Questionnaire Component: Donated Products Question Series

Respondents were shown two questions related to donated products. The first is a dichotomous question (yes/no) about donating, and the second was a follow-up number capture question on the value of any donated products. What follows are the major findings for these two questions.

Finding #8: Respondents use disparate definitions of the words ‘donate’ and ‘product.’

We asked respondents about what we mean when we used the word ‘donate’ in the question stem. The result was a myriad of ways to operationalize the concept of ‘donation,’ and differing ways of classifying ‘products’ that are donated.

Some respondents focused on distributing products without payment, regardless of the organization receiving. One succinctly said that the question is asking “did we give anything away without any payment received for it.” Another said to donate is to “give products to nonprofits or [other] organization free of charge.”

A few stumbled on the ‘production’ aspect – as in, only including products that were manufactured by the company – but ultimately, most of our respondents limited their responses to tangible, manufactured goods. One said that the question “means materials we are making that we are giving away.” Another limited her response to “the finished goods we have - if we donated finished goods.” One noted that the wording of the question limits it to tangible goods, saying that he “would interpret this as [asking about] stuff that we make that we donate to other places for charitable donations. It doesn't specify assets, so you think tangible items that we produce.”

One question that did come up more than once was whether products that a company purchases and then distributes without payment should be considered donated products. One respondent mentioned that donations from his company would include “anything we sent to local food bank or charity;” however, upon further probing, he admitted that if the company “paid for it and gave it away, but we didn't manufacture it,” then he would answer ‘no’ to donations because “if we hadn't manufactured it …[it is] basically a cash donation, so not a product.” This respondent was able to use the context clues of the question (donated any products) to clarify what the question is asking. Similarly, another respondent mentioned that she would have to “ask our tax team if we donated,” but that “since we don't [manufacture these goods for donation], we would have to buy and then donate,” so she would answer ‘no.’

Finally, some respondents specifically pointed to COVID-19 in responding to this question⁵. One noted that the question was unclear for him because “we donate things all the time so I don't know if this is looking specifically for COVID-related donations.” Said another, who answered ‘no’ to the question of having donated products, “we gave some items…we had some plastic bags made into flood bag kits [to respond to hurricane-related flooding in the area]. But we haven't done anything in response to the pandemic.” Note here that the response should have been a ‘yes’ – he indicated that they donated flood bag kits produced by their plant – but because the donation was not COVID-specific, he selected no. Another noted that she would be unable to tell if a donation was COVID-related or otherwise, saying “we donate [manufactured products] for things, but nothing [specifically] for COVID-19,” noting that the donation information she has received to date “doesn't look any differently than it has previously,” before the start of the pandemic.

Response Error(s): In the absence of a definition, respondents are assuming what “donate” and “products” means in the question, and some are limiting their responses only to donations made in support of COVID-19 relief efforts. This will result in a potential undercount of donated manufactured products.

Recommendation: To ameliorate miscommunication on this question, we recommend specifying to respondents that this question asks about any manufactured donations, not just COVID related. Potential question wording could be: “Did this establishment donate any products for any reason in 2020?”

Resolution: Subject area staff will review the question wording and make changes as they deem appropriate.

Finding #9: Data on donations may be inconsistent.

Most respondents are keeping track of donations, either systematically within their department, or systematically from some other department. One respondent described the meticulous way that her company keeps records on donations, saying:

We do track these data - not as an official number, and not published with our annual reports. We do have some information we gather for social responsibility reporting. [Donations are] part of the budget; all of the books have a budget for donations. We do products and monetary contributions, and we track them individually.

Another echoed this sentiment, saying that the question is “a reasonable request,” that the data “would not be in our sales but in our expenses,” and that he “could give an exact value” of donations from his company. Still another reported that it is “relatively easy to get” the data on donations, including “an exact amount” of the value of donations.

However, other respondents noted that data on donations are not being tracked as closely within their company. These responses usually noted that to get donation data, the respondent may have to reach out to different groups within the company or to many different locations to aggregate donation data. For example, one mentioned that to get donation information, he has to “ask the staff to list the donations, and we go through the accounts where donations get saved, and we do review them. But, throughout the year I don't always know what they are doing [regarding donations]. Every plant does its own thing. Corporate does some stuff. We only track donations that are over $500 and I look at it at the end of the year.” This quote illustrates that while there is some tracking happening, it is localized, and it is limited.

Finally, some respondents mentioned that data on donations is either not available or it is unclear if these data are available. One said that “cash donations are easy” to report out because these donations are “coded in the ledger”; however, this same respondent admitted that he didn’t know “if there is a spot for donated products.” A few generally mentioned that they suspected that someone – most cited is ‘the tax department’ – might be tracking these data, with one saying “I would probably talk to someone in [the] tax [department] about [donations data]; I would assume [that] they are tracking for tax purposes.” In a worst case scenario, one respondent noted that not only did he not know if the data were being tracked, he was not confident on who would know about these data, saying that his company is “not centrally tracking this” and that he is “not even sure who I would ask; I don't know.”

Response Error(s): Because of the inconsistent ways that donations are being tracked, there is a chance that respondents will guess, will leave the information blank, or will otherwise not respond to the questions as expected.

Recommendation: Since these data are less concrete for some establishments compared to others, we recommend explicitly stating that respondents can provide their best estimate for these questions. That way, we can lower the risk of item non-response, while also recognizing that the data are a best estimate.

Resolution: Subject area experts will rely on the General Reporting Guidelines in the ASM.

Questionnaire Component: Days Closed Question Series

Finally, respondents were shown a question asking for the number of days an establishment was closed because of the coronavirus pandemic. These data are requested by quarter.

Finding #10: Respondents use disparate definitions of the word ‘closed.’

First, we asked respondents about what we mean when we say ‘closed’ in this question. It turns out it is a little more ambiguous than we anticipated.

Two respondents said that during mandatory closing, they were able to complete scheduled repairs or other maintenance, and wondered if this counted as closed or not, with one saying that “Yes, we definitely have closed [locations] and we have that information. It is a bit tricky to answer because the plants have plant maintenance so they used the shutdown to do maintenance. So, [the closing] was planned for September but they did it in May because they were closed. Was it closed because of COVID? Yes. Did we necessarily lose revenue? Yes and no. But we do know how many plants for how many days were shut down.” Note here that this respondent is vacillating on whether or not closing for production, but using that closure for planned maintenance, ‘counts’ as closed.

A few respondents mentioned that parts of the business were closed – like the retail or wholesale arm of a manufacturing plant. Said one, this question “is hard because of COVID” since “from late March through early June, we closed retail sales but the e-Commerce stayed open, and manufacturing stayed open.” This respondent went on to point out that “there was a big layoff” in combination with rolling closures per the local ordinances. Another summed it up this complication stating that his company has “closed pieces of the facility” including “clos[ing] the retail office to protect the staff,” and that she does not “think the closed days were really tracked.”

Beyond the issues of planned closures and partial closures, though, there is the larger question of the location of work being an indicator (or not) of closure. This becomes particularly problematic in manufacturing establishments that also house administrative staff or other non-production workers. Considering work as location based, we get the paradoxical response from one respondent: “I don’t think we closed at all…the buildings all closed, but everyone is working remotely.” In this statement, the respondent tells us that while the buildings were closed, the ‘work’ continued remotely, decoupling work from location. Similarly, another respondent noted that the question does not specify if closure is for production workers or non-production workers, saying “I’m still working from home but some people came into the office – if anyone is in the building, is it ‘closed’?” Again, the respondent is thinking about the connection between work and place, asking if the building is closed or if the work has stopped.

Finally, the question of closure due to relationship to COVID was an issue for some respondents. In some cases, COVID has negatively impacted productivity not just because of local closure regulations, but also because of the impact of COVID on the manufacturing landscape writ large. One noted that “when the ‘COVID stuff’ started, [my company] was furloughing [workers] individually, ask[ing] for volunteers first,” but when that approach “didn’t solve the issue,” the company “started furloughing groups of people for one or two weeks at a time on a rotating basis.” This has since progressed to “the stage where we are furloughing the entire plant a week or two at a time” depending on demand. Here, it is not the local regulations about COVID keeping the plant closed per se, but rather the impact of the pandemic on the larger economic ecosystem that has closed the plant. Another echoed this sentiment, saying that “business is bad” and that “about 30 percent of [production machinery] are not [producing] right now” causing the plant to be “down a week in July, August, and September,” and pushing toward “going to every other week [closures] in October.” Here again, the closure is a result of the impact of COVID-19, but because of falling demand, not because of local closure ordinances. This entire issue was summed succinctly by one respondent who said simply: “we were shut down by both the state mandates and because of a slowdown in production [based on demand]; [we] couldn’t get [products] shipped out, either!”

Response Error(s): With no definition to guide response, respondents are ‘filling in’ their own understanding of what ‘closed’ means to their company. This is leading some to restrict closure information to just mandated closures, while others are including closures due to slowed demand. Some are including – and some are not – when a building is closed but work continues remotely. This will result unreliable estimates of closures because it will be dependent upon disparate understanding of the question and inconsistent reporting.

Recommendation: Because of this ambiguity, we recommend providing guidance for respondents. This may be in the form of a definition, such as ‘Closed means any 24 hour period wherein no productive work happened at the site of the manufacturing plant.’ It could also take the form of rewriting the question for more specificity, such as: How many days per quarter has your establishment been unproductive or had substantial reduction in productivity as a direct or indirect result of the coronavirus global pandemic? The first step is to identify how specific or non-specific the intention of the question should be, and then to work on the language that will evoke a response to that level of specificity.

Resolution: Subject area experts will review the question wording and make changes as they deem appropriate.

Finding #11: Data on closures may be inconsistent.

We asked respondents about the availability of data regarding closures. Like donations, there are some companies that are carefully and formally tracking this information. One mentioned a specific interest in keeping these data, saying “We have information [on closures] - I was involved in the COVID-19 stuff from the start, back in March and all of the initial discussions. We have been spending time on tracking this information; all of our plants shut down at the same time.” Another mentioned that her company “know[s] exact dates so we could put [the number of closed days] into quarters.”

However, other respondents noted that data on closures is being tracked less formally and rigorously. One respondent noted that his company was “down a week in July, August, and September,” and when asked how he knew this information, he replied “I work here…”. This response suggests that for some companies – particularly small companies, and those deemed essential – are aware of closed days but are not specifically tracking that information.

Finally, a few companies are not tracking this information at the company level; respondents mentioned that they would need to reach out to each establishment in order to get their information, and hope that they are tracking it at the establishment level. Said one, “I would have to look into detail - some establishments closed at different times. I'd have to average across all of our locations.” Another said explicitly that “there are people who are not working full weeks or have been furloughed” but that “this is not [information] that we are tracking at a company level.” Here, again, the respondent conflates closure and reduced production, and admits that these data are not systematically tracked.

Response Error(s): Since this phenomenon – closures due to a global pandemic – is unprecedented, some companies have not yet built the data infrastructure to track this information. This may lead to increased item non-response for this question, which will not be missing completely at random (MCAR), and could introduce bias into the resultant data.

Recommendation: We recommend allowing respondents again to estimate their responses, and explicitly stating that estimates are acceptable. Again, the goal would be to get any data – even estimated data – instead of missing data for this question.

Resolution: Subject area experts will rely on the General Reporting Guidelines in the ASM.

About the Response Improvement Research Staff (RIRS)

The Response Improvement Research Staff (RIRS) in the Office of Economic Planning and Innovation (OEPI) assists economic survey program areas and other governmental agencies with research associated with the behavioral aspects of survey response and data collection. The mission of RIRS is to improve data quality in surveys while reducing survey nonresponse and respondent burden. This mission is achieved by:

• Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with post-collection evaluation methods, to assess the effectiveness and efficiency of the data collection instruments and associated materials.

• Conducting early stage scoping interviews to assist with the development of survey content (concepts, specifications, question wording and instructions, etc.) by getting early feedback on it from respondents.

• Assisting program areas with the development and use of nonresponse reduction methods and contact strategies.

• Conducting empirical research to help better understand behavioral aspects of survey response, with the aim of identifying areas for further improvement as well as evaluating the effectiveness of qualitative research.

For more information on how RIRS can assist your economic survey program area or agency, please visit the RIRS intranet site or contact the staff chief, Diane Willimack.

Appendix A: Cognitive Interviewing Methodology

Cognitive research is used in survey methodology “(a) to understand the thought processes used to answer survey items, and (b) to use this knowledge to find better ways of constructing, formulating, and asking survey questions” (Forsyth and Lessler, 1991).⁶ Cognitive interviews traditionally focus on the four steps of Tourangeau’s (1984) cognitive response model: comprehension, retrieval, judgment, and communication/reporting.⁷ Comprehension refers to the respondent’s interpretation and understanding of the question’s language, structure, and grammar. In order to answer the question, a respondent must understand what information is being requested on the survey. Retrieval is the step where relevant information is obtained, either from records or from memory. The next step, judgment, describes the respondent’s evaluation of the completeness or relevance of the data obtained. It is here that estimates are made based on partial or incomplete data. The last step, communication or reporting, deals with mapping the response to the answer space provided and possibly altering the answer.

While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting presents additional factors that must be considered. First, instead of or in addition to a reliance on memory, establishment surveys rely heavily on records and the information contained within them. Second, organizations tend to have distributed knowledge. Some people are experts in one type of information, while others keep information about something else. Third, competing priorities, both for the organization and the individual(s) completing the questionnaire, mean that the survey sometimes does not receive the amount of attention that researchers and data collectors would like. Finally, organizations regularly authorize only a few individuals to release data. If the data provider is not authorized to release the data, an additional step must be added to the response process. Tourangeau’s model was expanded by Sudman et al (2000) to account for these factors. ⁸

Once the protocol is established, researchers will recruit a set of approximately five undergraduate and graduate students to participate in cognitive interviewing. Blair and Conrad (2011, 637) state that “one of the few explicit discussions [within cognitive interviewing methodology] of sample size largely supports small samples.” Willis (2005, 226-28) notes that statistical estimation is not the purpose of cognitive interviewing: “…we do not evaluate the [item being tested] simply by counting the number of interviews in which a problem occurs.” Lastly, Blair and Conrad (2011) state that careful selection of interview subjects counteracts issues that could occur with small sample sizes. The interview subjects have been carefully selected from a larger sample and were selected based on the variety of their skills and experience with geographic information systems and demographic characteristics. The purpose of cognitive interviewing is to understand the prevalence of an issue. Blair et al. (2006) found that five interviews will help the project team to identify the major issues with the platform; while adding more interviews may increase the likelihood of finding additional minor issues, it does not necessarily lead to the discovery of additional major issues. Likewise, because we are conducting usability testing in the form of eye tracking and hot spot analysis, the results of the cognitive interviewing will augment findings from these other forms of testing; the combination of methods will produce the most robust results, identifying issues in the platform (Houle 2009). The interviews will last 60 to 90 minutes, will be audio recorded and have screen capture, and be conducted by methodologists trained in cognitive interviewing practices.

Appendix B: Cognitive Interviewing Protocol

ASM COVID Protocol

Start of Block: Dashboard

Q1
2020 Annual Survey of Manufactures Testing Protocol

Research Questions:

• Do respondents understand what information they are being asked to provide with these questions?

• Do respondents understand the terminology used in the questions?

• Are respondents able to answer the questions that are being asked?

• Is the requested information available in respondents’ records?

• Does the revised order of the Payroll and Employment questions make sense to respondents?

• Does the order of the questions about donations (Sales first then Special Inquiry) make sense to respondents?

Expected Length of Interview: 30 minutes

Company Information:
Company Name: ${e://Field/NAME1}
State: ${e://Field/STATE}
Single Unit or Multi Unit?: ${e://Field/SURVUNIT_TYP}

Respondent Information:
Name:  ${e://Field/CONTACT_NAME}
Phone Number: ${e://Field/CONTACT_PHONE}${e://Field/CONPHONE}
Email: ${m://Email1}
Appointment: ${e://Field/APPOINTMENT}

Q19 Who is conducting this interview?

▼ Amy (1) ... Temika (7)

Page Break

Q20 20 to 30 minutes before the interview, send the following email:
Email:  ${m://Email1}

Dear ${e://Field/CONTACT_NAME} -

Thank you so much for agreeing to participate in a 30 minute interview with the Census Bureau.  Your appointment is: ${e://Field/APPOINTMENT}.

Before we talk, please use the link below to complete your consent form.  We cannot conduct the interview if your consent form has not been completed:
${e://Field/PRALink}
 
 
// IF YOU HAVE OBSERVERS:// At your appointment time, please use the following information to log in to the conference call:
Phone number:
Conference ID:
 
//IF YOU DO NOT HAVE OBSERVERS://
I will call you at your appointment time at the phone number that we have on file:  ${e://Field/CONTACT_PHONE}${e://Field/CONPHONE} .  If this phone number is out of date or if you have a different preferred number, please respond to this email and let me know.
  During our time together, we will reference a few new survey questions, available at the link below; please wait until we are on the phone together to open this link, I'd like to hear your first impressions:
${e://Field/SurveyLink}
 
Many thanks, ${Q19/ChoiceGroup/SelectedChoices}

Page Break

Q22
Time to call! 
Respondent's Name:  ${e://Field/CONTACT_NAME}
If you have no observers, use the phone number we have on file: ${e://Field/CONTACT_PHONE}${e://Field/CONPHONE}
If you have observers, use the SfB call information that you sent to the respondent.

Hello!  May I speak to ${e://Field/CONTACT_NAME}?
Thanks for taking the time with me today. 
  From time to time, the Census Bureau evaluates questions and forms to make sure they are measuring what it is that we are intending for them to measure.  In this case, I am going to ask you to review several new questions that we are developing to refine our measurements in response to the COVID-19 pandemic.  Your feedback will be combined with other respondents, and then I'll take my recommendations to the team to improve our survey instruments.

In the consent form that I sent to you earlier, it noted that this study is being conducted under the authority of Title 13 of the United States Code. That means that anything you tell me is strictly confidential, and I will not be sharing your name or the name of your company in my findings.

Before we begin, do you have any questions?

Let's get started with some background information about your business.

End of Block: Dashboard

Start of Block: Background

Q2 First, tell me about your company: what do you do or make?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q3 And what is your role or title?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q4 What is your role in completing government surveys?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Background

Start of Block: ASM Questions

Q5
How much experience do you have completing the ASM?
What has your experience completing the ASM been like?
Do you work with anyone else in your company to get the data for the ASM? How?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: ASM Questions

Start of Block: Employment and Payroll

Q6
Thank you for that background information.  Now I would like to show you some new and revised questions that we are testing for the Annual Survey of Manufactures.


In the email that I sent to you before this interview, I included a link.  Could you please click on it and let me know when it is loaded on your screen?

Q7 Can you tell me how you would go about providing these payroll data? I'm especially interested in providing quarterly payroll figures for production workers - do you have this information readily available?  Would you have to do any manipulation/calculation with your data in order to get these values?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q8
The question asks for overall company totals and then asks about breakdowns by production workers and all other workers.  What do you think about the flow of this question?


Does the order make sense?
Would your experience change if the question asked about details first followed by totals?

________________________________________________________________

Page Break

Q9
Go ahead and open both the 2019 and 2020 ASM questions.  They should each open in their own window once you click on it. 
 
[INTERVIEWER:  files here - 2019 ASM and 2020 ASM]
 
How accessible are these data for your company?
How much easier or more difficult, if at all, is the new version compared to the previous version?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Employment and Payroll

Start of Block: Item 22

Q10
This question currently appears on the ASM form.  Each manufacturing establishment receives a custom list of the products that we believe they produce.  For the 2020 survey, will be adding 6 products to all forms, listed under the heading as "Additional products manufactured (not typically produced by this establishment)".

Take a moment now to read over the question and orient yourself to it, and let me know when you are ready to talk about it.

Q11 Do any of your manufacturing establishments produce the products listed under the "additional products manufactured (not typically produced by this establishment) heading?

• Yes (1)

• No (2)

• Unsure/Don't know (3)

Display This Question:

If Do any of your manufacturing establishments produce the products listed under the "additional pro... = Yes

Q12 Did you produce these product(s) prior to 2019?

If no, why did you begin producing these product(s)?

Are you tracking the detailed sales/receipts/revenue for these new products?

What would you have to do in order to gather the data for these new products?

Are there any terms that you aren’t familiar with? 
Are there any other new products that any of your establishments have begun producing in 2020 because of the pandemic that aren’t listed? 

Are there any products that you are planning to begin producing because of the pandemic that are different than what you have historically manufactured?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

Display This Question:

If Do any of your manufacturing establishments produce the products listed under the "additional pro... = No

Or Do any of your manufacturing establishments produce the products listed under the "additional pro... = Unsure/Don't know

Q13 Are there any terms that you aren’t familiar with? 

Are there any other new products that any of your establishments have begun producing in 2020 because of the pandemic that aren’t listed? 

Are there any products that you are planning to begin producing because of the pandemic that are different than what you have historically manufactured?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Item 22

Start of Block: Donated Items

Q14
Donated items:


What does "donate any products" mean here?  What does include and what could it exclude?

How would you go about answering this question?

________________________________________________________________

________________________________________________________________

________________________________________________________________

Q15
IF FIRM HAS DONATED:

• Can you tell me a little about the products that were donated?

• How do you track donations in your records?

• Is this easily accessible to you?

• Is the value of the donations already included within any total sales, shipments, receipts, or revenue values that you have on your books?

• If not, how would you go about including the value of the donations made in 2020 to your total figure?

• Would you have to work with others in your company to identify the value of donations?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Donated Items

Start of Block: Closed Days

Q16
CLOSED DAYS

• Can you tell me a little about how you would go about answering this question for any manufacturing establishments that you are responsible for reporting on for the ASM?  

• Did any of the establishments in your company experience closures due to the pandemic?

  • If yes, did your company track this information?

• Although we haven’t finished 2020, how would you go about answering this question for the first three quarters of the year?

  • Do you have access to this information?

  • If you do not have access to this information, would others in your company have access to this information?

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

________________________________________________________________

End of Block: Closed Days

Start of Block: Wrap up

Q17
We are almost done; I just have a few wrap up questions.

Do you think the new questions are easy to answer, difficult to answer, or somewhere in between?  Why?  which?

Is there any other feedback you'd like to give me now on anything we've talked about during our time together?

________________________________________________________________

________________________________________________________________

Q18 Thank you so much for your time and attention.

End of Block: Wrap up

Appendix C: Test Instrument

ASM Test Instrument

Start of Block: Default Question Block

Q1 ITEM 7: EMPLOYMENT, PAYROLL, AND FRINGE BENEFITS

Include:

• Full- and part-time employees working at this establishment whose payroll was reported on Internal Revenue Service Form 941, Employer's Quarterly Federal Tax Return, and filed under the Employer Identification Number (EIN).

• All persons on paid sick leave, and paid vacation during the year at this establishment.

Exclude:

• Full- or part-time leased employees whose payroll was filed under an employee leasing company's EIN.

• Temporary staffing obtained from a staffing service.

• Purchased professional and technical services.

• Subcontractors and their employees.

• Fishermen, agricultural employees, members of the Armed Forces, and pensioners carried on your active rolls.

Q4 Click to write the question text
A.  TOTAL EMPLOYMENT AND PAYROLL
 
For all employees at this establishment, what was the:

1. Total number of employees for pay period including March 12? : _______ (1)

2. Total annual payroll (before deductions)? : _______ (2)

3. Total first quarter payroll (January - March)? : _______ (3)

Total : ________

Q5 B. PRODUCTION WORKER EMPLOYMENT AND PAYROLL

1. Production Worker Employment
What was the number of production workers at this establishment (direct labor including first-line supervisors) for the pay period including:

a. March 12 (Q1)? : _______ (1)

b. June 12 (Q2)? : _______ (2)

c. September 12 (Q3)? : _______ (3)

d. December 12 (Q4)? : _______ (4)

Total : ________

Q6 2. Production Worker Annual Payroll (before deductions)
For production workers at this establishment, what was the

Exclude: Employer-paid annual cost for fringe benefits.

Annual Payroll (before deductions)? : _______ (1)

Total : ________

Q7 3. Production Worker Quarterly Payroll
For production workers at this establishment, what was the

Exclude: Employer-paid annual cost for fringe benefits.

a. First quarter payroll (January - March)? : _______ (1)

b. Second quarter payroll (April - June)? : _______ (2)

c. Third quarter payroll (July - September)? : _______ (3)

d. Fourth quarter payroll (October - December)? : _______ (4)

Total : ________

Q8 C. NON-PRODUCTION EMPLOYMENT AND PAYROLL
For non-production employees at this establishment, what was the

1. Number of employees for the pay period including March 12? : _______ (1)

2. Annual Payroll (before deductions)? : _______ (2)

3. First quarter payroll (January - March)? : _______ (3)

Total : ________

Page Break

Q17 Click here for the 2019 ASM Production Worker questions

Click here for the 2020 ASM Production Worker questions

End of Block: Default Question Block

Start of Block: Default Question Block

Q3 ITEM 22: DETAIL OF SALES, SHIPMENTS, RECEIPTS, OR REVENUE

Of the total Sales, Shipments, Receipts, or Revenue, what was the value of each product or service?

General – Please do not combine product lines. If the information is not directly available from your  records, reasonable estimates are acceptable.

The manufactured products and services listed below are generally made in your industry. If you make products or have revenue from sources not listed, click the “Add Product Not Listed” button and search for an existing product, or use the section for “Add product(s) not listed above.”

Manufacturing of Products – Report the value of the products shipped and services performed at the net selling value, free on board (FOB) plant to the customer, after discounts and allowances.

Include:

• Products made elsewhere by others from materials supplied by this establishment. Report these products on the specific lines as if they were made in this establishment.

• Products transferred to other establishments within your company. These products should be assigned the full economic value (market value); i.e., include all direct costs of production and a reasonable proportion of all other costs and profits.

Exclude:

• Wholesale products (previously Resales), which include products that are bought from other establishments or transferred from other establishments of your company and then sold without further manufacture, processing, or assembly by this establishment. Report Wholesale products in any relevant prelisted product code, click the "Add Product Not Listed" button and search for an existing Wholesale product, or use the section for "Add product(s) not listed above."

• Products made from materials owned by others (i.e., the customer). Report your commission or contract receipts in the appropriate Contract Manufacturing product line(s).

• Freight charged

• Excise taxes

Industry specific listing of North American Product Classification System (NAPCS) inserted here.

Additional products manufactured (not typically produced by this establishment)

• 10. Manufacturing of nonelectric breathing devices (including N95 and other respirators), incubators, inhalators, and resuscitators, and other surgical and medical apparatus and instruments, excluding anesthetic apparatus and parts 2017900000 (10)

• 11. Manufacturing of electromedical equipment (including diagnostic, therapeutic, patient monitoring equipment, and ventilators), excluding ionizing radiation equipment 2018000000 (11)

• 12. Manufacturing of personal safety equipment and clothing, industrial and nonindustrial, including respiratory protection, face shields, masks, and protective clothing, excluding footwear, gloves, and surgical and medical respirators 2050375000 (12)

• 13. Manufacturing of surgical appliances and supplies, including surgical gloves, bandages, gauze, cotton (sterile and non-sterile), and other surgical dressings, excluding orthopedic and prosthetic appliances 2045875000 (13)

• 14. Manufacturing of bath, facial, and hand soaps, including hand sanitizers 2010475000 (14)

• 15. Manufacturing of other household specialty cleaning and sanitation products, including disinfectants 2007875000 (15)

• 12. Add additional product or service (18) ________________________________________________

End of Block: Default Question Block

Start of Block: Block 2

Q9 ITEM 29A: DONATED PRODUCTS
Did this establishment donate any products during 2020?

• Yes (1)

• No (2)

Q10 ITEM 29B: VALUE OF DONATED PRODUCTS
What was the value of the donated products?

________________________________________________________________

Page Break

Q11 ITEM 30: NUMBER OF DAYS ESTABLISHMENT CLOSED
How many days during 2020 was your establishment closed by quarter due to the coronavirus pandemic?

A.  First quarter (January - March)? : _______ (1)

B. Second quarter (April - June)? : _______ (2)

C. Third quarter (July - September)? : _______ (3)

D. Fourth quarter (October - December)? : _______ (4)

Total : ________

End of Block: Block 2

1 Campanelli, P. (2007). Methods for Testing Survey Questions [Short Course]. Joint Program in Survey Methodology, Bethesda, Maryland.

2 Willis, Gordon & Lessler, Judith. (1999). Question Appraisal System QAS-99. Rockville, Maryland: Research Triangle Institute.

3 Ericsson, K. A., & Simon, H. A. (1984). Protocol analysis: Verbal reports as data. The MIT Press.

4 North American Industry Classification (NAICS) codes – specific industry coded to the ninth digit, broader industry coded to the third digit.

5 A caveat: in the test instrument, the questions about donations were immediately preceded by the questions about additional products manufactured in response to the pandemic. It may be that some of this confusion is due to artificial order effects from the test instrument. However, it could also be a natural result of the current milieu in which we are conducting this testing.

6 Forsyth, B.H. and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy.” In Measurement Errors in Surveys, P.P. Biemer, R.M. Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.

7 Tourangeau, R. (1984). “Cognitive Sciences and Survey Methods.” In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M. Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.

8 Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). “Exploratory Research at the U.S. Census Bureau on the Survey Response Process in Large Companies.” Paper prepared for presentation at the Second International Conference on Establishment Surveys, Buffalo, NY.