Round 2 Cognitive Testing Report

Findings and Recommendations from the
Second Round of Cognitive Testing for
the 2015 Management and
Organizational Practices Survey (MOPS)
Prepared for:

Lucia Foster, Center for Economic Studies (CES)
Cathy Buffington, Research and Methodology Directorate (RMD)
Scott Ohlmacher, CES

Prepared by:
Kenneth Pick, Data Collection Methodology and Research Branch (DCMRB)
Micheal Brennan, DCMRB

Economic Directorate
U.S. Census Bureau

October 13, 2015

Table of Contents
Research Objectives ...................................................................................................................................... 4
Research Methodology ................................................................................................................................. 5
Findings and Recommendations ................................................................................................................... 7
General Findings and Recommendations ..................................................................................................... 8
Finding #1: Characteristics of establishments visited ........................................................................... 8
Finding #2: Roles of the respondents within the organization ............................................................. 9
Finding #3: Response Process ............................................................................................................... 9
Finding #4: 5-year time frame............................................................................................................... 9
Finding #5: Confidence in estimating 2017 data ................................................................................ 10
Finding #6: Forecasting ....................................................................................................................... 10
Finding #7: Respondent burden associated with new questions ....................................................... 10
Item–Specific Findings and Recommendations .......................................................................................... 12
Finding #8: Question 17 ...................................................................................................................... 12
Finding #9: Question 24 ...................................................................................................................... 12
Finding #10: Question 25 .................................................................................................................... 13
Finding #11: Question 26 .................................................................................................................... 13
Finding #12: Question 27 .................................................................................................................... 13
Finding #13: Question 28 .................................................................................................................... 14
Finding #14: Question 29 .................................................................................................................... 14
Finding #15: Uncertainty Instructions................................................................................................. 15
Finding #16: Question 30 .................................................................................................................... 16
Finding #17: Question 31 .................................................................................................................... 16
Finding #18: Question 32 .................................................................................................................... 16
Finding #19: Question 34 .................................................................................................................... 17
Finding #20: Question 35 .................................................................................................................... 17
Finding #21: Question 36 .................................................................................................................... 17
Finding #22: Question 37 .................................................................................................................... 18
Finding #23: Question 43 .................................................................................................................... 18
Finding #24: Question 44 .................................................................................................................... 18
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 2 of 26

Finding #25: Question 45 .................................................................................................................... 19
Finding #26: Question 46 .................................................................................................................... 19
About the Data Collection Methodology & Research Branch (DCMRB) ..................................................... 20
Appendix A - Protocol ................................................................................................................................. 21

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 3 of 26

Research Objectives
The Management and Organizational Practices Survey, or MOPS, provides information on the use of
decentralized decision rights, data-driven decision making, and investments in human capital. The data
collected on the MOPS offer a better understanding of the benefits from these practices when
measured in terms of firm productivity and/or firm market value. Understanding the drivers of
productivity growth is essential to understanding the dynamics of the U.S. economy and the
management of firms plays a central role in this.
For the 2015 MOPS, there were a number of questions added to the form to further explore the use of
decentralized decision rights, data-driven decision making, and investments in human capital. Because
of these additional questions, the Census Bureau’s A2 quality standard states that cognitive testing must
be done to assess the efficacy of these questions. With that in mind, Census Bureau survey
methodologist conducted two rounds of cognitive interviews: the first round was exploratory in nature
and the second round was confirmatory. The goals of the cognitive testing were to:
•
•
•

Determine whether respondents interpret the new questions and instructions in a manner that
meets the questionnaire’s intent;
Identify likely respondents and data retrieval strategies for these new questions;
Identify other related reporting issues or concerns related to the 2015 MOPS.

This report summarizes the findings from the second round of confirmatory cognitive interviews.

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 4 of 26

Research Methodology
To address these research objectives, we conducted 14 in-person cognitive interviews during September
and October of 2015 in the San Francisco metropolitan area and Boston metropolitan area. These inperson cognitive interviews took place in respondents’ office locations and generally lasted 45 minutes.
Where possible, interviews took place at the establishment and not the corporate headquarters.
Respondents represented units of varying industries, size, and roles within their organization (although
most respondents were in either accounting-related roles or operational-related roles). Subject matter
experts from the Census Bureau observed all cognitive interviews.
Table 1: Total number of debriefings per metropolitan area
San Francisco MPA
7

Boston MPA
7

Table 2: Total number of cognitive interviews per NAICS code
NAICS Code
311
2

316
1

323
2

334
8

335
1

Table 3: Total number of cognitive interviews by single-unit and multi-unit establishments
Single-Unit
6

Multi-Unit
8

The qualitative methodology used to answer these research objectives was cognitive interviews.
Cognitive interviews are used in survey methodology to “(a) understand the thought processes used to
answer survey items, and (b) to use this knowledge to find better ways of constructing, formulating, and
asking survey questions” (Forsyth and Lessler, 1991). 1 Cognitive interviews traditionally focus on the
four steps of Tourangeau’s (1984) cognitive response model: comprehension, retrieval, judgment, and
communication/reporting. 2 Comprehension refers to the respondent’s interpretation and
understanding of the question’s language, structure, and grammar. In order to answer the question, a
respondent must understand what information is being requested on the survey. Retrieval is the step
where relevant information is obtained, either from records or from memory. The next step, judgment,
1

Forsyth, B.H. and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy.” In Measurement Errors in Surveys, P.P. Biemer, R.M.
Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
2

Tourangeau, R. (1984). “Cognitive Sciences and Survey Methods.” In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M.
Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 5 of 26

describes the respondent’s evaluation of the completeness or relevance of the data obtained. It is here
that estimates are made based on partial or incomplete data. The last step, communication or
reporting, deals with mapping the response to the answer space provided and possibly altering the
answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on
memory, establishment surveys rely heavily on records and the information contained within them.
Second, organizations tend to have distributed knowledge. Some people are experts in one type of
information, while others keep information about something else. Third, competing priorities, both for
the organization and the individual(s) completing the questionnaire, mean that the survey sometimes
does not receive the amount of attention that researchers and data collectors would like. Finally,
organizations regularly authorize only a few individuals to release data. If the data provider is not
authorized to release the data, an additional step must be added to the response process. Tourangeau’s
model was expanded by Sudman et al (2000) to account for these factors. 3

3 Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). “Exploratory Research at the U.S. Census Bureau on the Survey
Response Process in Large Companies.” Paper prepared for presentation at the Second International Conference on Establishment Surveys,
Buffalo, NY.

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 6 of 26

Findings and Recommendations
The findings and recommendations report is divided into two sections: (1) general findings and
recommendations and (2) item-specific findings and recommendations. The protocol used during
cognitive testing is found in Appendix A.

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 7 of 26

General Findings and Recommendations
Finding #1: Characteristics of establishments visited
For the second round of MOPS cognitive testing, we visited six single-unit and eight multi-unit
manufacturing establishments in the San Francisco and Boston metropolitan areas. These
manufacturing establishments were in a variety of industries including high tech robotics, consumer
packaged goods, and leather manufacturing. (For a complete list of companies visited per NAICS code,
please see Table 2 under Research Methodology.) Respondents at these establishments were recruited
from the Annual Survey of Manufactures (ASM) frame.
The single-unit manufacturing establishments were generally small businesses with family members in
numerous positions in upper management or were described as being “family-like.” Similar to the first
round of cognitive testing, a number of these single-unit establishments provided custom product
design tailored to the specifications desired by the customer. In these cases, respondents found it
difficult to generalize their responses since answers were viewed as “job specific.” (A similar finding was
found during the first round of cognitive testing.)
In contrast, the multi-unit manufacturing establishments that we visited were a part of large,
international companies. Unlike the first round of cognitive testing, respondents we visited during the
second round were located at their plants. These respondents stated that although their corporate
offices frequently completed government surveys, they would occasionally be asked to complete a
government survey if it required plant-specific information. Although we met with respondents in both
accountant and operational roles at the plants, the consensus was that the plant controller would be the
most appropriate person to complete the MOPS survey.
It should be noted that one respondent originally recruited in San Francisco was out of scope for the
MOPS cognitive testing and are instead a non-manufacturing start-up still in the research and
development phase. When asked why they completed the ASM (from which they were recruited), their
response was “we [completed it] so the Census wouldn’t threaten us and leave us alone.”
Recommendation: Based off the feedback received from the second round of cognitive testing,
we recommend sending the survey invite directly to the plant. It is our opinion that the plant
controller will most likely complete the survey, and/or work with the plant manager to complete
the survey. Nonresponse follow-up should be directed to the corporate offices if the plant fails
to respond.
We also recommend deleting Survey ID 2021304682 from the ASM and MOPS frame since they
are not a manufacturing establishment.
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 8 of 26

Finding #2: Roles of the respondents within the organization
Respondents were in a variety of roles within their establishment, both at single-unit and multiunit establishments. This included financial analysts, plant controllers, and VPs of operations
and finance. The consensus among respondents was that someone in accounting at the plant
would be the most appropriate person to complete the MOPS. Even if they did not have the
necessary information to complete a section, they would know the appropriate person in
operations to get the information needed.
Recommendation: See our recommendation for Finding #1.
Resolution:

Finding #3: Response Process
Respondents at both single-units and multi-units establishments, including those at the plant,
felt a person in the accounting role would be the most appropriate person to complete the
MOPS. It was repeated that if a respondent in the accounting role did not know how to answer a
question, they would go to someone in operations to get the information.
Recommendation: See our recommendation for Finding #1.
Resolution:

Finding #4: 5-year time frame
For both single-unit and multi-unit establishments, there were a few respondents that were not at their
establishment in 2010. For these respondents, most said they would either ask someone that was there
in 2010 for the answer or make an educated guess.
One respondent stated that he would skip some 2010 questions since he was unfamiliar with the
organization at that time.
Recommendation: We recommend providing guidance on the form tailored to respondents
who were not employed in 2010 at the establishment on how to answer these historical
opinion-based questions (i.e., estimates are acceptable, etc.). Having explicit instructions may
decrease item nonresponse.
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 9 of 26

Finding #5: Confidence in estimating 2017 data
As previously reported during the first round of cognitive testing, respondents at both single-unit and
multi-unit establishments were not confident in their 2017 forecasts. They frequently reported that
some of their answers may be best guesses or somewhat subjective. A few asked for an example as to
interpret the task and report the question.
Recommendation: As discussed during the first round of cognitive testing, the sponsor would
like to keep the 2017 forecast questions on the questionnaire. As such, we recommend
providing an example to the respondents on how to complete these questions to allay any
comprehension concerns (see our recommendation for Finding #15). However, doing this will
not change the subjective nature of the responses received for these questions.
Resolution:

Finding #6: Forecasting
For single-unit establishments, very little, if any, forecasting was done. This was either due to the
custom nature of their work, the volatility of the industry, or because much of the business is led by “gut
feelings” or “knowing the market.”
For multi-unit establishments, forecasting was done, but not to the granularity asked for by some of the
questions on the MOPS form.
Recommendation: See our recommendation for Finding #15.
Resolution:

Finding #7: Respondent burden associated with new questions
For the new questions tested, respondents at both single-unit and multi-unit establishments reported
that it would take anywhere from 30 minutes to an entire day to complete the survey (the larger
estimate being if multiple areas within an organization needed to be contacted).
In general, respondents perceived forecasting and estimation questions to be the largest burden since
not only would they have to collect records for 2015, which could be time consuming, but they would
then have to come up with reasonable estimates for future values. However, if respondents provided
educated guesses versus doing reasonable calculations, this would reduce their overall burden in
completing the MOPS.

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 10 of 26

Recommendation: We recommended either increasing the burden statement, reducing the
number of questions, or rewording questions so that they capture data more readily available in
business records.
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 11 of 26

Item–Specific Findings and Recommendations
Finding #8: Question 17
For the second round of cognitive testing, most respondents comprehended the intent of Question 17
and were able to answer it. However, a few respondents interpreted this question as asking if the
headquarters had moved between 2010 and 2015. In addition, a few other respondents, both at singleunit and multi-unit establishments, did not understand the use of “headquarters.”
Recommendation: We believe that the confusion not only lies in the wording of the current
question, but also the fact that it is asking about multiple constructs. We recommend breaking
out the question into two separate constructs. First, ask about 2010: In 2010, was this
establishment physically located at the same location as the headquarters for the company?
Yes/No. Repeat for 2015.
Alternatively, consider deleting the 2010 request entirely from the question since this data is
available for some respondents from the previous data collection. After speaking to the sponsor,
it is unclear as to how, or if, the 2010 will be used.
Resolution:

Finding #9: Question 24
For both single-unit and multi-unit establishments, some respondents had difficulty with the general
vagueness of the phrase “availability of data.” Since some respondents that we met with were in
accounting roles, they immediately thought of accounting and financial data when answering this
question (unless they were primed by an introductory manufacturing discussion).
It was also interesting to note that most respondents struggled with believing they could ever have all
the data they will need and marked “a great deal…” versus “all the data we need…”
Recommendation: We recommend explicitly clarifying the intent of the question in the question
itself. For example, if the survey’s authors are asking about operational data, then state
“operational data” in the question. If the survey’s authors are asking about decision making “in
operations”, then again state it in the question. As the question is currently worded, and based
off the cognitive interviews, the questions will currently collect vague answers skewed towards
financial data (as most respondents are accountants).
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 12 of 26

Finding #10: Question 25
We received the same sort of feedback for Question 25 as we did for Question 24 (see Finding #9).
Recommendation: See recommendation for Finding #9.
Resolution:

Finding #11: Question 26
Similar to the first round of cognitive testing, it was observed that some respondents perceived
this question, and the use of the term “data,” to be vague. However, most respondents were
able to answer it. One respondent did feel that the inclusion of “managers at other
establishments” in the second response should be a separate response. He stated that, “We
compete with other plants in our company, and don’t take direction from them,” The
respondent was at a plant that was a part of a multi-unit, international company.
Recommendation: If the type of data is clarified in questions 24 and 25, also clarify it here in
Question 26.
Also, consider breaking out headquarters into a separate response option.
Resolution:

Finding #12: Question 27
In general, this question preformed well. Respondents had little difficulty answering for the response
options and even a few respondents referred back to the definition in Question 2, as indicated in
“production performance indicators…” A couple of respondents were confused by the vagueness of the
phrase, “data from outside the firm”, but then used the examples listed in parenthesis to answer the
question.
A couple of respondents wanted to answer somewhere between monthly and yearly, and suggested
having a quarterly option. Similar to this feedback, a few respondents marked multiple boxes per row
(i.e., weekly, monthly, and yearly).
Finally, one respondent read the construct in the question stem as “collected” versus “used.”
Recommendation: We recommend clarifying the “Mark all that apply” instructions if only one
box should be marked per horizontal response option. We also recommend including a
quarterly option separating monthly and yearly.
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 13 of 26

In addition, we also recommend adding a “not applicable” response option if it is important to
differentiate between “never” and “not applicable” in the data analysis.
Finally, consider emphasizing the word “used” in the question stem to emphasize the term. This
can be done either by bolding the word or using capital letters.
Resolution:

Finding #13: Question 28
A few respondents had issues with “design of new products or services” since their product may be one
that doesn’t change much, or is so custom that all products are different from job to job. In addition,
some respondents were unclear what supply chain management was, while others knew that it was
something they looked into daily and were very confident in answering the question. In contrast, most
respondents had a firm grasp of demand forecasting, but some felt it may not apply to their custom
businesses and were tempted to leave it blank or mark never.
Similar to Finding #12, a couple of respondents wanted to answer somewhere between monthly and
yearly, and suggested having a quarterly option. Again, similar to Finding #12, a few respondents
marked multiple boxes per row (i.e., weekly, monthly, and yearly).
Recommendation: We recommend clarifying the “Mark all that apply” instructions if only one
box should be marked per horizontal response option. We also recommend including a
quarterly option separating monthly and yearly.
In addition, we also recommend adding a “not applicable” response option if it is important to
differentiate between “never” and “not applicable” in the data analysis.
Finally, it is observed that the question asks, “How often…” when similar questions ask, “How
frequently…” We recommend changing the wording so it is consistent.
Resolution:

Finding #14: Question 29
Similar to the first round of cognitive testing, a few respondents were unclear as to what type of data
this question referred to. However, when pressed, respondents were able to select an answer. Likewise,
to the previous findings, a few respondents marked multiple boxes per year.
Recommendation: We recommend clarifying the “Mark all that apply” instructions if only one
box should be marked per column. We also recommend including a quarterly option separating
monthly and yearly.
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 14 of 26

Resolution:

Finding #15: Uncertainty Instructions
When asked to paraphrase the uncertainty instructions, the majority of respondents understood them.
However, this did not necessarily translate into all of the respondents comprehending how to complete
the forecasting questions later in the Section D.
Recommendation: We recommend replacing the instructions with a neutral example illustrating
how a person would complete this section. For example:
John Doe is Vice President of Finance at ABC Manufacturing Plant. In 2015, ABC Manufacturing
had approximately $4,500,000 million in annual sales, with a forecast of $4,750,000 in 2016.
Reported annual sales in 2015 and 2016:
For 2015 calendar year……………………………………………………........$4,500,000
Estimate for 2016 calendar year………………………………….………….$4,750,000
John also knows that business is forecasted to grow an additional 5% to annual sales of $5
million dollars in 2017. However, John also knows there is some uncertainty with that forecast
and those annual sales next year could be more or less than $5 million dollars depending on
consumer demand, price of materials, and other uncertainties in the market. Given this
uncertainty, this is how John would complete the following uncertainty forecast table for ABC’s
Manufacturing Plant annual sales for 2017
Approximate annual sales in 2017:
2017
scenarios,
from lowest
to highest

$Billion

$Million

$Thousand

Percentage
likelihood
(values in this
column
should sum
to 100%)

Lowest

$3

000

5%

Low

$4

000

10%

Medium

$5

000

70%

High

$6

000

10%

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 15 of 26

Highest

$7

000

5%
100%

Resolution:

Finding #16: Question 30
All respondents reported that they could provide the information requested for Question 30 from their
2015 records. As for providing an estimate for 2016, most respondents said this would be possible either
by using forecasting methods or by looking at previous numbers for the third and fourth quarters of the
previous year.
Recommendation: We have no recommendation for this finding.
Resolution:

Finding #17: Question 31
In both Boston and San Francisco, the majority of respondents comprehended this question and were
able to complete it when pushed to do so. This is in contrast to our findings in Houston and Detroit
during the first round of cognitive testing, where the majority of respondents did not comprehend the
question. Even though respondents did have an easier time completing this question during the second
round of cognitive testing, respondents did have a few suggestions on how to improve the question. For
example, a couple of respondents suggested changing the scenarios from five to three; a few
respondents were a bit confused with the likelihood column; a couple of respondents did not like the
labeling of the scenarios from lowest to highest, and suggested an alternative such as worst to best; and
a couple of respondents suggested an example on how to complete this question.
Recommendation: The survey sponsor would like to keep this question as is. As such, we
recommend keeping Question 31 but providing an example, as outlined in the recommendation
for Finding #15.
Resolution:

Finding #18: Question 32
The majority of respondents reported that they could provide the information request from this
question using their 2015 records. As for providing an estimate for 2016, most respondents said this
would be possible either by using forecasting methods or by looking at previous numbers for the third
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 16 of 26

and fourth quarters of the previous year. A few respondents felt their forecast for this question would
be more difficult as they were unsure what kinds of capital expenditures they would have in 2016.
Recommendation: We have no recommendation for this finding.
Resolution:

Finding #19: Question 34
Most respondents stated that they could report a number for 2015 using their existing HR records and
provide an estimate for 2016 looking at past trends.
The majority of respondents defined “employees” as MOPS defines the term.
Recommendation: We have no recommendation for this finding.
Resolution:

Finding #20: Question 35
Respondents stated that they could complete Question 35 but it would be a best guess. A few indicated
that they would use their forecasting for Question 31 to forecast Question 35.
Recommendation: We have no recommendation for this finding.
Resolution:

Finding #21: Question 36
Although most respondents indicated that they could answer Question 36 using their existing records,
there were multiple interpretations of the term “production related costs.” For example, some
respondents wanted to use the costs of their goods sold and then adjust it based off the examples listed
in the question. Yet other respondents debated including or excluding labor hours.
Recommendation: We recommend removing the term “production related costs” and just using
the portion of the example that is most relevant to what the sponsor wants collected: i.e., the
cost of materials, parts, containers, and packaging.
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 17 of 26

Finding #22: Question 37
For Question 37, respondents were able to complete this question but had the same misinterpretations
of “production related costs.”
Recommendation: We recommend implementing the same recommendation outlined in
Finding #21.
Resolution:

Finding #23: Question 43
Although most respondents could answer Question 43, there was some confusion over how to interpret
the terms “flexible hours” and “cross-trained.” Some establishments reported not tracking either of
these two constructs.
It is observed that the first two constructs are HR related (part-time and flexible hours) while the third
construct is operational.
Recommendation: We recommend providing definitions for flexible hours and cross-trained so
that respondents know how to interpret these constructs and provide an appropriate response.
Resolution:

Finding #24: Question 44
In San Francisco, most respondents did not have a difficult time identifying the type of production
occurring at their establishment. However, when thinking-aloud, they often identified more than one
word to describe their manufacturing process.
In Boston, almost all of the establishments felt compelled to pick more than one response. Several
respondents noted that cellular manufacturing and continuous flow go hand in hand and are not
mutually exclusive. A few respondents suggested providing definitions for each term.
Recommendation: We recommend bolding best to emphasize to respondents that they should
select one response. Also, consider adding definitions to each term.
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 18 of 26

Finding #25: Question 45
While most respondents were able to complete this question, a few were confused with the term
“family firm.” This confusion lied in firms that went public but still had a strong family presence in them.
One respondent was confused on how to quantify if an establishment is a family firm or not a family
firm.
Recommendation: We recommend changing family firm to owned by a family, or something
similar, if the intent of the question is to capture family-ownership information. We also
recommend adding “50% or more” so respondents are able to quantify the level of ownership.
For example, the question could read, “Is this establishment owned 50% or more by a family?”
Resolution:

Finding #26: Question 46
Although the majority of respondents were able to answer Question 46, a few were confused by the
term “multinational.”
Recommendation: We recommend dropping the term multinational so the question reads, “Is
this establishment part of a firm which has production establishments in other countries?”
Resolution:

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 19 of 26

About the Data Collection Methodology & Research Branch (DCMRB)
The Data Collection Methodology and Research Branch (DCMRB) in the Economic Statistical Methods
Division (ESMD) assists economic survey program areas and other governmental agencies with research
associated with the behavioral aspects of survey response and data collection. The mission of DCMRB is
to improve data quality in surveys while reducing survey nonresponse and respondent burden. This
mission is achieved by:
•

•

•
•

Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with postcollection evaluation methods, to assess the effectiveness and efficiency of the data collection
instruments and associated materials.
Conducting early stage scoping interviews to assist with the development of survey content
(concepts, specifications, question wording and instructions, etc.) by getting early feedback on it
from respondents.
Assisting program areas with the development and use of nonresponse reduction methods and
contact strategies.
Conducting empirical research to help better understand behavioral aspects of survey response,
with the aim of identifying areas for further improvement as well as evaluating the effectiveness
of qualitative research.

For more information on how DCMRB can assist your economic survey program area or agency, please
contact the Branch Chief, Amy Anderson Riemer.

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 20 of 26

Appendix A - Protocol
Research Questions:
•
•
•
•
•

Do respondents interpret the new questions and instructions in a manner that meets the
questionnaire’s intent?
Do respondents comprehend the terminology used in the 2015 MOPS? (If not, what terminology
are they using?)
Overall, is the requested information available in respondent’s records? (Furthermore, is it
available for the periods requested in the questionnaire?)
Did respondents need to go to other people or departments within their organization to report
the data requested on the 2015 MOPS?
How are the changes to the questions implemented after the first round of cognitive testing
performing?

Expected Length of Interview: 45-60 minutes
Materials Needed:
•
•
•
•
•

2-3 copies of the draft 2015 MOPS
Consent forms
Digital recorder
Business cards
Pen and/or pencils

Introduction:
•
•
•
•
•

People around the table: how long at company, position/title and responsibilities
Explain purpose of meeting: to understand the process used to answer questions
Explain that we are not testing the respondent – we only want information
Structure of meeting: general and specific questions about the form itself, as well as the
organization
Permission to record discussion? Have R sign consent form.

General probes to use throughout the interview:
•
•
•
•

Reflect back R’s specific answer: “you said…”
How did you arrive at this number/answer this question? / Can you tell me more about this?
What records (if any) would you look at? What line or line items would be of interest?
Specifically, what would you include in this number? What would you exclude?

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 21 of 26

Before discussing the 2015 MOPS questionnaire:
•
•

To begin, can you please tell me a little about your plant/or the plant you reported for?
o (If needed) What sorts of activities is it involved in?
What is your role at this plant/company?
o (If needed) Have you filled out government forms in the past? Which ones?
o (If needed) Generally speaking, how are government forms completed at your
plant/company?
o (If needed) Have you completed the MOPS before? (R would have received it in 2011 for
2010) (If yes) What was your overall impression of the MOPS?

Now, I am going to hand you a copy of the 2015 MOPS questionnaire. Since we are only interested in
feedback on some of the questions in the 2015 MOPS, we will only go through some of the questions on
some of the pages. Feel free to write in the questionnaire and think aloud as to how you would
complete these questions. After you complete a question, I will ask you follow-up questions about the
process you took to formulate an answer.
To begin, let us turn to Page 6 and look at Question 17. Please take a moment to read the question
and answer it.
•
•
•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
Did you notice the skip pattern?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question for 2010? What about 2015?

(Data and Decision Making)
Next, let us go to Question 24. Again, please take a moment to read the question and answer it.
•

•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
o (If needed) What kind of data was available in 2010? What kind of data is available in
2015?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question for 2010? What about 2015?

Question 25:
•
•

How would you go about completing this question?
o (If needed) What does the phrase “use of data” mean to you?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you
in the answer that you would provide to this question for 2010? What about 2015?

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 22 of 26

Question 26:
•
•
•

How would you go about completing this question?
Are there any other response options that would make sense to you?
On a scale of 1-5, which 5 being very confident and 1 not very confident, how confident are you
in the answers that you would provide to this question for 2010? What about 2015?

Question 27:
•

•
•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
o What does the phrase “production performance indicators from production technology
or instruments” mean to you? Can you give me some examples?
o What does the phrase “formal or informal feedback from managers” mean to you? Can
you give me some examples?
o What does the phrase “formal or informal feedback from production workers” mean to
you? Can you give me some examples?
o What about the phrase “data from outside the firm”; what does that phrase mean to
you?
How would you answer this question if one of the sources of data did not apply to your
establishment?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question for 2015? What about 2010?

Question 28:
•

•
•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
o What does the phrase “design of new products or services” mean to you?
o What does the term “demand forecasting” mean to you?
o What does the term “supply chain management” mean to you?
o Are there other activities at your plant influenced by data analysis that are not currently
listed as response options?
How would you answer this question if one of the activities did not apply to your establishment?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question for 2015? What about 2010?

Question 29:
•

In your own words, what is this question asking you?

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 23 of 26

•
•

o (If needed) How would you go about answering this question?
o What does the term “predictive analytics” mean to you?
How would you answer this question if your establishment did not use predictive analytics?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question for 2010? What about
2015?

(Uncertainty)
Instructions:
•
•

Take a minute and read the instructions under Section D – Uncertainty. Please let me know
when you are done.
In your own words, what are these instructions telling you to do?

Question 30:
•
•
•

How did you go about completing this question?
How would you know the approximate dollar value of products shipped for the 2016 calendar
year?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question for 2015? What about your estimate for
2016?

Question 31:
•

•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
o (If needed) Looking at the left column, can you tell me in your own words what this
column is asking you to do? What is the difference between lowest and low? How about
low and medium?
o (If needed) How would you know the approximate dollar value of shipments in 2017?
o (If needed) Looking at the right column, can you tell me in your own words what this
column is asking you to do?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question?

Question 32:
•

How would you go about completing this question?
o (If needed) What does the term “capital expenditures” mean to you?

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 24 of 26

•
•

How would you know the approximate dollar value of capital expenditures for the 2016
calendar year?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question?

Question 34:
•

•

How would you go about completing this question?
o (If needed) What does the term “employee” mean to you? Does the term include or
exclude contractors?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question?

Question 35:
•
•

In your own words, what is this question asking you?
o (If needed) How would you go about answering this question?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question?

Question 36:
•
•
•
•

How would you go about completing Question 36?
What does the term “production related costs” mean to you?
How would you know the approximate dollar value of intermediate inputs for the 2016 calendar
year?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answer that you would provide to this question?

(Background Characteristics)
Okay, let us skip to Question 43 on Page 13:
•
•
•
•
•

How would you go about completing this question?
What does the term “part-time” mean to you?
What about the term “cross-trained” or “rotated jobs”?
What about the term “working flexible hours” – what does that mean to you?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question?

How about Question 44?
Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 25 of 26

•
•
•

How would you go about completing this question?
(If R hesitates when seeing responses) What does “[insert term]” mean to you?
On a scale of 1-5, with 5 being very confident and 1 not very confident, how confident are you in
the answers that you would provide to this question?

Now, let us skip to Question 45 on Page 14:
•

What is Question 45 asking you?
o (If needed) How would you answer this question?
o What does the term “family firm” mean to you?

Finally, let us look at Question 46:
•

In your own words, what is Question 46 asking you?
o (If needed) How would you answer this question?
o What does the term “multinational firm” mean to you?

Wrapping Up:
•
•
•

Overall, what do you think of these questions that we looked at today?
Thinking about these questions you looked at today, who is the most appropriate person(s) in
your company to answer them.
Again, thinking about these questions that you answered today, how much time do you think it
would take you to complete them?

Thank you for your time today!

Findings and Recommendations from the Second Round of Cognitive Testing for the 2015 Management and Organizational Practices Survey
October 13, 2015
Page 26 of 26