Expert Reviews of Additional MOPS-HP Content and Decision Document

Economic Programs Directorate
Decision Document

DECISION DOCUMENT

Document Title

Approved
Denied

Standards Review of Questions added to MOPS-HP w/o Cognitive Testing

Document Purpose

Document decision
Request decision

Document Type

Policy (not security)
Security
Project - Specify project name: ___MOPS-HP______
Other - Please specify: __________________________________

EDMS #
Divisions/Offices
Affected

CES
EAD
EID
EMD
ERD

ESMD
EWD

Decision Document #
(if applicable)

N/A

Systems Affected
(for software development
projects only)

N/A

Author

Alice Zawacki, Center for
Economic Studies
Diane Willimack, ESMD

Date Submitted for
Approval

Approver (s)

Chief, Economic Statistical
Methods Division
Chief, Economy-Wide
Statistics Division
Director, Center for Economic
Studies

Date Approved/Denied

May x, 2020

1

I.

STATEMENT OF PROBLEM/ISSUE

The current pandemic highlights the relevance of hospital management practices especially as they relate to
hospitals’ ability to respond to shocks to their organization and the health care system. The Census Bureau
can help improve measurement of hospital preparedness in 2014 and 2019 by adding two questions
proposed below to the Management and Organizational Practices Survey-Hospitals (MOPS-HP). These
questions would provide information on two elements of responsiveness, hospitals’ coordinated
deployment of frontline clinical workers and hospitals’ ability to quickly respond to needed changes in
standardized clinical protocols. 1
The Census Bureau’s Statistical Quality Standard A2, Developing Statistical Data Collection Instruments
and Supporting Materials (see https://www.census.gov/about/policies/quality/standar)ds/standarda2.html)
requires that any new or substantially revised survey questions be pretested with respondents (see subrequirement A2-3.3). The MOPS-HP, as a brand new survey, underwent substantial cognitive testing in
July - August, 2018, to aid its development. With the MOP-HP Information Collection Request (ICR)
ready for final submission to OMB, and data collection planned to begin in autumn 2020, there is
insufficient time to complete cognitive testing of the two new questions proposed for this collection.
Statistical Quality Standard A2 describes contingencies when cognitive pretesting with respondents may be
infeasible under circumstances such as “extreme budget, resource, or time constraints.” In these instances,
according to A2, “subject matter and cognitive experts must discuss the need for and feasibility of
pretesting,” and the program manager may request that cognitive experts conduct a methodological expert
review. “If the cognitive experts do not agree that an expert review would satisfy this requirement, the
program manager must apply for a waiver” from the M&S Council. Thus, the standard seems to indicate
that if the cognitive reviewers and the SMEs can reach agreement through collaboration motivated by the
expert review, then a waiver may not be required.
On April 16, 2020, EWD’s MOPS-HP survey managers and CES stakeholders reached out to cognitive
reviewers in ESMD for guidance, and requested a cognitive expert review of the two additional questions
proposed for the survey. Two members of ESMD’s Data Collection Methodology & Research Staff were
assigned to independently2 conduct expert reviews of the proposed questions, and delivered their reviews
on May 5, 2020. Subsequently, the cognitive reviewers, survey managers, and subject matter experts met
to discuss the reviewers’ recommendations, seek clarifications, and collaboratively developed acceptable
alternatives, upon which mutual agreement was reached.
II.

DECISION

The decision is to implement in the 2019 MOPS-HP instrument the versions of the two additional
questions as agreed upon by the ESMD/DCMRB cognitive reviewers, the CES and EWD SMEs, and the
EWD survey managers, without additional cognitive testing.
Final questions and their placement follow:

1

Since two new questions are being added, the Census Bureau and the sponsor have also identified two existing
questions on medical record documentation that will be dropped from the MOPS-HP.
2
At their April 9, 2020, meeting, the M&S Council recommended that more than one expert review be conducted
independently when expert reviews are requested under the A2 contingency.

2

Question 1: Moving frontline clinical workers to different units within the hospital
Final wording
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different units within this
hospital when needed (for example, in response to understaffing or increased patient care needs)?
Select one box for each year
o Frontline clinical workers were moved to different units within this hospital when needed, and ONE
central office coordinated this process
o Frontline clinical workers were moved to different units within this hospital when needed, but NO
one central office coordinated this process
o Frontline clinical workers were not moved to different units within this hospital when needed
Placement
This question will be added to Section F: Staffing and Allocation of Human Resources on the MOPS-HP
survey form after existing question #29 that asks who decided how work was allocated.
Question 2: Modifying and updating standardized clinical protocols
Final wording
In 2014 and 2019, within what time period did this hospital typically MODIFY or UPDATE its
standardized clinical protocols after the need to do so was first identified?
Select one box for each year
o Within one week of identifying the need
o Within one month of identifying the need
o Within three months of identifying the need
o Within six months of identifying the need
o More than six months after the need was first identified
Placement
Added to Section G: Standardized Clinical Protocols on the MOPS-HP survey form after existing question
#34 that asks who modified or updated protocols. This new question will be skipped if response to #34
indicates the hospital does not usually modify or update their protocols.

ESMD’s Methodology Director for Measurement & Response Improvement and ESMD’s chief concur
that this decision and the process used to achieve it suffice and satisfy the intent of the directions per
Statistical Quality Standard A2 for Developing Statistical Data Collection Instruments and Supporting
Materials.
III.

SUPPORTING INFORMATION

The final versions of the two proposed questions added to 2019 MOPS-HC appear in Attachment 1, which
also includes documentation of the original question versions and the collaborative process that led to their
adoption. Attachments 2 and 3 contain the written reports completed by the cognitive expert reviewers, as
required by A2.

3

IV.

DECISION DOCUMENT UPDATES/CHANGE HISTORY
Last
Update

Reason for Change

Document
Version

Approved by:

1
2

V.

APPROVAL
Approving Official

Signature

Date:

Carol V. Caldwell
Chief, ESMD
Kimberly P. Moore
Chief, EWD
Lucia Foster
Director, CES

cc:

Alice Zawacki, CES
Scott Ohlmacher, CES
Edward Watkins, EWD
Savita Sathya, EWD
Diane K. Willimack, ESMD
Amy Anderson Riemer, ESMD
Kristin Stettler, ESMD
Rebecca Keegan, ESMD
Econ Hub Change Coordination Process (CCP) SharePoint Site

4

ATTACHMENT 1
Documentation of MOPS-HP Question Additions and EWD/ESMD Collaborations
To: Diane Willimack, Amy Anderson Reimer, Kristin Stettler, and Rebecca Keegan
From: Alice Zawacki
CC: Edward Watkins, Savita Sathya, Scott Ohlmacher
Date: 5.14.2020
DRAFT Decision Document for adding 2 new questions motivated by the COVID-19 pandemic to
the Management and Organizational Practices-Hospitals (MOPS-HP)
The current pandemic highlights the relevance of hospital management practices especially as they relate to
hospitals’ ability to respond to shocks to their organization and the health care system. The Census Bureau
can help improve measurement of hospital preparedness in 2014 and 2019 by adding the following
proposed questions to the Management and Organizational Practices Survey-Hospitals (MOPS-HP). These
questions would provide information on two elements of responsiveness, hospitals’ coordinated
deployment of frontline clinical workers and hospitals’ ability to quickly respond to needed changes in
standardized clinical protocols. 3
Following communications between the testing staff, survey director, survey sponsor, and CES, the 2
following questions have been finalized. This document summarizes the final decisions on this content
based on responses to the findings and recommendations from the expert reviews conducted by Kristin
Stettler and Rebecca Keegan, the positioning of these questions within the existing MOPS-HP survey, and
other details are provided below.
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different units within
this hospital when needed (for example, in response to understaffing or increased patient care
needs)?
Select one box for each year
o Frontline clinical workers were moved to different units within this hospital when needed, and
ONE central office coordinated this process
o Frontline clinical workers were moved to different units within this hospital when needed, but NO
one central office coordinated this process
o Frontline clinical workers were not moved to different units within this hospital when needed
In 2014 and 2019, within what time period did this hospital typically MODIFY or UPDATE its
standardized clinical protocols after the need to do so was first identified?
Select one box for each year
o
o
o
o
o

Within one week of identifying the need
Within one month of identifying the need
Within three months of identifying the need
Within six months of identifying the need
More than six months after the need was first identified

3 Since two new questions are being added, the Census Bureau and the sponsor have also identified two existing
questions on medical record documentation that will be dropped from the MOPS-HP.

5

I.

Question 1: Moving frontline clinical workers to different units within the hospital
The first question will collect data from 2014 and 2019 to measure (1) whether frontline clinical
workers were moved to different units within the hospital when needed and (2) if moved, did one
central office coordinate the process. This final content reflects edits made to the original proposed
question (see below).
Final wording
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different units
within this hospital when needed (for example, in response to understaffing or increased
patient care needs)?
Select one box for each year
o Frontline clinical workers were moved to different units within this hospital when needed,
and ONE central office coordinated this process
o Frontline clinical workers were moved to different units within this hospital when needed,
but NO one central office coordinated this process
o Frontline clinical workers were not moved to different units within this hospital when
needed
Placement
This question will be added to Section F: Staffing and Allocation of Human Resources on the
MOPS-HP survey form after existing question #29 that asks who decided how work was allocated.
Other notes: Show/Hide definition of frontline clinical workers.
Responses to recommendations from expert reviews
•

•
•

•
•
•
•
•
•

Since the question’s focus is on how frontline clinical workers are moved, and not on why they’re
moved, language was added with 2 major reasons why a hospital unit might be busy and in need of
frontline clinical workers (increased patient demand and/or decreased labor supply). It’s noted that
this might lead to some responses that do not reflect the movement of frontline clinical workers for
other reasons, such as cross-training or employee requests to change units.
Note: If a respondent was not working at the hospital in 2014, they are not asked to respond for
2014
Sponsor prefers this format rather than making it a yes/no question, because she would like to keep
track of movements without coordination, and not sure a yes/no question would collect this
information
Responses rearranged from most to least likely
Removed ‘applicable skill set’ from response options
Definition of frontline clinical workers will be a show/hide feature on the Centurion instrument.
As recommended, we capitalized ‘NO’ and also ‘ONE’ in the first response.
“ONE central person” was replaced with “ONE central office” because we think it is more
realistic to think more than one person would coordinate movement of frontline clinical workers
We also replaced “across hospital units” with “different units within this hospital” since workers
may be moved across hospitals under the same ownership and we want to collect establishmentlevel data.

6

Original Proposed Question #1
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different
hospital units that became busier than others?
Select one box for each year
o
o
o

Frontline clinical workers did not move across hospital units
Frontline clinical workers moved across hospital units based on their applicable skills
but no central person coordinated this process
Frontline clinical workers moved across hospital units based on their applicable skills and
one central person coordinated this process

FRONTLINE CLINICAL WORKERS include all clinical staff with direct patient care
responsibilities (such as nurses, nurses’ aides, physical/occupational/speech/respiratory
therapists, radiology and laboratory technicians), who do NOT have employees directly
reporting to them. Do NOT include non-clinical frontline staff such as food services,
housekeeping, or maintenance staff.

II.

Question 2: Modifying and updating standardized clinical protocols
The second question will collect data from 2014 and 2019 to measure how quickly the hospital
modifies and updates standardized clinical protocols after the need to do so has been identified. This
final content reflects edits made to the original proposed question (see below).
Final wording
In 2014 and 2019, within what time period did this hospital typically MODIFY or UPDATE its
standardized clinical protocols after the need to do so was first identified?
Select one box for each year
o
o
o
o
o

Within one week of identifying the need
Within one month of identifying the need
Within three months of identifying the need
Within six months of identifying the need
More than six months after the need was first identified

Placement
Added to Section G: Standardized Clinical Protocols on the MOPS-HP survey form after existing
question #34 that asks who modified or updated protocols. This new question will be skipped if
response to #34 indicates the hospital does not usually modify or update their protocols.
Responses to recommendations from expert reviews
•

Added, replaced, and collapsed responses to arrive at the final options. Not including a response
for ‘never,’ since a ‘never’ response would essentially say that the hospital doesn’t modify or
update their protocols. The previous MOPS-HP question includes a response for hospitals that
don’t modify or update their protocols and they would skip this proposed question.

7

•
•
•

‘Typically’ or ‘typical’ is used in Q17-21 and Q30 similarly to how this question is drafted, and
will keep.
Clinical standardized protocols was a well-understood concept and it was decided that examples
and/or a definition weren’t necessary to include
Since we use ‘weekly’ references elsewhere on the MOPS-HP, keep responses to this question
consistent and use ‘week’ rather than ‘7 days’

Original proposed question #2
In 2014 and 2019, how long did this hospital typically take to MODIFY or UPDATE its
standardized clinical protocols after the need to do so was first identified?
Select one box for each year
o Within one day of identifying the need
o Within one week of identifying the need
o Within one month of identifying the need
o Within three months of identifying the need
o Within one year of identifying the need
o Never

8

ATTACHMENT 2
Cognitive Expert Review by Kristin Stettler, ESMD/DCMRB

9

Expert Review for the Measurement and
Organizational Practices-Hospitals
Survey (MOPS-HP), Two New Proposed
Questions
Prepared for:
Alice Zawacki, Center for Economic Studies (CES)

Prepared by:
Kristin Stettler, Data Collection Methodology and Research Branch (DCMRB)
Economic Statistical Methods Division

May 1, 2020 draft

Table of Contents
Executive Summary....................................................................................................................................... 3
Proposed New Questions.............................................................................................................................. 4
Findings and Recommendations ................................................................................................................... 6
Finding #1 – Proposed New Question #1.................................................................................................. 6
Finding #2 – Proposed New Question #2.................................................................................................. 7
Economic Directorate Guidelines on Questionnaire Design......................................................................... 9
About the Data Collection Methodology and Research Branch (DCMRB) ................................................. 13
Appendix A: Proposed New Questions and Preceding Questions ............................................................. 14

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 2 of 15

Executive Summary
Kristin Stettler from the Data Collection Methodology and Research Branch, in the Economic Statistical
Methods Division, reviewed two new proposed questions under consideration to be added to the
Management and Organizational Practices Survey for Hospitals (MOPS-HP). The survey is designed to be
completed by the Chief Nursing Officer.
The two new proposed questions are intended to learn more about how hospitals prepare to adjust
resources and efforts in a crisis. The first proposed question raises some concerns about what the
measurement goal is and how respondents will interpret the question and answer categories. The
second proposed question seems fairly clear and only two minor revisions are suggested. In both cases,
it would help to understand what the measurement goal of the questions is.
Additionally, we note that reporting for 2014 will likely be difficult for many respondents, especially
those who did not work there at the time. This issue was discussed in further detail in the cognitive
interviewing reports for Rounds 1 and 2.

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 3 of 15

Proposed New Questions
The two new proposed questions are intended to learn more about how hospitals prepare to adjust
resources and efforts in a crisis.
The first new question would be placed in Section F (Staffing and Allocation of Human Resources), after
existing question #29 that asks who decided how work was allocated.
The second new question would be placed in Section G (Standardized Clinical Protocols), after existing
question #34 that asks who modified or updated protocols. (Note: This new question will be skipped if
response to #34 indicates the hospital does not modify or update their protocols.)
See Appendix A for the two new questions and their two preceding questions.
Proposed New Question #1
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different hospital units that
became busier than others?
Select one box for each year
Frontline clinical workers did not move across hospital units
Frontline clinical workers moved across hospital units based on their applicable skills but no central
person coordinated this process
Frontline clinical workers moved across hospital units based on their applicable skills and one central
person coordinated this process
FRONTLINE CLINICAL WORKERS include all clinical staff with direct patient care responsibilities (such as
nurses, nurses’ aides, physical/occupational/speech/respiratory therapists, radiology and laboratory
technicians), who do NOT have employees directly reporting to them. Do NOT include non-clinical
frontline staff such as food services, housekeeping, or maintenance staff.
Proposed New Question #2
In 2014 and 2019, how long did this hospital typically take to MODIFY or UPDATE its standardized
clinical protocols after the need to do so was first identified?
Select one box for each year
Within one day of identifying the need
Within one week of identifying the need
Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 4 of 15

Within one month of identifying the need
Within three months of identifying the need
Within one year of identifying the need
Never

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 5 of 15

Findings and Recommendations
Finding #1 – Proposed New Question #1
The question asks about how frontline clinical workers were moved to different hospital units that
“became busier than others”. The answer categories, however, do not mention why the workers were
moved. It seems likely that workers are moved between units for many reasons. Therefore, the first
response category, “Frontline clinical workers did not move across hospital units” seems unlikely to be
selected. (At a minimum, it should be clarified, “Frontline clinical workers did not move across hospital
units due to ….”)
The phrase “become busier than others” may not be the best way to describe the situation. Perhaps
referencing changes in demand or patient load or staffing needs would be more appropriate?
The last two answer categories assume that the workers were moved “based on their applicable skills”
but that does not seem to be the issue of concern. What if it was not “based on their applicable skills”?
Consider deleting “based on their applicable skills” in both answer categories.
What does one “central” person mean? (For consistency, add “one” before “central person” in the
second answer category.) Perhaps “one single person”? If it was not one “central” person, what are the
likely scenarios? What if it was a team or another decision-making body? How would/should they
answer?
What are you trying to measure with this question?
Reporting for 2014 will likely be difficult for many respondents, especially those who did not work there
at the time. This issue was discussed in further detail in the cognitive interviewing reports for Rounds 1
and 2.
Finally, the definition of FRONTLINE CLINICAL WORKERS provided under the question text may not be
necessary, as the definition was provided in Question #6 and the term was used in multiple other
questions, without accompanying definition.
Possible suggested rewording:
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved across hospital units due to large
changes in patient loads?
One single person coordinated movement of frontline clinical workers across hospital units due to large
changes in patient loads
No single person coordinated movement of frontline clinical workers across hospital units due to large
changes in patient loads
Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 6 of 15

Frontline clinical workers did not move across hospital units due to large changes in patient loads

Finding #2 – Proposed New Question #2
This question seems fairly clear.
Is “typically” something respondents can comfortably report for? How many modifications of
standardized clinical protocols occur in a year? If two, and one took one day and one took 8 months,
how would they answer? If dozens, and there is a wide range, how would they answer?
What about those who took more than a year? Consider adding another answer category before Never:
“More than one year after identifying the need”
What are you trying to measure with this question?
It was noted that those who answered “Standardized clinical protocols were not usually modified or
updated at this hospital” to the previous question (#34) would skip this question. However, since it says
“not usually” that means that they could be modified or updated, so you may still want them to answer
this question. Perhaps allow a “Not applicable” answer category?
Was “standardized clinical protocols” ever defined in this questionnaire?
Reporting for 2014 will likely be difficult for many respondents, especially those who did not work there
at the time. This issue was discussed in further detail in the cognitive interviewing reports for Rounds 1
and 2.
Possible suggested rewording:
In 2014 and 2019, how long did this hospital typically take to MODIFY or UPDATE its standardized
clinical protocols after the need to do so was first identified?
Select one box for each year
Within one day of identifying the need
Within one week of identifying the need
Within one month of identifying the need
Within three months of identifying the need
Within one year of identifying the need
Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 7 of 15

More than one year after identifying the need
Never
Not Applicable

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 8 of 15

Economic Directorate Guidelines on Questionnaire Design
For further information about the Economic Directorate Guidelines on Questionnaire Design, please
click here.

Is the
guideline
followed?

Guideline

Recommendation

Wording
1. Phrase data requests as
questions or imperative
statements, not sentence
fragments or keywords.

Yes

2. Break down complex
questions into a series of
simple tasks.

Yes

3. Use a consistent reference
period.

Yes

4. Avoid jargon and terms that
may be unfamiliar to
respondents.

Sometimes

5. Use consistent terms
throughout the question
(Use the same terms
consistently throughout the
questionnaire).

Yes

6. For Web instruments,
consider using automatic
calculations for questions
that require calculations

N/A

7. For Web instruments, use
automatic fills when
questions reference

N/A

Some of the wording may be a bit jargon.

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 9 of 15

information reported on
earlier screens or available
only on paper
questionnaires (e.g., mailing
labels).

Response Options and Answer Spaces
8. Use similar answer spaces
when requesting the same
type of information.

Yes

9. Use consistent response
across similar items.

N/A

10. Clearly indicate the unit of
measurement for each data
item.

N/A

11. Decide whether to provide
previously reported data to
respondents after weighing
the potential data quality
benefits and risks and the
potential disclosure and
security risks.

N/A

12. Provide “Mark X if None”
checkboxes only if it is
necessary to differentiate
between item nonresponse
and reported values of zero.

N/A

Visual Features and Layout
Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 10 of 15

13. Use font variations
consistently and for a single
purpose with a
questionnaire.

N/A

14. Group data items and their
answer spaces / response
options.

N/A

15. Evaluate the necessity of
any graphics, images, and
diagrams to ensure that
they are useful for
respondents.

N/A

Navigation
16. Use a consistent page or
screen layout.

N/A

17. Clearly identify the start of
each question and section.

Yes

18. Group similar data items
together.

Yes

19. Use blank space to separate
questions and make it easier
to navigate within
questionnaires.

N/A

20. Align questions and answer
spaces / response options.

N/A

21. Use strong visual features to
emphasize skip instructions.

N/A

22. Inform respondents of the
navigational path when a
question continues on
another page.

N/A

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 11 of 15

Instructions
23. Use mode-specific
instructions.

N/A

24. Incorporate questionspecific instructions into a
survey instrument where
they are needed. Avoid
placing instructions in a
separate sheet / booklet /
webpage.

Yes

25. Consider reformulating
important instructions as
questions.

N/A

26. Convert narrative
paragraphs to bulleted lists.

N/A

27. When possible, use an
actual date, rather than a
vague timeframe, to
reference due dates.

N/A

Matrices
28. Limit the use of matrices.
Consider the potential
respondent’s level of
familiarity with tables when
deciding whether to use
them.

Yes

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 12 of 15

About the Data Collection Methodology and Research Branch (DCMRB)
The Data Collection Methodology and Research Branch (DCMRB) in the Economic Statistical Methods
Division assists economic survey program areas and other governmental agencies with research
associated with the behavioral aspects of survey response and data collection. The mission of DCMRB is
to improve data quality in surveys while reducing survey nonresponse and respondent burden. This
mission is achieved by:
•

•

•
•

Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with postcollection evaluation methods, to assess the effectiveness and efficiency of the data collection
instruments and associated materials;
Conducting early stage scoping interviews to assist with the development of survey content
(concepts, specifications, question wording and instructions, etc.) by getting early feedback on it
from respondents;
Assisting program areas with the development and use of nonresponse reduction methods and
contact strategies;
And conducting empirical research to help better understand behavioral aspects of survey
response, with the aim of identifying areas for further improvement as well as evaluating the
effectiveness of qualitative research.

For more information on how DCMRB can assist your economic survey program area or agency, please
visit the DCMRB intranet site or contact the branch chief, Amy Anderson Riemer.

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 13 of 15

Appendix A: Proposed New Questions and Preceding Questions
29. In 2014 and 2019, who decided how work was allocated to clinical staff at this hospital?
Select all that apply
Senior clinical managers (CNO,CMO)
Senior non-clinical managers (CEO, CFO, COO)
Clinical managers
Non-clinical managers
Providers
Frontline clinical workers
Proposed New Question #1
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different hospital units that
became busier than others?
Select one box for each year
Frontline clinical workers did not move across hospital units
Frontline clinical workers moved across hospital units based on their applicable skills but no central
person coordinated this process
Frontline clinical workers moved across hospital units based on their applicable skills and one central
person coordinated this process
FRONTLINE CLINICAL WORKERS include all clinical staff with direct patient care responsibilities (such as
nurses, nurses’ aides, physical/occupational/speech/respiratory therapists, radiology and laboratory
technicians), who do NOT have employees directly reporting to them. Do NOT include non-clinical
frontline staff such as food services, housekeeping, or maintenance staff.

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 14 of 15

34. In 2014 and 2019, who of the following MODIFIED or UPDATED standardized clinical protocols at
this hospital?
Select all that apply
Senior clinical managers (CNO, CMO)
Senior non-clinical managers (CFO, CEO, COO)
Clinical managers
Non-clinical managers
Providers
Frontline clinical workers
Standardized clinical protocols were not usually modified or updated at this hospital
Proposed New Question #2
In 2014 and 2019, how long did this hospital typically take to MODIFY or UPDATE its standardized
clinical protocols after the need to do so was first identified?
Select one box for each year
Within one day of identifying the need
Within one week of identifying the need
Within one month of identifying the need
Within three months of identifying the need
Within one year of identifying the need
Never

Expert Review for MOPS-Hospitals, Proposed New Questions
April 30, 2020
Page 15 of 15

ATTACHMENT 3
Cognitive Expert Review by Rebecca Keegan, ESMD/DCMRB

10

Expert Review for the Management and
Organizational Practices Survey for
Hospitals
Prepared for:
Alice Zawacki, Center for Economic Studies (CES)

Prepared by:
Rebecca Keegan, Data Collection Methodology and Research Branch (DCMRB)
Economic Statistical Methods Division

May 1st, 2020

Table of Contents
Executive Summary....................................................................................................................................... 3
Findings and Recommendations: First Draft Question ................................................................................. 4
Finding #1 – Rearrange response options in order of most to least likely to be selected ....................... 4
Finding #2 – Remove “applicable skill set” from response options.......................................................... 4
Finding #3 – Move definition of “Frontline Clinical Worker” above response options ............................ 5
Finding #4 – Consider emphasizing the main distinction between response options ............................. 5
Findings and Recommendations: Second Draft Question ............................................................................ 5
Finding #5 – Clarify “Never” in response options ..................................................................................... 5
Finding #6 – Change “one week” time frame to say “seven days”........................................................... 6
Finding #7 – Consider adding a six month response option ..................................................................... 6
About the Data Collection Methodology and Research (DCMR) Branch...................................................... 7
Appendix A: Proposed Draft Questions ........................................................................................................ 8

Expert Review for MOPS-Hospitals
May 1st, 2020
Page 2 of 8

Executive Summary
Staff from the Data Collection Methodology and Research Branch in the Economic Statistical Methods
Division reviewed a list of questions that was prepared for the Management and Organizational
Practices Survey for Hospitals (MOPS).
The MOPS-HP is a survey that aims to measure the management and organizational practices of General
Medical and Surgical Hospitals, as well these establishments’ productivity growth.
Staff reviewed two new questions that will be added to the MOPS-HP for clarity and to ensure that the
questions measure what they are intending to measure. The following section reports on findings and
recommendations for areas that we believe can be improved. See Appendix A for the proposed draft
questions.
The key findings are as follows:
Findings and Recommendations: First Draft Question
•
•
•
•

Finding #1 – Rearrange response options in order of most to least likely to be selected
Finding #2 – Remove “applicable skill set” from response options
Finding #3 – Move definition of “Frontline Clinical Worker” above response options
Finding #4 – Consider emphasizing the main distinction between response options

Findings and Recommendations: Second Draft Question
•
•
•

Finding #5 – Clarify “Never” in response options
Finding #6 – Change “one week” time frame to say “seven days”
Finding #7 – Consider adding a six month response option

Expert Review for MOPS-Hospitals
May 1st, 2020
Page 3 of 8

Findings and Recommendations: First Draft Question
Finding #1 – Rearrange response options in order of most to least likely to be
selected
Ease respondent cognitive burden by reordering the response options by listing the response option
presumed to be most frequently selected first. For example, if it is most often the case that a central
person coordinates clinical worker’s movements, list that option first. If presumably it would be unlikely
that there was no movement across hospital units, list that option last.
• Suggested revision:
o
o
o

Frontline clinical workers moved across hospital units based on their applicable skills and
one central person coordinated this process
Frontline clinical workers moved across hospital units based on their applicable skills but no
central person coordinated this process
Frontline clinical workers did not move across hospital units

Finding #2 – Remove “applicable skill set” from response options
The response options for this question describe two different concepts in one response. The response
options attempt to determine whether or not a central person was involved in the movement of a
frontline worker from one unit to another, but additionally response options assume the reasoning
behind this movement is attributed to the clinicians “applicable skill set”. This may lead to issues where
the respondent is unsure how to answer, because one answer may apply to one of the concepts (i.e.,
presence or absence of central person), and a different answer may apply to a different concept (i.e.,
specific reason for moving frontline worker).
It may be beneficial to remove the aspect of the question responses that pertains to the reason why the
frontline worker was moved. Consider removing the extraneous wording “based on their applicable
skills” which assigns one reason in particular for the move occurring. This could potentially confuse
respondents whose frontline clinical workers were moved to other units for reasons besides their
applicable skill sets, such as seniority in their position, managerial potential, or for some other reason.
•

Suggested revision:
o Frontline clinical workers moved across hospital units, and one central person
coordinated this process
o Frontline clinical workers moved across hospital units, but no central person
coordinated this process
o Frontline clinical workers did not move across hospital units

Another option that would disentangle whether a central person was involved in moving frontline
clinicians and the reason why the move occurred, would be to rewrite this question as a “Yes/No”
question.
Expert Review for MOPS-Hospitals
May 1st, 2020
Page 4 of 8

•

Suggested revision:

In 2014 and 2019, was a central person involved in moving FRONTLINE CLINICAL WORKERS to different
hospital units that became busier than others?
o
o
o

Yes
No
Frontline Clinical Workers were not moved to different hospital units.

Finding #3 – Move definition of “Frontline Clinical Worker” above response
options
Provide any definitions of terminology used in the question prior to listing response options, so that
respondents do not need to search for clarification. Placing the definition of Frontline Clinical Worker
directly below the question text will ease cognitive burden.

Finding #4 – Consider emphasizing the main distinction between response
options
The two response options involving either the presence or absence of a central person coordinating the
frontline worker’s move are worded very similarly, and therefore it may benefit respondents to
emphasize the operative distinction between the two response options. Consider either bolding,
italicizing or capitalizing the word “no” within the response option.
•

Suggested revision:
o Frontline clinical workers moved across hospital units based on their applicable skills
and one central person coordinated this process
o Frontline clinical workers moved across hospital units based on their applicable skills but
NO central person coordinated this process
o Frontline clinical workers did not move across hospital units

Findings and Recommendations: Second Draft Question
Finding #5 – Clarify “Never” in response options
The response option “Never” can be further clarified. Although it is noted that this question should be
skipped and thus not applicable to respondents whose protocols did not need updating, respondents
could nonetheless misinterpret the “never” response option in this case to mean that they never had to
modify or update their protocols. If this response option in in fact intended to indicate that although a
Expert Review for MOPS-Hospitals
May 1st, 2020
Page 5 of 8

need to update or modify protocols was identified, updates were never made, then this should be
explicitly stated.
•

Suggested revision:

How long did this hospital typically take to MODIFY or UPDATE its standardized clinical protocols after
the need to do so was first identified?
o
o
o
o
o
o

Within one day of identifying the need
Within seven days of identifying the need
Within one month of identifying the need
Within three months of identifying the need
Within one year of identifying the need
Protocols were never modified or updated despite identifying the need

Finding #6 – Change “one week” time frame to say “seven days”
Respondents may interpret the phrase “one week” differently, with some interpreting this to mean a
business week from Monday to Friday, and others including the weekend. As it is the case that hospitals
maintain working hours throughout the weekend, it may benefit respondents to clarify the “one week”
phrase by instead using a more specific definition of “seven days”.
•

Suggested revision:
o Within one day of identifying the need
o Within seven days of identifying the need
o Within one month of identifying the need
o Within three months of identifying the need
o Within one year of identifying the need
o Never

Finding #7 – Consider adding a six month response option
It may benefit respondents to expand the time ranges available to choose from to include a six month
time frame. Some respondents might prefer to report a change that occurred within a half-year block of
time, as opposed to the current 9 month break between three months and a year. Including a six month
option may increase reporting accuracy.
•

Suggested revision:
o Within one day of identifying the need
o Within one week of identifying the need
o Within one month of identifying the need
o Within three months of identifying the need
o Within six months of identifying the need
o Within one year of identifying the need
o Never
Expert Review for MOPS-Hospitals
May 1st, 2020
Page 6 of 8

About the Data Collection Methodology and Research (DCMR) Branch
The Data Collection Methodology and Research (DCMR) Branch in the Economic Statistical Methods
Division assists economic survey program areas and other governmental agencies with research
associated with the behavioral aspects of survey response and data collection. The mission of DCMR is
to improve data quality in surveys while reducing survey nonresponse and respondent burden. This
mission is achieved by:
•

•

•
•

Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with postcollection evaluation methods, to assess the effectiveness and efficiency of the data collection
instruments and associated materials;
Conducting early stage scoping interviews to assist with the development of survey content
(concepts, specifications, question wording and instructions, etc.) by getting early feedback on it
from respondents;
Assisting program areas with the development and use of nonresponse reduction methods and
contact strategies;
And conducting empirical research to help better understand behavioral aspects of survey
response, with the aim of identifying areas for further improvement as well as evaluating the
effectiveness of qualitative research.

For more information on how DCMR can assist your economic survey program area or agency, please
visit the DCMR intranet site or contact the branch chief, Amy Anderson Riemer.

Expert Review for MOPS-Hospitals
May 1st, 2020
Page 7 of 8

Appendix A: Proposed Draft Questions
Management and Organizational Practices-Hospitals (MOPS-HP):
Proposed additional questions (A. Zawacki. 4.16.2020)
Additional Proposed Questions (See below for MOPS-HP’s current questions)
1. Section F: Staffing and Allocation of Human Resources
Placement: After existing question #29 that asks who decided how work was allocated
In 2014 and 2019, how were FRONTLINE CLINICAL WORKERS moved to different hospital
units that became busier than others?
Select one box for each year
o Frontline clinical workers did not move across hospital units
o Frontline clinical workers moved across hospital units based on their applicable skills but no
central person coordinated this process
o Frontline clinical workers moved across hospital units based on their applicable skills and one
central person coordinated this process
FRONTLINE CLINICAL WORKERS include all clinical staff with direct patient care responsibilities
(such as nurses, nurses’ aides, physical/occupational/speech/respiratory therapists, radiology and
laboratory technicians), who do NOT have employees directly reporting to them. Do NOT include nonclinical frontline staff such as food services, housekeeping, or maintenance staff.
2. Section G: Standardized Clinical Protocols

Placement: After existing question #34 that asks who modified or updated protocols
Note: This new question will be skipped if response to #34 indicates the hospital does not modify or
update their protocols.
In 2014 and 2019, how long did this hospital typically take to MODIFY or UPDATE its
standardized clinical protocols after the need to do so was first identified?
Select one box for each year
o Within one day of identifying the need
o Within one week of identifying the need
o Within one month of identifying the need
o Within three months of identifying the need
o Within one year of identifying the need
o Never

Expert Review for MOPS-Hospitals
May 1st, 2020
Page 8 of 8