Supporting Statement Part A -- Pilot Test of the Proposed Value and Efficiency Surveys and Checklist 1-10-2013

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality; and

4. research on the costs and utilization of, and access, to health care.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

Maximizing value within the American health care system is an important priority. Value is often viewed as a combination of high quality, high efficiency, and low cost care, and there is general agreement by consumers, policy makers, payers, and providers that it is lacking in the U.S. A recent report by the Institute of Medicine estimated that 20 to 30 percent ($765 billion a year) of U.S. healthcare spending was inefficient and could be reduced without lowering quality.¹

Multiple overlapping initiatives are currently seeking to improve value using a variety of approaches. Public reporting efforts led by the Centers for Medicare and Medicaid Services (CMS), other payers and consumer groups seek to enable consumers to make more informed choices about the quality, and in some cases, the costs of their care. A variety of demonstration projects and payment reforms initiated by CMS and private insurers are attempting to more closely link care quality with payments to incent higher value care. And national improvement initiatives led by AHRQ (comprehensive unit-based safety programs [CUSP] for central line-associated blood stream infection [CLABSI], catheter-associated urinary tract infections [CUTI], and surgical units [SUSP]) and CMS (hospital engagement networks, QIO scopes of work) are seeking to raise care quality and reduce readmissions. Results from the CUSP: CLABSI project have demonstrated that central line infections can be reduced and unnecessary costs can be removed from the health care system by concerted, unit-based improvement efforts. As a systems level example, Denver Health, with initial funding from AHRQ, has taken major steps towards redesigning clinical and administrative processes so as to reduce staff time, patient waiting, and unnecessary costs. These improvements occurred without harm to quality and in some instances actually improved quality.

In many cases, improving quality improves efficiency naturally. Reducing the number of hospital errors for example will reduce costs associated with longer length of stay or error-triggered readmissions. It is more cost-effective to do things right the first time. But higher value is more likely if organizations doing quality improvement link efforts to improve care quality with efforts to reduce unnecessary costs. Ignoring the financial implications of quality improvement efforts will fail in the real world where many providers face acute financial challenges and where costs are leading to consumer bankruptcies andincreased insurance costs. AHRQ understands that many of the root causes of inefficiencies that drive up costs are closely linked to root causes of inefficiencies that lead to poor quality, uncoordinated care where redundancies and system failures place patients at risk. Adding value within healthcare requires understanding the contribution that organizational culture makes to value and working to foster a culture where high value is a cultural norm. AHRQ’s development of the Hospital Survey on Patient Safety Culture (HSOPS) has contributed greatly to efforts to promote the important role culture plays in providing safe care. HSOPS is used extensively in national improvement campaigns and many hospitals and health systems now regularly assess their safety cultures and use culture scores on organizational dashboards and as parts of variable compensation programs.

If organizations lack cultures committed to value then discrete efforts to raise dimensions of value are likely to yield limited and unsustainable results. And if organizational leaders have no plausible way to know whether their organizational culture is committed to value, then their ability to make value a higher organizational priority will be very limited. Thus, developing value and efficiency survey instruments for hospitals and medical offices fills an important need for many ongoing and planned efforts to foster greater value within American health care.

Given the widespread impact of cost and waste in health care, AHRQ will develop the Value and Efficiency (VE) Surveys for hospitals and medical offices. These surveys will measure staff perceptions about what is important in their organization and what attitudes and behaviors related to value and efficiency are supported, rewarded, and expected. The surveys will help hospitals and medical offices to identify and discuss strengths and weaknesses within their individual organizations. They can then use that knowledge to develop appropriate action plans to improve their value and efficiency. To develop these tools AHRQ will recruit medical staff from 42 hospitals and 96 medical offices to participate in cognitive testing and pretesting.

In addition to the VE surveys, AHRQ also intends to develop a Communicating About Value Checklist (CV checklist). The objective of the CV checklist is to aid clinicians in having conversations with patients about value. Since the proper goal for any health care delivery system is to improve the value delivered to patients, such a tool will address the important aspects of health care that are of value to patients. Value in health care is typically measured in terms of the patient outcomes achieved per dollar expended. But a good outcome must be defined in terms of what is meaningful and valuable to the individual patient.

Better identification of patients' preferences is not only the right ethical thing to do but it also can reduce the cost of healthcare. Studies indicate that engaged and informed patients often choose to have less intensive care and become more careful about having lots of procedures.²^,³^,⁴ In addition, participatory decision making can reduce medication non-adherence which has been directly linked to increased morbidity, mortality and potentially avoidable healthcare costs totaling $290 billion annually in the U.S.⁵

The CV checklist will address three major topics: who should talk with patients about value issues (e.g., nurses, physicians, etc.), when should these conversations occur (e.g., when patients may incur costs, when they express financial concerns, etc.), and how can clinicians prepare for and effectively facilitate such discussions.

This research has the following goals:

1) Develop, cognitively test and modify as necessary the VE surveys (one for hospitals and one for medical offices);

2) Pretest the VE surveys in hospitals and medical offices and modify as necessary based on the results;

3) Develop, cognitively test and modify as necessary the CV checklist;

4) Pretest the CV checklist in hospitals and medical offices and modify as necessary based on the results;

5) Make the final VE surveys and CV checklist available for use by the public.

To achieve these goals the following activities and data collections will be implemented:

1) Cognitive interviews for the VE surveys. One round of interviews on the VE surveys will be conducted by telephone with 9 respondents from hospitals and 9 respondents from medical offices. The purpose of these interviews is to understand the cognitive processes the respondent engages in when answering a question on the VE survey and to refine the survey’s items and composites. These interviews will be conducted with a mix of senior leaders and clinical staff (i.e., unit/department managers, practitioners, nurses, technicians, and medical assistants) from hospitals and medical offices throughout the U.S. with varying characteristics (e.g., size, geographic location, type of medical office practice/hospital, and possibly extent of experience with waste-reduction efforts). The cognitive interview guides are included in Attachments A and B.

2) Pretest for the VE surveys. The surveys will be pretested with senior leaders and clinical staff from 42 hospitals and 96 medical offices (see Attachments C and D for draft questionnaires for hospitals and medical offices). The purpose of the pretest is to collect data for an assessment of the reliability and construct validity of the surveys’ items and composites, allowing for their further refinement. A site-level point-of-contact (POC) will be recruited in each medical office and hospital to manage the data collection at that organization (compile sample information, distribute surveys, promote survey response, etc.). Instructions for the site-level POCs are included in Attachments E and F. Exhibit 1 includes a burden estimate for the POC’s time to manage the data collection. The draft pretest survey cover letters and follow-up reminder notices are included in Attachments G and H.

3) Medical office information form. This form will be completed by the medical office manager in each of the 96 medical office pretest sites to provide background characteristics, such as type of specialty(s) and majority ownership (see Attachment I). A hospital information form will not be needed because characteristics on pretest hospitals will be obtained from the American Hospital Association’s (AHA) data set based on a hospital’s AHA ID number.

4) Survey to identify items for CV checklist. In order to identify items to put on the checklist, a survey will be developed and sent to 160 representative participants (40 Physicians, 40 Registered Nurses, 20 Social Workers, 20 Health Educators, and 40 Patients). The CV checklist survey is included in Attachment J.

Once the survey responses have been collected, responses will be analyzed to help inform the development of the CV checklist. Checklist items will be chosen based on what is learned. For example, if clinicians strongly believe that it is inappropriate to discuss costs and value with patients, the checklist may require different items than if clinicians recognize the importance of such conversations but believe they lack required information to facilitate them.

5) Cognitive Interviews for the CV checklist. Once checklist items have been identified, cognitive interviews will be conducted with 9 respondents in hospitals and 9 respondents in medical offices to understand the cognitive processes the respondent engages in when using the CV checklist. Cognitive interviewing will allow checklist developers to identify and classify difficulties respondents may have regarding checklist items. To get different perspectives, interviews will be conducted with a mix of physicians, nurses, social workers, health educators, and patients in hospitals and medical offices. The cognitive interview guide is included in Attachment K.

6) Pretest the CV checklist. The draft CV checklist (Attachment L) will then be pretested to solicit feedback from 50 physicians in hospitals and 50 physicians in medical offices. The pilot testing process will involve completion of a CV checklist evaluation (Attachment M) to help identify areas where users of the checklist have trouble understanding, learning, and using the checklist. It also provides the opportunity to identify issues that can prevent successful deployment of the checklist.

7) Dissemination activities. The final VE Surveys and CV checklist will be made available to the public through the AHRQ website. This activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.

This study is being conducted by AHRQ through its contractor, Health Research & Educational Trust (HRET), and subcontractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. How, by Whom, and for What Purpose Information Will Be Used

The information collected will be used to test and improve the draft survey items in the VE Surveys and CV checklist.

The final VE instruments will be made available to the public for use in hospitals and medical offices to assess value and efficiency from the perspectives of their staff. The survey can be used by hospitals and medical offices to identify areas for improvement. Researchers are also likely to use the surveys to assess the impact of hospitals’ and medical offices’ value and efficiency improvement initiatives.

The final CV checklist will be made available to hospital and medical office clinicians to aid in having conversations with patients about value.

3. Use of Improved Information Technology

Data collection will not involve the use of information technology. The pretests are designed to use a paper data collection modes because staff in medical offices and hospitals do not always have confidential access to the Internet.

4. Efforts to Avoid Duplication

VE surveys. We consulted hospital and medical office experts in value and efficiency and conducted a review of the literature, searching for staff surveys that measure value and efficiency in hospitals and medical offices. In reviewing the literature and speaking with experts, we have identified a number of composites that contribute to a high value and efficiency in hospitals and medical offices. These composites include: (1) focus on value and efficiency, (2) waste reduction, (3) senior leadership and department manager support for value/efficiency, (4) measurement and data monitoring, (5) organizational learning—continuous improvement/openness to change, (6) collaboration across positions and functional areas, (7) staff engagement and empowerment, (8) patient-centeredness, and (9) staff training. Listed below is information on seven surveys that measure some aspects of value, efficiency, waste reduction, and readiness for change. However, as noted below, there is a void for validated staff surveys that address all of the relevant composites for high value and efficiency in both hospitals and in medical offices.

The extent to which an organization is ready and open to change to becoming a high value and efficient organization is an important component to assess. There is a variety of readiness assessments that exist to help organizations determine their level of readiness for engaging in particular quality improvement (QI) projects. However, these tools are quite narrow to the specific QI effort. For example, AHRQ and the Department of Defense support the TeamSTEPPS^® project⁶ which stands for Team Strategies and Tools to Enhance Performance and Patient Safety. As a component of this QI project, an organization can implement a readiness assessment to see if their organization has the capacity to implement and sustain TeamSTEPPS^®. Another example of a readiness assessment was developed by Pare et al⁷. The questionnaire was designed to measure organizational readiness for clinical information systems. While some of the items seemed relevant to our purposes (e.g., capturing management support and the organization’s history of change), its purpose was to specifically assess readiness for mobile computing technology in home care organizations.

Another survey that helps organizations assess how they are performing and learn what should be improved or changed is the “Are We Making Progress?” survey from the Baldrige National Quality Program⁸. While this survey asks questions that are applicable to a high value and efficient culture, including concepts like leadership, a focus on the customer and management of measurement and analysis, this survey does not address efficiency, cost, or waste-reduction.

Noémi Imre⁹ and colleagues sought to develop a tool to measure the “leanness” of existing cultures. The items in the tool pertain to continuous improvement, long-term thinking, and emphasis on people. The researchers administered the survey to a limited number of staff (18) in a nonprofit outpatient clinic. This tool does not have items specific to cost, waste reduction, senior leadership support, and training, and had limited testing evidence.

Lawrence Miller developed a survey, “Lean Culture and Leadership Factors”¹⁰, to assess an organization’s progress toward becoming a lean organization. The survey is not specifically designed for hospitals or medical offices – it is general to various settings (e.g., manufacturing, marketing/sales, health care, and service organizations). While it focuses on leadership factors which are an important aspect of becoming a high value and efficient organization, it does not contain any items on cost, measurement, or a focus on patient-centeredness. Further, since the survey is targeted toward staff perceptions of leadership, it more narrowly asks for staff opinions about management.

Shwartz et al.¹¹ (2010) designed a survey to assess organization-level commitment to improving patient quality in respect to readiness for change and organizational support for the 6 Institute of Medicine aims (i.e., safe, effective, patient-centered, timely, efficient, and equitable health care). The survey also assessed perceptions about workforce development and culture, staff engagement, work environment/culture, and the success of quality improvement efforts. Although this survey covers many important components of value and efficiency, it does not address healthcare cost, attentiveness to waste reduction, or a focus on measurement and transparency of measures which the literature suggests are important domains of high value and efficient organizations.

In addition, the U.S. Office of Personnel Management (OPM) developed a tool that measures federal employees' perceptions of whether, and to what extent, conditions that characterize successful organizations are present in their agencies. We reviewed this survey, the Federal Employee Viewpoint Survey¹², because we thought that some items that help identify successful federal organizations could be transferrable to a successful (i.e., high value and efficient) hospital and medical office setting. The Federal Employee Viewpoint Survey asks respondents whether they feel encouraged to come up with new and better ways of doing things and whether they are held accountable for achieving results. These concepts are applicable since it is important that staff are engaged and encouraged to improve efficiencies and reduce waste (as found in the value and efficiency composite of staff engagement and empowerment). Additionally, the importance of accountability for results applies to the value and efficiency composite of measurement and data monitoring. Aside from those two composites, this survey did not cover other composites of value and efficiency and therefore, the new VE surveys would not duplicate these efforts.

In summary, we did not locate any validated surveys that assess all the key composites of value and efficiency as noted earlier for hospitals and medical offices from the perspective of all staff. The VE surveys will be developed to be used as either stand-alone surveys or to be used as supplemental item sets that can be added to the existing AHRQ Surveys on Patient Safety Culture (SOPS) or other surveys hospitals or medical offices would administer to staff.

CV checklist. A review of the literature indicates that shared decision making (SDM) is a process where a patient and clinician faced with more than one medically acceptable treatment option jointly decide which option is best based on costs, current evidence and the patient’s needs, preferences and values. Many believe shared decision making can help bridge the gap between the care patients want and the care they receive. At the same time, SDM may help constrain heath care spending by avoiding treatments that patients don’t want. However, barriers exist to wider use of shared decision making including lack of reimbursement for physicians to adopt SDM under the existing fee-for-service payment system that rewards higher service volume; insufficient information on how best to train clinicians to weigh evidence and discuss treatment options for preference-sensitive conditions with patients; low health literacy; and fears they will be denied needed care.

In some models, the SDM process has clearly deﬁned steps or competencies, which have been identiﬁed through focus groups or literature reviews.¹³^-¹⁴ ¹⁵ ¹⁶ In other models, the process is left to the discretion of the individual patient and clinician.¹⁷^,¹⁸ Of those models that deﬁne speciﬁc steps, many acknowledge the patient’s right to relinquish the decision to the clinician and proceed in a paternalistic model,¹⁹ several call for evidence-based presentations of information,²⁰^,²¹ a few call for physicians to express their preferences,²²^,²³ and a few call for an explicit check of patient understanding.²⁴ One model²⁵ acknowledges the practical limitations of medical practice and proposes a hierarchy of decision complexity, with more complex decisions requiring a greater intensity of interaction than simple decisions.

Systematic reviews of decision aids, which are based on the tenets of shared decision making and provide patient education and values clariﬁcation, have shown no consistent demonstrable effect on health outcomes.^27–32 This result might have been anticipated because decision aids help patients choose among alternatives in which the balance of beneﬁts and harms is a “close call,” thereby creating an environment in which the net health beneﬁt across a population of reasonable people who choose differently might approach zero. Only a few studies^33,34 have measured adherence to a chosen course of action; such measurements could give further insight into the effects of decision aids and shared decision making.

Several studies²⁶^,²⁷ offered possible explanations for mixed interest, including lack of a clear distinction between medical problem solving, which requires tasks for which patients are not qualiﬁed, and medical decision making, which could be shared. The authors also noted that patients may not realize that medicine is an inexact science, believing that their clinicians prescribed the only treatment available

Other reasons for lack of interest in participation include lack of understanding that there is a decision to be made, discomfort with a new role in decision making, inexperience with clinicians employing this approach to decisions, steadfastness in preconceptions about the course of care,²⁸ membership in an ethnic group that does not value patient autonomy,²⁹ and fear of regret for decisions that turn out badly.

Even when patients are interested in shared decision making, inability to understand medical concepts may limit their participation. For instance, multiple studies have demonstrated that some patients have difﬁculty understanding risk concepts,³⁰^,³¹ which calls into question their ability to accurately weigh the beneﬁts and harms of preventive services. Low functional literacy and numeracy, each of which affects approximately a quarter of the U.S. population,³² exacerbate these difﬁculties.

A hierarchical approach to shared decision making (i.e., using shared decision making for some, but not all, decisions), might be expected to maximize the cost-effectiveness of shared decision making for preventive services. Even when clinicians have the necessary time and ﬁnancial support for shared decision making, additional barriers exist. Many clinicians lack training in the interviewing techniques needed to engage patients in decision making.³³ The lack of accurate, organized scientiﬁc evidence about the beneﬁts and harms of many preventive services leads to confusion about the potential beneﬁts and harms of screening and chemoprevention services, making shared decision making more complex.

Patient-physician communication about cost when making treatment decisions has been promoted as a potential solution to the rising cost of oncologic care and suggested as an important component of high-quality oncologic care. In a study by Patel, et al.³⁴ (2009) they found only half of physicians asked about cost when prescribing new asthma medications for children and pediatricians were significantly less likely to ask than family physicians. Beard, et al.³⁵ (2010) found that despite medication changes being made in more than 50% of the visits, only 34% of those visits included discussions of medication-related costs; 48% of these discussions were initiated by patients. All of these studies will assist in constructing the checklist but overall, little is known regarding the perspectives of patients on such discussions with their physicians. Developing the checklist will provide further insight into the effects of decision aids and shared decision making. We believe more information is needed about what clinicians know about patient costs, what their attitudes are towards discussing value-related questions with their patients, and whether they have the communication skills required for such conversations.

5. Involvement of Small Businesses

Some of the medical offices participating in this pretest will be small businesses. The data collection instruments and procedures are designed to minimize burden on individual medical office staff respondents and will not have a significant impact on them.

6. Consequences if Information Collected Less Frequently

This effort is a one-time data collection.

7. Special Circumstances

The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on January 7^th, 2013 for 60 days, and again on April 18^th, 2013 for 30 days. No substantive comments were received. The Federal Register Notice is included as Attachment N.

8.b. Outside Consultations

A number of expert reviewers and consultants were consulted on the need for a value and efficiency survey for the hospital and medical office settings. They will also be consulted for review and comment on the draft surveys domains and items—see Attachment O for a list of those consulted both within and outside the Agency thus far. Most of these individuals comprise a formal Technical Expert Panel (TEP) and will provide periodic input throughout the project.

9. Payments/Gifts to Respondents

Cognitive Interview Respondents

VE surveys. To successfully recruit 18 cognitive interview participants, it is appropriate to offer a cash incentive. We propose a $175 cash remuneration for hospital administrators/executives and medical office physicians and a $100 cash remuneration for other clinical staff.

CV checklist. To successfully recruit 18 cognitive interview participants, it is appropriate to offer a cash incentive. We propose a $175 cash remuneration for physicians and a $100 cash remuneration for other respondents.

Pilot Study Respondents

VE surveys. We plan to recruit a site-level POC in each medical office who will manage data collection at that medical office (compile sample information, distribute surveys, promote survey response, etc.) and will provide $100 per medical office for their participation. For the hospital pretest, we do not propose site-level remuneration; rather, we think the benefit of receiving a complimentary feedback report on their survey results is sufficient incentive for participation.

CV survey. Patients will be recruited through Craigslist and offered a $5 incentive for completing the survey. Data collection will be a separate process from the VE surveys and POCs will manage the data collection for this part of the research.

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

Information that can directly identify the respondent, such as name and/or social security number will not be collected.

11. Questions of a Sensitive Nature

We do not believe there are questions of a particularly sensitive nature included in the survey, but if during cognitive testing sensitivities are discovered, such questions will be modified to ensure they are not of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the respondents’ time to participate in this research. Cognitive interviews for the Hospital VE survey will be conducted with 9 hospital staff (approximately 3 managers, 3 nurses, and 3 technicians) and will take about one hour and 30 minutes to complete. Cognitive interviews for the Medical Office VE survey will be conducted with 9 medical office staff (approximately 4 physicians and 5 medical assistants) and will take about one hour and 30 minutes to complete. The Hospital VE survey will be administered to about 4,032 individuals from 42 hospitals (about 96 surveys per hospital) and requires 15 minutes to complete. A site-level POC will spend approximately 16 hours administering the Hospital VE survey. The Medical Office VE survey will be administered to about 504 individuals from 96 medical offices (about 5 surveys per medical office) and requires 15 minutes to complete. A site-level POC will spend approximately 6 hours administering the Medical Office VE survey. The medical office information form survey will be completed by a medical office manager at each of the 96 medical offices participating in the pretest and takes 10 minutes to complete.

One-hundred and sixty individuals (40 physicians, 40 nurses, 20 social workers, 20 health educators, and 40 patients) will participate in the survey to identify items for the CV checklist and will take 15 minutes to complete. Cognitive interviews for the CV checklist will be conducted with 18 individuals (9 in hospitals and 9 in medical offices, consisting of approximately 4 physicians, 4 nurses, 2 social workers, 2 health educators, and 6 patients) and will take about one hour to complete. One hundred physicians will participate in the pretest of the CV checklist (50 in hospitals and 50 in medical offices). The total burden is estimated to be 2,534 hours annually.

Exhibit 2 shows the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated to be $115,559 annually.

Exhibit 1. Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Cognitive interviews for the Hospital VE survey	9	1	1.5	14
Cognitive interviews for the Medical Office VE survey	9	1	1.5	14
Pretest for the Hospital VE survey	4,032	1	15/60	1,008
Pretest for the Medical Office VE survey	504	1	15/60	126
POC Administration of the Hospital VE survey	42	1	16	672
POC Administration of the Medical Office VE survey	96	1	6	576
Medical office information form	96	1	10/60	16
Survey to identify items for CV checklist	160	1	15/60	40
Cognitive interviews for the CV checklist	18	1	1	18
Pretest for the CV checklist	100	1	30/60	50
Total	5,066	na	na	2,534

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Cognitive interviews for the Hospital VE survey	9	14	$36.16^a	$506
Cognitive interviews for the Medical Office VE survey	9	14	$46.87^b	$656
Pretest for the Hospital VE survey	4,032	1,008	$36.02^c	$36,308
Pretest for the Medical Office VE survey	504	126	$27.73^d	$3,494
Administration of the Hospital VE survey	42	672	$55.80^e	$37,498
Administration of the Medical Office VE survey	96	576	$50.98^f	$29,364
Medical office information form	96	16	$50.98^f	$816
Survey to identify items for CV checklist	160	40	$45.02^g	$1,801
Cognitive interviews for the CV checklist	18	18	$39.84^h	$717
Pretest for the CV checklist	100	50	$87.98ⁱ	$4,399
Total	5,066	2,534	na	$115,559

^a Based on the weighted average wages for 3 Registered Nurses (29-1111, $33.56), 3 Medical and Clinical Laboratory Technicians (29-2012, $19.11), and 3 General and Operational Managers (11-1021, $55.80) in the hospital setting;

^b Based on the weighted average wages for 4 Family and General Practitioners (29-1062; $87.18) and 5 Medical Assistants (31-9092, $14.63) in the medical office setting;

^c Based on the weighted average wages for 1,937 Registered Nurses, 1,131 Medical and Clinical Laboratory Technicians, 526 General and Operational Managers and 446 Physicians (29-1069; $66.23) in the hospital setting;

^d Based on the weighted average wages for 91 Family and General Practitioners and 413 Medical Assistants in the medical office setting;

^e Based on the average wages for General and Operational Managers in the hospital setting;

^f Based on the average wages for General and Operational Managers in the medical office setting;

^g Based on the weighted average wages for 40 Physician and Surgeons (29-10692; $88.78), 40 Registered Nurses (29-1111; $33.23), 20 Social Workers (21-1022; $24.28), 20 Health Educators (21-1091, $25.07), and 20 Patients (00-0000; $21.74);

^h Based on the weighted average wages for 4 Physician and Surgeons, 4 Registered Nurses, 2 Social Workers, 2 Health Educators, and 6 Patients;

ⁱ Based on the weighted average wages for 50 Physician and Surgeons in the hospital setting and 50 Family and General Practitioners in the medical office setting;

* National Occupational Employment and Wage Estimates in the United States, May 2011, “U.S. Department of Labor, Bureau of Labor Statistics” (available at http://www.bls.gov/oes/current/naics4_621100.htm [for medical office setting] and http://www.bls.gov/oes/current/naics4_622100.htm [for hospital setting]).

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. The only cost to the respondent will be that associated with their time to respond to the information collection, as shown in Exhibits 1 through 5.

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost to the government for this data collection. Although data collection will last for less than one year, the entire project will take about 2 years. The total cost for the three surveys is approximately is $1,001,202.

Exhibit 3. Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$273,838	$136,919
Data Collection Activities	$153,119	$76,560
Data Processing and Analysis	$171,764	$85,882
Publication of Results	$14,753	$7,377
Project Management	$10,032	$5,016
Overhead	$377,696	$188,848
Total	$1,001,202	$500,601

15. Change in Burden

This is a new activity.

16. Time Schedule, Publication and Analysis Plans

As soon as OMB approval is received, pilot study activities will begin. The estimated time schedule to conduct these activities is shown below:

One round of cognitive interviews (2 months)
Pilot study data collection (4 months)
Data analysis, feedback report production, and development of technical report (4.5 months)
Final survey and development of toolkit materials (2 months)

The final versions of the Hospital VE Survey, the Medical Office VE Survey, and the CV Checklist will be made available in the public domain on the AHRQ Web site.

VE surveys. The goal of our data analysis will be to assess the multi-level psychometric properties of the VE surveys using a multi-staged approach to assess construct validity through factor analyses and intercorrelations among composites and to assess reliability. Psychometric analysis will be conducted on the pilot data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items included in the surveys. Because the survey items are being developed to measure specific aspects of value and efficiency in the hospital and medical office settings, the factor structure of the survey items will be evaluated through multilevel confirmatory factor analysis. On the basis of the data analyses, items or factors may be dropped.

Descriptive Statistics

The means, standard deviations, and response frequencies for the survey items will be examined to ensure that respondents and hospitals/medical offices exhibit adequate response variability on the survey items. In addition, items will be examined to ensure that there are low rates of missing data (lower than 8% missing response per item). Poorly functioning items will be identified.

Individual Level Factor Analysis

An individual level factor analysis will be conducted to initially examine whether groups of items intended to measure a specific composite are interrelated, ignoring the nesting of respondent data within hospitals and within medical offices. Factor loadings for each item in an a priori composite will be considered as having an adequate contribution to a particular composite or factor if the strength of the item's relationship to that factor (i.e., its factor loading), is 0.40 or greater. Potential composites or factors that the survey items may measure include:

senior leadership and management support for value, efficiency and waste reduction;
waste reduction;
measurement and data monitoring;
openness to change and continuous improvement;
collaboration across positions and functional areas;
staff engagement and empowerment;
patient-centeredness;
staff training;
priority on value and efficiency;
experience and involvement with quality or process improvement activities.

The percent of variance accounted for by the factor will also be examined. The more variance that is accounted for by a factor, the more justifiable it is to combine the items into a single composite score. At least 50% of the variance should be accounted for by the items in a composite.

Intraclass Correlations (ICCs) and Design Effects

Intraclass correlations (ICCs) will be computed for each composite. ICC's determine if substantial variation exists between groups compared to variation within groups. ICCs above 0.05 or 5% indicate that the between group variance is greater than expected by chance and imply that nesting in groups does have an effect on the responses of individuals.

Given that ICCs are likely to be inflated when there are many groups with few individuals within the groups (compared to few groups with many individuals within the groups), we will also examine design effects, which take into account within-group sample size. A design effect of 2 or more implies that group membership or nesting of individuals within groups does have an effect on the responses of the individuals and therefore multilevel modelling should be conducted to account for the multilevel nature of the data.

Multilevel Confirmatory Factor Analysis (MCFA)

Individuals responding to the surveys are located within hospitals and within medical offices. When data are nested in groups like this, results from an individual-level factor analysis may be biased or incorrect. Therefore, multilevel confirmatory factor analysis will be conducted on the surveys’ a priori composites to examine the structure of the factors at the hospital level of analysis and medical office level of analysis, taking into consideration that the data are nested.

An MCFA will be conducted to test the fit of the measurement model for the survey’s composites, taking into consideration the nested nature of the data at the hospital level of analysis and medical office level of analysis. We will first evaluate the MCFA results by examining the item factor loadings on the composites. Factor loadings should be 0.40 or greater.

We will also examine overall model fit indices using standard fit statistics: the chi-square, comparative fit index (CFI), and the standardized root mean square residual (SRMR). For chi-square statistics, lower and non-significant chi-squares indicate good fit. The factor structure is determined to adequately fit the data if the CFI is at least 0.90. A value of zero for the SRMR indicates perfect fit, but a value less than 0.08 is considered a good fit.

Reliability Analysis

Reliability analyses will then be performed on the composites to examine whether individuals responded consistently to the items within each composite. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least 0.70.

Intercorrelations

Intercorrelations among the survey’s composites will also be examined. Intercorrelations will be explored at the individual and medical office or hospital levels of analysis. While the composites should be correlated since they measure aspects of value and efficiency, the intercorrelations should not be extremely high (0.90 or higher) because very high intercorrelations indicate that the composites may not be unique enough to be considered separate constructs or measures. While there is no steadfast criterion about the magnitude of dimension intercorrelations and construct validity, in general, such correlations should be less than 0.80 for the composites to be considered unique and avoid problems with multicollinearity.

The above analyses will be used to determine which items and composites are poorer functioning and remove them from the surveys to derive a final set of items and composites with good psychometric properties and reduce the overall length of the final survey. The Technical Expert Panel will be informed of the data analysis results and be allowed to weigh in on which items to retain or drop when the psychometric results do not provide enough guidance and decisions can be made based on the content value of the items alone.

The final surveys will be made available in the public domain for use by hospitals, medical offices, and researchers to assess value and efficiency.

CV checklist.

Reliability Analysis

Reliability analyses will then be performed on the checklist to examine whether individuals responded consistently to the items within each composite. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least 0.70.

Validity

Two separate investigations will be conducted to empirically validate the checklist. The first investigation will be a nonexperimental study that seeks input from subject matter experts and targeted users regarding the merit of the checklist. The second investigation will be a real-world evaluation to assess the effectiveness of the checklist. The methods applied to this study included qualitative and quantitative survey research and case study research. The use of a mixed method approach was deemed important to achieve the primary 6 benefits of development and expansion of the checklist (Greene, Caracelli, & Graham, 1989). The use of an expert panel to provide critical feedback served to facilitate the development and refinement of the checklist prior to application in the second investigation. In addition, the expert panel is considered an integral design element to demonstrate content relevance and representativeness along with the substantive and consequential aspects of validity. The case study research method facilitates learning about the practicality of the checklist in its contextual setting thereby extending the scope and breadth of the validity study (Yin, 2009). As with all evaluation checklists, a number of years of field-testing and use are required to thoroughly validate and refine the tool. Repeated use of a checklist is the preferred method for validation.

17. Exemption for Display of Expiration Date

No exemption is being requested.

List of Attachments

Attachment A – Cognitive Interview Guide for Hospital VE Survey

Attachment B – Cognitive Interview Guide for Medical Office VE Survey

Attachment C – Pretest Survey for Hospital VE Survey

Attachment D – Pretest Survey for Medical Office VE Survey

Attachment E – Site-level POC Instructions for Hospital VE Survey

Attachment F – Site-level POC Instructions for Medical Office VE Survey

Attachment G – Pretest Cover Letter and Reminder Notice for Hospital VE Survey

Attachment H – Pretest Cover Letter and Reminder Notice for Medical Office VE Survey

Attachment I – Medical Office Information Form

Attachment J – CV Checklist Survey

Attachment K – Cognitive Interview Guide for CV Checklist

Attachment L – Draft CV Checklist

Attachment M – Draft CV Checklist Evaluation

Attachment N – Federal Register Notice

Attachment O – List of Expert Reviewers and Consultants

1 Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. The Institute of Medicine: Washington, DC, September 6, 2012.

2 Dartmouth Atlas of Health Care (2012). ‘Understanding of the efficiency and effectiveness of the health care system’. Dartmouth Atlas website. Available at:www.dartmouthatlas.org (accessed on 30 April 2012).

3 Kennedy A, Sculpher M, Coulter A, Dwyer N, Rees M, Abrams K, Horsley S, Cowley D, Kidson C, Kirwin C, Naish C, Stirrat G (2002). ‘Effects of decision aids controlled trial’. Journal of the American Medical Association, vol 288, no 21, pp 2701–8.

4 Deyo RA, Cherkin DC, Weinstein J, Howe J, Ciol M, Mulley AG, Jr (2000). ‘Involving patients in clinical decisions: impact of an interactive video program on use of back surgery’. Medical Care, vol 38, no 9, pp 959–69.

5 New England Healthcare Institute, Thinking Outside the Pillbox, Retrieved April 28, 2011

from: http://www.nehi.net/news/press_releases/110/nehi_research_shows_patient_

medication_nonadherence_costs_health_care_system_290_billion_annually

6 AHRQ. Is Your Organization Ready for TeamSTEPPS? Available at: http://teamstepps.ahrq.gov/readiness/.

7 Pare G, Sicotte C, Pobe-Nzaou P, and Balouzakis G. Clinicians’ perceptions of organizational readiness for change in the context of clinical information system projects. Implementation Science 2011. 6(5); 1-14.

8 National Institute of Standards and Technology. Are we making progress? Available at: http://www.quality.nist.gov/PDF_files/Progress.pdf

9 Imre N, Jenei I, Losonci D. What is lean culture – and how to measure it? 18th International Annual EurOMA Conference, 3-5 July, 2011, Cambridge, United Kingdom. Available at: www.uni-corvinus.hu/index.php?id=41618&type=p&file_id=264

10 Miller LM. Lean culture and leadership factors: A survey of lean implementers’ perceptions of execution and importance. 2011. Available at: http://www.lmmiller.com/blog/wp-content/uploads/2011/06/Report-Lean-Culture-and-Leadership-Factors4.pdf

11 Shwartz M, Cramer IE, Holmes SK, et al. Survey-assessed quality and organizational factors related to quality in pursuing perfection hospitals. Q Manage Health Care 2010. 19(4); 349–363.

12 U.S. Office of Personnel Management (OPM). Federal Employee Viewpoint Survey. 2011 Results available at: http://www.fedview.opm.gov/2011/Reports/

13 Elwyn G, Edwards A, Kinnersley P, Grol R. Shared decision making and the concept of equipoise: the competences of involving patients in healthcare choices. Br J Gen Pract 2000;50:892–9.

14 Braddock CH 3rd, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. JAMA 1999;282:2313–20.

15 Entwistle VA, Sheldon TA, Sowden A, Watt IS. Evidence-informed patient choice. Practical issues of involving patients in decisions about health care technologies. Int J Technol Assess Health Care 1998;14:212–25.

16 Mead N, Bower P. Patient-centredness: a conceptual framework and review of the empirical literature. Soc Sci Med 2000;51:1087–110.

17 Charles C, Gafni A, Whelan T. Decision-making in the physician–patient encounter: revisiting the shared treatment decision-making model. Soc Sci Med 1999;49:651–61.

18 Emanuel EJ, Emanuel LL. Four models of the physician–patient relationship. JAMA 1992;267:2221–6.

19 Holmes-Rovner M, Llewellyn-Thomas H, Entwistle V, Coulter A, O’Connor A, Rovner DR. Patient choice modules for summaries of clinical effectiveness: a proposal. BMJ 2001;322:664–7.

20 Walter LC, Covinsky KE. Cancer screening in elderly patients: a framework for individualized decision making. JAMA 2001;285:2750–6.

21 Entwistle VA, Sheldon TA, Sowden A, Watt IS. Evidence-informed patient choice. Practical issues of involving patients in decisions about health care technologies. Int J Technol Assess Health Care 1998;14:212–25.

22 Stewart M, Brown JB, Donner A, et al. The impact of patient-centered care on outcomes. J Fam Pract 2000;49:796–804.

23 Stewart M. Towards a global definition of patient centred care. BMJ 2001;322:444–5

24 Elwyn, et. al. op cit.

25 Braddock CH 3rd, Edwards KA, Hasenberg NM, Laidley TL, Levinson W. Informed decision making in outpatient practice: time to get back to basics. JAMA 1999;282:2313–20.

26 Deber RB, Kraetschmer N, Irvine J. What role do patients wish to play in treatment decision making? Arch Intern Med 1996;156:1414–20.

27 Deber RB. Physicians in health care management: 7. The patient–physician partnership: changing roles and the desire for information. CMAJ 1994; 151:171–6.

28 Deber, et. al. 1996, op. cit

29 Blackhall LJ, Murphy ST, Frank G, Michel V, Azen S. Ethnicity and attitudes toward patient autonomy. JAMA 1995;274:820–5.

30 Chase GA, Faden RR, Holtzman NA, et al. Assessment of risk by pregnant women: implications for genetic counseling and education. Soc Biol 1986;33:57–64.

31 Feldman-Stewart D, Kocovski N, McConnell BA, Brundage MD, Mackillop WJ. Perception of quantitative information for treatment decisions. Med Decis Making 2000;20:228–38.

32 Kirsch I, Jungeblut A, Jenkins L, Kolstad A. Adult literacy in America: a first look at the findings of the National Adult Literacy Survey. Washington DC: National Center for Education Statistics, U.S. Department of Education, 1993.

33 Kaplan SH, Greenfield S, Gandek B, Rogers WH, Ware JE Jr. Characteristics

of physicians with participatory decision-making styles. Ann Intern Med 1996;124:497–504.

34 Patel MR, Coffman JM, Tseng CW, Clark NM, Cabana MD. Physician communication regarding cost when prescribing asthma medication to children. Clin Pediatr (Phila). 2009 Jun;48(5):493-8. Epub 2009 Jan 21.

35 Beard AJ, Sleath B, Blalock SJ, Roth M, Weinberger M, Tudor G, Chewning B. Predictors of rheumatoid arthritis patient-physician communication about medication costs during visits to rheumatologists. Arthritis Care Res (Hoboken). 2010 May;62(5):632-9.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Laura Gray
File Modified	0000-00-00
File Created	2021-01-29

Supporting Statement Part A -- Pilot Test of the Proposed Value and Efficiency Surveys and Checklist 1-10-2013