OMB passbacks

OMB passback 3/10/09

Question: In the supporting statement, it says that the selection criteria will depend on AHRQ’s priorities, including identifying innovations that reduce disparities, improve value of health care, and have been supported by AHRQ.

  

Will these priorities be included in the instructions applicants receive?

Response: Yes.

 

Question: Also, how is AHRQ evaluating whether the innovation improves the value of health care?

Response: AHRQ will judge whether an innovation improves the value of health care using information, knowledge, and resources related to efforts to support AHRQ’s Value Portfolio of research, http://www.ahrq.gov/fund/portfolio.htm; in its simplest form, value is a reduction of unnecessary costs (waste) while maintaining or improving quality.

 

 

Question: And why are the AHRQ-supported projects being given priority?

Response: AHRQ-supported projects are being given a priority for three major reasons: (1) while AHRQ funds innovative health service delivery research, it does not have a database and web site in which to capture and publicly share, respectively, context-specific elements related to that work – AHRQ sees the Health Care Innovations Exchange as a window through which health professionals can better see the innovative work AHRQ has supported; (2) there will be more innovative service delivery changes with a higher level of evidence (and thus more certainty about effect) in AHRQ-supported projects; and (3) the funding available for the Health Care Innovations Exchange requires that priorities be set – the Innovations Exchange project will not be funded to be a high production resource; we will attempt to include all innovations that meet inclusion criteria but if we have more submissions than we are able to process then focusing on AHRQ-funded innovative service delivery changes will assure that those will consistently be represented.

 

 

 

OMB passback 3/4/09

Question: please send us a copy of the inclusion/exclusion criteria.

INCLUSION/EXCLUSION CRITERIA

See attachment

Question: Also, please clarify who the main users/target audience of this database is anticipated to be.

MAIN USERS/TARGET AUDIENCE

The main users of this database will be physicians, nurses and other professionals working on the front-line of quality improvement (QI) – they will, through their own QI efforts, realize a problem exists in their care delivery and search or browse the database for possible solutions to that problem.

 

 

Question: Can you clarify also that in this instance for this particular ICR, CDC was not directly consulted?

AHRQ has discussed the Healthcare Innovations Project with the following CDC staff:

 

Lynda A. Anderson, PhD
Director, Healthy Aging Program
Division of Adult and Community Health (MS K45)
National Center for Chronic Disease Prevention and Health Promotion
Coordinating Center for Health Promotion
Centers for Disease Control and Prevention

 

Joe Boone, PhD (now retired)

Acting Director, Division of Laboratory Systems

Julie Taylor, PhD

Acting Associate Director, Division of Laboratory Systems

Susan Snyder, PhD, MBA

Laboratory Practice Evaluation & Genomics Branch

Division of Laboratory Systems

National Center for Preparedness, Detection, and Control of Infectious Diseases

Centers for Disease Control and Prevention

 

 

Shawna Mercer, PhD

Branch Chief of Community Guide

Center for Health Information and Service

National Center for Health Marketing

Division of Health Communication and Marketing

Centers for Disease Control and Prevention

 

 

David Sleet, PhD

Associate Director for Science

Division of Unintentional Injury Prevention

National Center for Injury Prevention and Control

Coordinating Center for Environmental Health and Injury Prevention

Centers for Disease Control and Prevention

 

 

 

OMB passback 2/27/09

 

QUESTION/CONCERN 1

The ICR is not adequately focused to have utility. “Innovations that relate directly or indirectly to patient care” is huge, and that gives us concern for 2 reasons. First, the broader the scope, the greater the risk of duplication. Also, it’s not clear how to assess how useful or effective this “one stop shop” will be to the public with such a broad scope. Our recommendation here is for AHRQ to think more about how to carve out a niche for what it is doing here. It could be that AHRQ starts out focusing on something specific (e.g. interventions aimed at better treatment compliance that have not been published and have worked in rural hospital settings among a diabetic patient population) and later expands to other areas. But we are not comfortable signing off on something that is this broad and potentially duplicative. 

 

 

RESPONSE 1

To clarify the purpose of this collection and better define AHRQ's niche, we have reworded the inclusion criteria that that currently reads “innovations that focus directly or indirectly on patient care” to “innovations that focus on how health care services are delivered to patients.”  We believe this narrows the focus considerably and distinguishes it from other activities across HHS.

 

 

QUESTION/CONCERN 2 

Did AHRQ consult with CDC? Also, did AHRQ consult with anyone at HRSA’s headquarters? (Looks like the HRSA person listed on the previous response is someone at a regional office who may not be familiar with the full scope of what HRSA does in the area of health disparities). 

 

RESPONSE 2

AHRQ consulted with senior leaders in HRSA’s Headquarter Office of Performance Review, Becky Sptizo and Dennis Malcomson. They came to AHRQ with Rick Wilk (from the Chicago office) expressing interest in helping us because they didn’t have an innovations database and could see the value in having one; they did not want to duplicate our efforts. All parties agreed at that time that the best help HRSA could give AHRQ was to identify innovators and encourage them to submit their work to the Innovations Exchange. Our connection to Rick Wilk has yielded a number of innovative community-level health service delivery changes that we would like to pursue.

 

AHRQ consults with CDC on a regular and on-going basis. The types of innovations to be included In the Innovations Exchange would not be those the CDC might develop or for which they’d support development. For example, to reduce tobacco use nationally, the CDC supported innovative policy development related to no-smoking bans in public places; this would not be appropriate for inclusion in the Innovations Exchange. Innovative public policies are NOT the types of innovations AHRQ would seek. 

 

To provide clarity about the types of innovations of interest to AHRQ, we will create an inclusion/exclusion list that could then serve as a quick reference guide to innovation submitters and the public.

 

 

QUESTION/CONCERN 3

It’s also not clear how useful information about innovation failures will be for this particular audience. We understand the basic distinction between efficacy and effectiveness and how things that worked in a lab setting might not work out so well in a real-life setting. We also understand the difference between robust “RCT”-like research and translational research. But it seems odd to include as part of the “innovations database” things that are neither effective nor efficacious. If I were a health care provider interested in implementing a quality improvement project at my hospital, I would want a list of things that clearly worked, as well as information about why they worked (e.g. the setting was important, the patient population was important, the provider mix, etc.). I wouldn’t find it particularly useful to see a list of things that didn’t work. Though I can see from a researcher’s point of view why that may be of interest, it seems like you are targeting a difference audience.

 

 

RESPONSE 3

AHRQ believes that there are both good scientific and practical reasons to encourage reports of "negative results" in this kind of a database.  First, negative results are often underreported in the professional literature.  Second, we have heard from a number of our stakeholders that familiarity with failed experiments helps them design better innovations.  A sampling what we have heard is below.

 

 

Healthcare and Quality Improvement Experts, Clinicians and Administrators Support Learning from Failure

 

Through various venues, especially those designed for the purpose of vetting ideas about the Innovations Exchange, AHRQ stakeholders have repeatedly stressed the importance of including both successes and failures in the Innovations Exchange.  Included below are some of their specific comments and recommendations on including Innovation Attempts (or failures) in the Health Care Innovations Exchange website.

         Greg Pawlson, MD, MPH, FACP, Executive Vice-President of the National Committee for Quality Assurance: There are two types of “failures” which could be included in the Healthcare Innovations Exchange. One relates to good innovations that run into hurdles or roadblocks that have to be overcome. The second kind of failure is an innovation that failed because it is not a good idea. The first type of failure belongs in the database and can be a standard part of any good innovation included. But it is not clear whether bad ideas should be included in the Exchange.

         Susan Edgman-Levitan, PA, Executive Director of the John D Stoeckle Center for Primary Care Innovation:  People like to read about programs that did not work as well as good ideas because unsuccessful ideas often provide the seeds for an idea that could ultimately be successful.

         Robert L. Ferrer , MD, MPH, Associate Professor of Family and Community Medicine, University of Texas Health Sciences Center at San Antonio: Reading about failures is often more interesting than reading about successes. In fact, people gravitate to failure examples; for example, the case study of the Colombia Space Shuttle disaster is widely read.

         Steve Shields, President and CEO of Meadowlark Hills Continuing Care Retirement Community: Innovators must clearly understand that failure is an integral, intrinsic part of the pathway to success.

         Christine G. Williams, M.Ed.: If something did not work, where it failed should be examined.  Factors that led to failure could be tweaked to lead to success.  Sometimes something that did not work in one system can be implemented in another, but often ideas do not spread beyond the system in which they originated.

         Donald Casey, Jr., M.D., M.P.H., M.B.A: .  Just because a large number of people failed at an innovation does not mean it is a failure.  For example, attempts to improve hospital chronic care fail frequently, because people do not want to make the necessary changes. 

         Conversations at other AHRQ-sponsored meetings with various clinicians and administrators: each was able to regale AHRQ staff with examples of waste they incurred in trying something new only to find out someone else had tried that same thing and got the same negative result – each noted that if there was a repository of such efforts they could have been spared their loss because they would have known what not to do.

 

 

Related to the last bullet, AHRQ’s patient safety work supports efforts to learn from failed innovations. Failure Mode and Effects Analysis (FMEA), which focuses on identifying and controlling for failure is a basic tenet of the health care and quality improvement field. The FMEA technique and quality method enables the identification and prevention of process errors. Within healthcare, the goal is to avoid adverse events that could potentially cause harm to patients, families, employees or others in the patient care setting. Another important tenet of healthcare quality improvement is learning from adverse events. Adverse event reporting systems within hospitals have now become commonplace as these instances of failure are considered valuable opportunities for improvement.  Similarly, learning from innovation failure can help potential innovators and adopters improve and refine their approaches to achieve greater success.

 

 

 

QUESTION/CONCERN 4

We really like the fact that AHRQ plans to provide an “evidence rating.” We would recommend that the evidence rating appear more prominently than it currently does (e.g. it is currently at the bottom of the “What they did” section. It should at least be at the top of that section or in the very first section where the innovation is summarized.

 

RESPONSE 4

AHRQ will place the evidence rating in the first section where the innovation is summarized. In addition, because limited user testing (n=8, 5 of which were AHRQ staff) did not indicate issues with the placement of the evidence rating in the results section, AHRQ will continue to include it there. Having the evidence rating up front and with the results will show AHRQ’s focus on the importance of considering evidence.

 

QUESTION/CONCERN 5

Continuing on the topic of the evidence rating, we do not understand the utility of including innovations that are “unproven,” as AHRQ has defined it. We would suggest that AHRQ limit the innovations that are included in this database to interventions whose evidence rating is at least “suggestive.” We would also suggest that AHRQ delete anecdotal reports as evidence that would count as “suggestive.”

 

 

RESPONSE 5

AHRQ will restrict the database to entries that are at least suggestive.  

 

 

QUESTION/CONCERN 6

We would suggest that AHRQ better define for the public what it would consider “truly innovative.” What AHRQ included in the response to OMB’s request for clarification is helpful, but we think the public would appreciate a bit more clarification.

 

 

RESPONSE 6

AHRQ believes the revised wording is appropriate but would be open for a suggested rewording from OMB.