Supporting Statement Part A_Version3_AHRQ

This package requests clearance to conduct an Expert Survey, Stakeholder Survey, and key informant interviews to support an evaluation of the AHRQ Healthcare Horizon Scanning System. The AHRQ Healthcare Horizon Scanning System alerts stakeholders (e.g., consumers, clinical researchers, health plans, health systems, government agencies, developers of new interventions) in the public and private sectors to emerging technologies and new ways of providing and managing patient care. The goal of the AHRQ Healthcare Horizon Scanning System is to enable more informed strategic planning for comparative effectiveness research (CER) priorities and for use by public and private decision makers considering new technology adoption and implementation. The evaluation is being conducted by AHRQ through its contractor, ECRI Institute; it is being implemented by Mathematica Policy Research.

The objective of the evaluation is to determine how effectively the AHRQ Healthcare Horizon Scanning System has implemented the key functions of identifying, monitoring, and assessing the potential for high impact of emerging technologies. The evaluation will also identify areas of improvement for the system.

The evaluation will analyze the accuracy, completeness, and usefulness of the Potential High Impact reports issued by the AHRQ Healthcare Horizon Scanning System. These reports discuss up to 20 individual interventions in each of the 14 AHRQ Priority Conditions deemed by expert comment processes to have potential for high impact. Primary data collection is necessary to obtain the data for the measures to evaluate the reports. Mathematica will ask experts to view a report and complete an online survey to rate the accuracy and completeness of the reports, and to provide their assessment of the potential for high impact for a set of interventions identified by the AHRQ Healthcare Horizon Scanning System. Cases of inaccurate or missing information reported by a sole expert in the Expert Survey will be confirmed through consultation with another expert. Mathematica will ask stakeholders to complete an online survey which will link them to a report and ask them to rate the relevance, clarity, and usefulness of the report. Mathematica will conduct semi-structured interviews with the AHRQ Healthcare Horizon Scanning System staff to learn about areas and suggestions for improvement in the identification, monitoring, and impact assessment processes. The evaluation will also include reviews of the reports by the evaluator to identify whether there were cases when an intervention should have been identified by the system earlier and to describe the variability in potential impact assessments over time. The review of reports will not involve primary data collection and therefore, are not discussed further in this clearance package.

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

1.a. Statement of Need for Evaluation of the AHRQ Healthcare Horizon Scanning System

The American Recovery and Reinvestment Act (ARRA) appropriated $1.1 billion for comparative effectiveness research (CER), of which $300 million was made available to the Agency for Healthcare Research and Quality (AHRQ). The goal of CER is to improve patient outcomes by providing clinicians and patients the information they need to choose between the preventive, diagnostic, treatment, or other healthcare options to identify the options that best fit an individual patient's needs and preferences. AHRQ has used Recovery Act funds to expand the activities of its CER initiative, the Effective Health Care Program (EHC). The EHC Program was created in response to Section 1013 of the Medicare Prescription Drug, Improvement, and Modernization Act (MMA) of 2003.

To better inform comparative effectiveness research investments at the EHC program, AHRQ used some of the ARRA funds to develop a horizon scanning system to identify and monitor emerging health care technologies and innovations. While horizon scanning systems exist in other countries, these systems do not take into account the unique political, regulatory, cultural, and economic context of the U.S. healthcare system. To meet this need, the AHRQ Healthcare Horizon Scanning System was implemented in November 2010. The AHRQ Healthcare Horizon Scanning System provides a systematic process to identify and monitor target technologies and innovations in health care and to create an inventory of target technologies that have the highest potential for impact on clinical care, the health care system, patient outcomes, and costs. It is also a tool for the public to identify and find information on new health care technologies and interventions. Additionally, the AHRQ Healthcare Horizon Scanning System serves as a resource for those involved in decision making about adoption, implementation, and coverage of new healthcare interventions.

To fulfill its purpose, the AHRQ Healthcare Horizon Scanning System performs three functions: (1) identification and prioritization of interventions in late phase development for tracking and monitoring; (2) monitoring of target interventions through the development of detailed information on interventions in late phase development; and (3) assessment of potential impact of target interventions through the gathering and synthesizing the perspectives of experts from various areas of the health care community about the potential impact those target interventions may have on the health care system, clinical care, patient outcomes, and health care costs.

As the first and only U.S. horizon scanning system, it is important to understand whether the AHRQ Healthcare Horizon Scanning System is implementing its functions effectively. This evaluation is also essential to determining whether the AHRQ Healthcare Horizon Scanning System is meeting the needs of patients, clinicians, private industry, and policymakers and how it can be improved to better meet those needs. The evaluation will address the following research questions:

How successfully did the AHRQ Healthcare Horizon Scanning System identify and prioritize interventions for monitoring?
How successfully did the AHRQ Healthcare Horizon Scanning System monitor the selected target interventions?
How accurately did the AHRQ Healthcare Horizon Scanning System assess the potential impact of the interventions?
How can the processes for identification, prioritization, monitoring, and assessment of potential impact of the interventions be improved?

This research has the following goals:

To assess the performance of the AHRQ Healthcare Horizon Scanning System in the identification and prioritization of interventions that are important topics for further assessment.
To assess the performance of the AHRQ Healthcare Horizon Scanning System in terms of the quality of information provided on the topics selected, and the accuracy of the assessment of potential impact.
To identify which, if any, of these areas of performance may require improvement so as to strengthen the effectiveness of the AHRQ Healthcare Horizon Scanning System.

To achieve the goals of this project the following data collections will be implemented:

Expert Survey – The purpose of this survey, completed by domain experts, is to measure the accuracy and completeness of the AHRQ Healthcare Horizon Scanning System Potential High Impact reports and to collect their assessment of the potential for high impact for the Potential High Impact interventions.
Expert Consultation – The purpose of this consultation with experts is to confirm the cases of inaccurate or missing information identified by a sole expert in the Expert Survey.
Stakeholder Survey – The purpose of this survey, completed by stakeholders and likely users of the reports issued by the AHRQ Healthcare Horizon Scanning System, is to rate the relevance, clarity, and usefulness of the Potential High Impact reports.
Key Informant Interview – The purpose of these interviews of the AHRQ Healthcare Horizon Scanning System staff is to learn about areas and suggestions for improvement in the identification, monitoring, and impact assessment processes.

This evaluation is being conducted by AHRQ through its contractor, ECRI Institute, and ECRI’s subcontractor, Mathematica Policy Research, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2) (Attachment A).

2. Purpose and Use of Information

Information will be collected and analyzed for this evaluation by Mathematica Policy Research with ECRI Institute, titled Subcontract Agreement for AHRQ Horizon Scanning System HHSA29020100006C.

The data collected by the Expert Survey (Attachment B) will be used to measure the accuracy and completeness of the Potential High Impact reports and the accuracy of the potential for high impact assessments. If the expert survey identifies cases of inaccurate or missing information that are not reported by multiple experts, we will conduct an Expert Consultation with another expert to confirm these cases (Attachment C). Accuracy of the potential for high impact assessments will be measured by the level of sensitivity (if experts agree that the Potential High Impact interventions identified by the AHRQ Healthcare Horizon Scanning System are high impact interventions) and specificity (if experts agree that the No Potential High Impact interventions identified by the AHRQ Healthcare Horizon Scanning System should be excluded from the group of Potential High Impact interventions).

The Stakeholder Survey (Attachment D) will collect data to measure the usability of the Potential High Impact reports and the specific report sections that include the potential high impact assessment, summary, and synthesis of expert comments. These data will be used to inform the improvement of the format and content of the report. The survey will also collect information on the sources and media these stakeholders use to find CER information to help AHRQ better target distribution of these reports to stakeholders.

A series of semi-structured Key Informant Interviews (Attachment E) will be conducted with staff and domain experts at ECRI Institute and other organizations that participate in the AHRQ Healthcare Horizon Scanning System in order to identify opportunities for improvements to the AHRQ Healthcare Horizon Scanning System process. Qualitative interviews are the main vehicle for gathering data to (1) learn which elements of the AHRQ Healthcare Horizon Scanning System Protocol are working well and the reasons why they are working well; and (2) understand which elements of the AHRQ Healthcare Horizon Scanning System Protocol can be improved, how they might be improved, and the relative importance of suggested improvements.

All of these information collection activities will allow for an evaluation of the AHRQ Healthcare Horizon Scanning System, thereby creating the opportunity to both maintain and improve this important national resource. The findings will be presented in a report to ECRI Institute and AHRQ.

3. Use of Improved Information Technology

The evaluation will use a web survey for the Expert Survey and Stakeholder Survey to collect data from the experts and stakeholders. The web survey will allow us to obtain reliable information in an efficient way that minimizes respondent burden. With the web survey, the respondent can choose to respond at any time and in multiple sessions. The web format will allow us to program skips to reduce respondent burden and maximize data quality. The data will be automatically stored in a secure database, eliminating the cost and potential for error in data entering the survey responses from paper surveys. The web survey will also provide a link to the reports, so these reports are easy to access.

We will conduct the Expert Consultation on inaccurate or missing report information by email or telephone, based on the preference of the expert. Given the small sample size and semi-structured nature of the Key Informant Interviews, administration of these interviews by phone is the most efficient mode. We will email the sample members to arrange a time for the interviews that is most convenient for them.

4. Efforts to Identify Duplication

No other evaluation of the AHRQ Healthcare Horizon Scanning System has been conducted. As the AHRQ Healthcare Horizon Scanning System is the only horizon scanning system in the U.S., there is no other comparable evaluation of a U.S. horizon scanning system.

5. Involvement of Small Entities

This information collection will not impact small businesses.

6. Consequences if Information Collected Less Frequently

This is a one-time information collection. The data collection is necessary for conducting the evaluation. The consequences of not collecting the specific data are discussed below.

Without the Expert Survey, AHRQ will not have a measure of the accuracy and completeness of the reports. Experts have the in-depth knowledge of the technologies required to reliably assess the quality of the reports and impact assessments.
Without the Expert Consultation to confirm cases of inaccurate or missing information reported in the Expert Survey, we will be uncertain as to whether these cases do represent inaccurate or missing information.
Without the Stakeholder Survey, AHRQ will not be able to measure how useful the reports are and how the reports are being used in order to improve the reports. Nor will AHRQ be able to learn about the sources and ways stakeholders access CER information to help AHRQ identify effective ways to distribute the reports.
Without the Key Informant Interviews with the staff, AHRQ will not have the information to identify which areas are in need of improvement and how the AHRQ Healthcare Horizon Scanning System can be improved. Directly involved in these processes, the staff are valuable sources for this information.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), a 60-day notice was published in the Federal Register (see Attachment F) on August 12, 2014 for public comments.

8.b. Outside Consultations

Several individuals outside of AHRQ have consulted on the development of the survey instruments and evaluation plan. These individuals include:

Eugene Rich, Ph.D., Mathematica Policy Research
Nancy Duda, Ph.D., Mathematica Policy Research
Dominick Esposito, Ph.D., Mathematica Policy Research
Rivka Weiser, Mathematica Policy Research

9. Payments/Gifts to Respondents

Respondents to the Expert Survey will not receive any gifts or payment in exchange for their participation.

Respondents to the Stakeholder Survey will not receive any gifts or payment in exchange for their participation.

AHRQ will not provide an incentive payment for the Expert Consultation to confirm cases of inaccurate or missing information reported in the Expert Survey, because the time demanded by the consultation will be very short, up to 10 minutes, and the experts have a relationship with ECRI.

The evaluation will not offer incentive payment to participants in the Key Informant Interviews because it is expected that the AHRQ Healthcare Horizon Scanning System staff will cooperate with the interviews.

10. Assurance of Confidentiality

The evaluation will not collect any personally identifiable information from respondents other than name and contact information for the purposes of issuing an incentive check and following up on responses, if needed. Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose. Mathematica will follow procedures for assuring and maintaining confidentiality, consistent with provision of the Privacy Act of 1974, as amended (45 CFR 5b). MPR and its subcontractors will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Further, personally identifiable data will not be entered into the analysis file and data records will contain a numeric identifier only. When reporting the results, data will be presented only in aggregate form so that individuals and organizations will not be identified. A statement to this effect will be included with all requests for data.

Prior to the start of data collection, each sample member will be sent an email or letter that will explain the study and purpose of the survey and inform the sample member of the following:

participation in the survey is voluntary
information provided is held in strict confidence and used only for study
names of individual and organization will never be used in reporting the results

Further, no individually identifiable information will be maintained by the study team. All members of the study team having access to the data will be trained on the importance of confidentiality and data security. All data will be kept in secured locations, and identifiers will be destroyed as soon as they are no longer required.

The following safeguards will be employed by MPR to carry out confidentiality assurances during the study:

All employees at MPR sign a confidentiality pledge emphasizing its importance and describing their obligation.
Access to identifying information on sample members is limited to those who have direct responsibility for providing and maintaining sample locating information. At the conclusion of the research, these data are destroyed.
Identifying information is maintained on separate forms and files, which are linked only by sample identification number.
Access to the file linking sample identification numbers with the respondents’ ID and contact information is limited to a small number of individuals who have a need to know this information.
Access to the hard-copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.
Computer data files are protected with passwords, and access is limited to specific users. Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use.

11. Questions of a Sensitive Nature

This information collection will not include any questions of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Mathematica expects a response rate of 80 percent from the sample of 67 experts for the Expert Survey, resulting in 54 completes. The Expert Survey is expected to require about 20 minutes, on average, to complete. Mathematica expects that Expert Consultation with 15 experts will be needed to confirm cases of inaccurate or missing information identified in the Expert Survey. The follow-ups should be about 10 minutes.

For the Stakeholder Survey, Mathematica expects that 30 percent of the sample of 700 stakeholders will be ineligible (i.e. will not find any of the presented reports relevant and therefore unable to rate a report) and that 65 percent of the eligible sample will complete, resulting in 319 completes. It should take about 30 minutes to complete the Stakeholder Survey. Mathematica will conduct semi-structured Key Informant Interviews, on average lasting 50 minutes, with 23 respondents.

Exhibit 1. Estimated annualized burden hours

Form Name	Number of respondents	Number of responses per respondent	Hours per response	Total burden hours
Expert Survey	54	1	.33	18
Expert Consultation	15	1	.17	3
Stakeholder Survey	319	1	.50	160
Key Informant Interviews	23	1	.83	19
Total	411			200

Exhibit 2. Estimated annualized cost burden

Form Name	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
Expert Survey	54	17.8	$92.25^**	$1,642
Expert Consultation	15	2.5	$92.25^**	231
Stakeholder Survey	319	159.5	$48.72^***	7,771
Key Informant Interviews	23	19.1	$38.68^****	739
Total	411			$10,383

*May 2013 National Occupational Employment and Wage Estimates, U.S. Department of Labor, Bureau of Labor Statistics. (http://www.bls.gov/oes/current/oes_nat.htm)

**Based on average wage for physicians and surgeons (29-1060).

***Based on average wage for medical and health services managers (11-9111).

****Based on average wage for social scientists and related workers (19-3000).

13. Estimates of Annualized Respondent Capital and Maintenance Costs

There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

The average annual cost (for the contractor) of conducting this evaluation is $130,421. The average annual cost to AHRQ for project oversight for this evaluation is $7,083. (Please see Exhibits 3 and 4, below.)

The contractor’s estimate includes development of the evaluation plan, survey instruments, and interview protocols ($55,982); respondent incentives ($9,080); data collection, analysis, and reporting ($72,623). These estimates include labor and travel; other direct costs for computer, telephone, postage, reproduction, fax, printing, and survey facilities; and indirect costs for fringe benefits, general and administrative costs, and fees. These activities will be carried out over 5 years (9/2010 – 8/2012, 1/2013-12/2015).

Exhibit 3. Estimated Annualized Cost for the Contractor

Cost Component	Annualized Cost
Development of the evaluation plan, survey instruments, and interview protocols	$55,982
Respondent incentives	$1,816
Data collection, analysis, and reporting	$72,623
Total	$130,421

Exhibit 4. Estimated Annualized Cost to AHRQ for Project Oversight

Project Officer GS 15 Step 5:

$ 141,660

5% FTE

$ 7,083

15. Changes in Hour Burden

This is a new collection of information on the AHRQ Healthcare Horizon Scanning System.

16. Time Schedule, Publication and Analysis Plans

16.a. Plans for Tabulation and Analysis

To measure report accuracy, the Expert Survey will ask respondents if there is at least one inaccurate statement in a report and to provide an example of an inaccurate statement. Because of the issue of conflict of interest among experts and the limited size of the expert sample, there is a potential for an idiosyncratic response, so Mathematica will confirm these responses. Mathematica will carry out this confirmation in two ways: (1) Mathematica will consider the inaccuracy confirmed if more than one expert identifies the same inaccurate statement in the comment form; and (2) if no other expert cites the same inaccurate statement, Mathematica will conduct an Expert Consultation with another of the experts in that area. Mathematica will measure accuracy by computing the number of inaccurate reports as a percentage of the total number of sampled intervention reports.

To measure report completeness, the Expert Survey will ask respondents if the report is lacking any important information and to describe the missing information. Mathematica will use the same approach detailed in the previous paragraph to confirm cases of missing information. Reports confirmed as missing information will be counted as incomplete reports. Mathematica will measure completeness as the number of incomplete reports as a percentage of the total sampled reports.

To measure the sensitivity and specificity of the Potential High Impact Interventions, Mathematica will use the data collected in the second section of the Expert Survey. This data will be collected only in three Priority Condition areas: cancer, functional limitations and disability, and infectious disease. Experts in these condition areas will rank the intervention in terms of overall potential impact in the top quartile, second quartile, or bottom half of the interventions for that condition. For a given condition, if at least half of the experts place an intervention in the top two quartiles, Mathematica will consider that intervention a true Potential High Impact Intervention. Mathematica will estimate sensitivity as the percentage of the Potential High Impact Interventions that at least half of the experts ranked in the top two quartiles. For example, if among the 6 Potential High Impact Interventions being rated, at least half of the experts identified 3 in the top two quartiles, the sensitivity is calculated as 3 of 6 or 50 percent. Mathematica will measure specificity as the percentage of non-Potential High Impact Interventions that were correctly excluded from the Potential High Impact Interventions list. For example, if among the 6 non-Potential High Impact Interventions being rated, at least half of the experts rated only 1 as not being in the top two quartiles, the specificity is calculated as 1 of 6 or 17 percent. Because of the limited size of the sample and responses, all reported scores will be descriptive simple averages and will not be associated with any measures of statistical probability, such as confidence intervals.

For the Stakeholder Survey, each respondent will be asked to review and rate a selected Potential High Impact Intervention report, from a sample of 16 reports, on several aspects of usability. Mathematica will present usefulness scores for each report and across all reports. To calculate the usability of the individual report, Mathematica will calculate the average of the overall usefulness scores across the raters for that report. To measure report usefulness across all 16 reports, Mathematica will calculate the average of the usefulness scores for all of the raters. Mathematica will also present average scores for selected individual dimensions, such as ease of understanding or credibility, for each report and across the full sample of reports. Mathematica will report the margin of error and confidence interval. Scores will be presented with reference to the rating scale. For example, an average score of 3 will be reported as 3 on a scale from 1 to 5.

To analyze the Key Informant Interviews, Mathematica will use a list of preliminary codes to identify common themes when reading the interviews (Table A.1); that list of codes will be refined as interviews are analyzed. The codes will be used to analyze which key elements of the AHRQ Healthcare Horizon Scanning System Protocol can be improved, how they can be improved, and the relative priority of the suggested enhancements. Mathematica will look for common themes.

Table A.1. Codes for Interview Transcript Analysis

Code	Application in Interview Transcript
Positive	Things going well, successes
Negative	Things not going well, challenges
Idea	Idea for improvement
Importance	Relative importance (used in conjunction with Positive, Negative, and Idea)
Staff	Use of AHRQ Healthcare Horizon Scanning System staff
Protocol	AHRQ Healthcare Horizon Scanning System protocol
Scan	Scanning and lead selection
Develop	Development of interventions from leads
Meeting	Topic nomination meeting
Profile	Profiles developed for target interventions
Expert	Expert comment and ratings input
PHI	Potential High Impact Interventions selection and assessment
Monitoring	Intervention archiving, monitoring, and updating

16.b. Plans for Publication

Mathematica will deliver a report that describes the evaluation design and presents the findings of the evaluation. A draft report will be submitted to ECRI and AHRQ in August 2015. The final report will be delivered to ECRI and AHRQ in September 2015.

16.c. Project Time Schedule

The Expert Survey, Expert Consultations, Stakeholder Survey, and Key informant interviews will be conducted from February through April 2015. The draft report will be submitted in August 2015 and the final report will be delivered in September 2015.

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

Attachments:

Attachment A: Healthcare Research and Quality Act of 1999

Attachment B: Expert Survey

Attachment C: Expert Consultation Protocol

Attachment D: Stakeholder Survey

Attachment E: Key Informant Interview Protocols

Attachment F: 60 Day Federal Register Notice

File Type	application/msword
File Title	Supporting Statement Template Part A
Subject	<p>(Supporting Statement Template and Instructions; instructions are in italics and should be deleted when complete)
Author	Heather Nalls
Last Modified By	erwin.brown
File Modified	2015-02-03
File Created	2015-02-03