SUPPORTING STATEMENT
Part A
Evaluation of Learning Health Systems K12 Training Program
Version: May 18, 2020
Agency for Healthcare Research and Quality (AHRQ)
Table of Contents
A. Justification 1
1. Circumstances that make the collection of information necessary 1
2. Purpose and use of information 3
3. Use of Improved Information Technology 4
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 5
6. Consequences if Information Collected Less Frequently 5
7. Special Circumstances 5
8. Consultation outside the Agency 6
9. Payments/Gifts to Respondents 6
10. Assurance of Confidentiality 6
11. Questions of a Sensitive Nature 7
12. Estimates of Annualized Burden Hours and Costs 7
13. Estimates of Annualized Respondent Capital and Maintenance Costs 8
14. Estimates of Annualized Cost to the Government 8
15. Changes in Hour Burden 9
16. Time Schedule, Publication and Analysis Plans 9
17. Exemption for Display of Expiration Date 11
List of Attachments 12
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see http://www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
research that develops and presents scientific evidence regarding all aspects of health care; and
the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special healthcare needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
This request is for Office of Management and Budget (OMB) clearance for the data collections for an evaluation of the Learning Health Systems K12 Training Program. AHRQ, in partnership with the Patient-Centered Outcomes Research Institute (PCORI), supports an innovative institutional mentored career development program (K12) to train clinician and research scientists to conduct patient-centered outcomes research within learning health systems (LHSs). LHSs provide an environment where science generated from health services research; patient-centered outcomes research (PCOR); and clinical research, informatics, incentives, and culture are aligned for continuous improvement and innovation. In addition, in an LHS, best practices are seamlessly embedded in the care process, in which stakeholders (i.e., providers, patients, and families) are active participants in all elements, and new knowledge is captured as an integral byproduct of the care experience.1 The following are the LHS K12 training program objectives:
Develop and implement a training program that includes both didactic and experiential learning and embeds the scholars in training at the interface of research, informatics, and clinical operations within LHSs
Identify, recruit, and train clinician and research scientists who are committed to conducting PCOR in healthcare settings that generate new evidence to facilitate rapid implementation of practices that will improve quality of care and patient outcomes
Establish Centers of Excellence (COEs) in LHS Research Training, focusing on the application and mastery of the newly developed core LHS researcher competencies
Promote cross-institutional scholar-mentor interactions, cooperation on multisite projects, dissemination of project findings, methodological advances, and development of a shared curriculum
In September 2018, AHRQ and PCORI awarded grants to 11 institutions across 5 years to establish LHS K12 COEs to support the training of the next generation of LHS researchers. The next generation of LHS researchers to undergo the training within the newly funded 11 LHS COEs will be equipped with the skills needed to conduct PCOR and implement the results to improve quality of care and patient outcomes. Each LHS K12 COE recruits trainees, otherwise known as scholars, to matriculate in the training program. Individual scholars’ training duration is typically 2–3 years. Each LHS K12 scholar’s training is supported through (1) formal mentorship; (2) experiential learning, including conducting a research project; and (3) didactic learning. Each LHS K12 scholar has a mentoring team that may consist of a research mentor and a health system advisor to guide the scholar’s career development and monitor project progress.
The purpose of this evaluation is to assess the overall achievement of the LHS K12 training program’s objectives, outcomes, and impact, as well as the program’s value to its stakeholders. The information collected through this data collection will allow AHRQ to improve the LHS K12 program and identify whether results correspond to intentional changes in program strategy and implementation.
To achieve the goals of this project the following data collections will be implemented. Each data collection is described in more detail in section 2 below.
Scholar Interview: Interviews with LHS K12 scholars assess the degree of scholar embeddedness in their respective health systems and query which aspects of the training program were most and least successful.
Health System Advisor Interview: Interviews with scholars’ health system advisors assess the perceived value of the LHS K12 training program to the health system and the role of health system advisors in supporting the research conducted by LHS K12 scholars.
Program Director Interview: Interviews with LHS K12 program directors assess the perceived value of the LHS K12 training program to the health system and the role of health system advisors in supporting the LHS K12 training program.
Health System Advisor Survey: Pre-post surveys with scholars’ health system advisors measure attitude changes toward the role of health systems research and the importance of patient, family, and other stakeholder engagement in research.
This study is being conducted by AHRQ through its contractor, 2M Research, pursuant to AHRQ’s statutory authority to ‘‘build capacity for comparative clinical effectiveness research by establishing a grant program that provides for the training of researchers in the methods used to conduct such research.’’ 42 U.S.C. 299b–37(e).
The evaluation will include two types of data collection: (1) semi-structured interviews with scholars who are close to completing the LHS K12 training program, their health system advisors, and program directors of each of the 11 institutions; and (2) surveys with health system advisors. The proposed data collection spans 3 years (2020–2023).
To achieve the goals of this project, the following data collections will be implemented:
1. Scholar Interview: Interviews with LHS K12 scholars assess the degree of scholar embeddedness in their respective health systems and query which aspects of the training program were most and least successful. Telephone interviews will be conducted one time with scholars who are currently enrolled but close to (within 2 to 3 months of) completing the LHS K12 training program. The total estimated number of scholars interviewed will be approximately up to 123 (or approximately 41 scholars annually).
2. Health System Advisor Interview: Interviews with scholars’ health system advisors assess the perceived value of the LHS K12 training program to the health system and the role of health system advisors in supporting the research conducted by LHS K12 scholars. One health system advisor from each scholar’s advisory committee will be interviewed by telephone. Health system advisors selected for interviews will include those with direct involvement with or knowledge of the LHS K12 scholars’ research projects. Health system advisors will be interviewed once around the same time that the scholar is interviewed. The total estimated number of health system advisors interviewed will be approximately up to 116 (or approximately 39 health system advisors annually).
3. Program Director Interview: Interviews with LHS K12 program directors assess the perceived value of the LHS K12 training program to the health system and the role of health system advisors in supporting the LHS K12 training program. The program director of each of the grantee institutions participating in the LHS K12 program will be interviewed by telephone in the final year of the LHS K12 program. The total number of program directors interviewed will be 10 (or approximately 4 program directors annually).
4. Health System Advisor Survey: Pre-post surveys with scholars’ health system advisors measure change in attitudes toward the role of health systems research and the importance of patient, family, and other stakeholder engagement in research. A brief survey will be administered electronically to health system advisors at two time points: Once at the beginning and conclusion of their respective scholar’s training. The total number of health system advisors surveyed will be approximately up to 190 (or approximately 63 health system directors annually).
AHRQ will use the information collected through this Information Collection Request to assess the program progress of the LHS K12 training program, and impact to its LHS stakeholders in a prospective manner. The information collected will facilitate program planning.
AHRQ is committed to complying with the E-Government Act of 2002 to promote the use of technology in order to lessen the burden of data collection on the public. Data from all interviews will be collected over the telephone and digitally recorded to reduce respondent burden. Data from the health system advisor annual survey will be collected electronically through a self-administered web-based survey instrument using low-burden and respondent-friendly survey administration processes. The technology to be employed can be configured to allow participants to complete as much of the questionnaire as desired in one sitting and to continue the questionnaire at another time. The technology also minimizes the possibility of participant error by electronically skipping questions that are not applicable to a particular participant. Screenshots of the programed HSA Survey are provided in Appendix D6.
AHRQ IT Survey Application
The AHRQ IT Survey Application is an online web-based tool that provides customizable surveys for AHRQ. Key features include customizable survey questions and webpages, communication modules for reaching out to survey respondents, and data analysis and visual representation tools. The Survey application collects and stores survey data and survey respondent information in a secure database behind the AHRQ network firewall. AHRQ survey administrators access the Survey application data through a secure two-factor single sign-on authentication. External administrators access the application through a secure online interface that requires a user identification and password entered through the Survey Login screen. All Survey data are manually deleted from the system 30 days after the survey end date.
Features of the AHRQ Survey application include the following:
Create and manage survey
Create and publish survey webpage
Pre-register survey respondents
Email notifications to survey respondents
Search and filter survey data
Export to spreadsheet
Survey analytics and reporting
Store survey data in the secure survey database
Manual delete/purge of survey data
Every effort has been made to avoid duplication. The current request supports the evaluation of the LHS K12 training program for AHRQ and, therefore, the instruments are specific and unique to this project. AHRQ has reviewed reporting requirements, and none of these sources provide the necessary data to evaluate the program.
This project does not intend to intentionally involve, exclude, or impact any small entities. However, to the extent an identified and recruited organization meets the requirements for participation and is a small entity, AHRQ will involve them and expects no greater impact than for other participating organizations. The instruments and procedures used to collect data are designed to minimize the burden on all respondents.
If AHRQ were to forgo this data collection, AHRQ would not have systematic information on the overall achievement of the LHS K12 training program’s objectives, outcomes, and impact, as well as the program’s value to its stakeholders.
All data collections are conducted once with an individual, with the exception of the health system advisor survey, which is administered at two points in time. It is important that this survey be administered at the beginning and at the end of the scholars’ participation in the program. The health system advisor survey measures change in health system advisors’ attitudes toward the role of research carried out by LHS K12 scholars in health systems and the importance of patient, family, and other stakeholder engagement in research over time. If the survey were to be administered fewer than one time, AHRQ would not be able to measure these aspects as the LHS K12 program becomes more established within the health systems.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on page14487, volume 85, number 49, March 12, 2020, and allowed 60 days for public comment (see Appendix E). AHRQ did not receive comments from the public during this period.
The Contractor conducted usability testing of the Health System Advisor Survey (Appendix D1) and related email communication materials (Appendices D2, D3, and D5) in April 2020 with eight members of their research team. The survey was tested using multiple modes (computer, cellular telephone [androids and iPhones], and tablets [iPads, Samsung tablets, and Kindle Fires]) and across various browsers (Chrome, Safari, Edge, Firefox, Internet Explorer, and Safari). The “close and continue at a later time” function of the survey was also tested. Minor changes were made to the survey materials including streamlining response options and clarifying the wording of one question.
The participants in the study will not receive an incentive payment or gift.
Information gathered from respondents participating in this study is for research purposes only and will be kept private to the full extent allowed by law. Data will only be reported in aggregate form, no respondent-level information will ever be made public, and individual respondents will not be identified from the data. Respondents will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act. 42 U.S.C. 299c-3(c). This law requires that information collected for research conducted or supported by AHRQ, which identifies individuals or establishments, be used only for the purpose for which the information was supplied.
Names, email addresses, and phone numbers of respondents will be collected for the sole purpose of contacting them to schedule interviews and conduct surveys. Respondents will be told the purposes for which the information is being collected, that the confidentiality of their responses will be maintained, and that no information that could identify an individual or establishment will be disclosed unless that individual or establishment has consented to such disclosure. The Contractor will assign a unique identification number to each respondent. A separate file, with access restricted to only the contractor staff who need the information, will associate the identification number with personal information. Once the contract is complete, the Contractor will be required to destroy the files that contain private information.
This data collection does not request information typically considered private or sensitive in nature. Verbal consent will be obtained prior to any interviews being conducting. Consent language is built into the introduction of each interview guide (Appendices A1, B1, and C1).
Table 1 shows the estimated annualized burden hours required for respondents to participate in this evaluation. Interviews (Appendices A1, B1, and C1) will be conducted with a total of 123 scholars, 116 health system advisors, and 10 program directors (approximately 41 scholars, 39 health system advisors, and 4 program directors each year).2 Each interview is expected to last approximately 60 minutes. Surveys (Appendix D1) will be conducted with a total of 190 health system advisors (or approximately 63 health system advisors each year).3 The survey is expected to take less than 10 minutes. The total hour burden is expected to be 284.13 hours (or approximately 94.71 hours each year) for this participant data collection effort.
Table 1. Estimated Annualized Burden Hours
Instrument |
Estimated Number of Respondents |
Frequency of Response |
Average Time Per Response (Hours) |
Total Annual Burden Estimate (Hours) |
Scholar Interviews |
41 |
1 |
1.00 |
41.00 |
Health System Advisor Interviews |
39 |
1 |
1.00 |
39.00 |
Program Director Interviews |
4 |
1 |
1.00 |
4.00 |
Health System Advisor Surveys |
63 |
1 |
0.17 |
10.71 |
Estimated Annual Total |
147 |
|
|
94.71 |
Table 2 shows the estimated annualized cost burden based on the time required for respondents to participate in this project. This cost was calculated using average hourly earnings for May 2018, obtained from the Bureau of Labor Statistics’ estimates for occupational employment wages.4 The total estimated cost burden for this data collection is $19,580.75 (or approximately $6,526.92 each year). The following hourly wages were used in the annualized cost calculations: $37.38 per hour for a scholar,5 $96.22 per hour for a health system advisor,6 and $52.81 per hour for a program director.7
Instrument |
Estimated Number of Respondents |
Total Annual Burden Estimate (Hours) |
Hourly Rate |
Total Cost |
Scholar Interviews* |
41 |
41.00 |
$37.38 |
$1,532.58 |
Health System Advisor Interviews** |
39 |
39.00 |
$96.22 |
$3,752.58 |
Program Director Interviews*** |
4 |
4.00 |
$52.81 |
$211.24 |
Health System Advisor Surveys** |
63 |
10.71 |
$96.22 |
$1,030.52 |
Estimated Annual Total |
147 |
94.71 |
|
$6,526.92 |
Bureau of Labor Statistics (BLS), U.S. Department of Labor. (2018). Occupational employment statistics May 2018 national wages. https://www.bls.gov/oes/home.htm
*The hourly wage for scholars varies depending on the scholar’s degree. AHRQ averaged hourly wages using the following occupations code to develop an estimate that represents the mix of medical and academic degrees: 29-0000, 29-1000, 21-0000.
**AHRQ anticipates that many health system advisors will be C-suite leaders. The hourly wage for BLS’s occupation code 11-1010 (chief executive) was used for this estimate.
***Program directors hold various roles and responsibilities and, therefore, have varied salaries. For the purpose of this estimate, the hourly wages for the following managerial and post-secondary occupational codes were averaged: 11-3131,11-1021,11-9030,11-9033,11-9039, and 11-9199.
There are no direct costs to respondents other than their time needed to participate in the study.
The annualized cost to the Federal Government for data collection and reporting for this project is $136,542.80. Table 3a includes costs associated with the Contractor conducting the evaluation of the project, which is approximately $133,055 each year. Table 3b includes the annualized cost of federal employee oversight of the project, which is estimated to be $3,487.80 each year. The estimate is based on one Contracting Officer's Representative (COR) GS-14, step 1, with an hourly wage of $58.13 and 60 hours of administrative and review time each year.8
Cost Component |
Total Cost |
Annualized Cost |
Project Management |
$49,776 |
$16,592 |
Project Development |
$38,428 |
$12,809 |
Data Collection Activities |
$171,893 |
$57,298 |
Data Processing and Analysis |
$60,248 |
$20,083 |
Reporting Results |
$78,821 |
$26,274 |
Estimated Total |
$399,166 |
$133,055 |
Activity |
Federal Personnel |
Hourly Rate |
Estimated Hours |
Annualized Cost |
Data collection oversight |
1 |
$58.13 |
52 |
$3,022.76 |
Review of results |
1 |
$58.13 |
8 |
$465.04 |
Estimated Annual Total |
|
|
60 |
$3,487.80 |
This is a new information collection request.
This study will use quantitative and qualitative methods to analyze the survey and interview data, respectively, and produce reports. The descriptive statistics will include frequencies, averages, and ranges for the data elements. Interview data will be coded using NVivo to identify themes for analysis.
Analysis of survey data
The survey data collected will be analyzed to measure change in health system advisors’ attitudes toward the role of research carried out by LHS K12 scholars in health systems, as well as toward the importance of patient, family, and other stakeholder engagement in LHS K12 research over time. The surveys are designed to yield answers that include a Likert-type scale. Descriptive characteristics will be tabulated and frequencies will be calculated for responses to questions.
Attitudes toward the role of research carried out by LHS K12 scholars in health systems and the importance of patient, family, and other stakeholder engagement in LHS K12 research will be measured by changes in overall pre-test and post-test scores. Comparisons between pre-test and post-test scores will be assessed using a paired t-test. Scores will be examined by job title and years of experience. Post-test scores may also be regressed against pre-test scores and adjusted for these additional characteristics using linear regression.
Analysis of interview data
The purpose of the qualitative analysis is to asses which aspects of the training program were most and least successful, along with the role of health system and health system advisors in facilitating the training and scholars’ research projects. The qualitative analysis will result in narrative summaries and illustrative quotations that describe these findings.
The interviews will be coded using qualitative analysis software (NVivo). Analysts will examine and analyze the coded data to identify themes, specifically focused on the evaluation questions.
AHRQ will receive summarized reports (created by the Contractor). If AHRQ desires, they may share this summary report with the participating institutions. No respondent-specific information will be shared outside of AHRQ. General results may be made publicly available, such as on the AHRQ website, meeting presentations, or manuscripts.
Below is a schedule for the proposed study.
Activity |
Due Date |
Recruitment for Interviews (Scholars and Health System Advisors) |
Rolling–within 3 to 4 months of scholar completing the program |
Train Data Collectors (Interviews) |
Within 1 month of data collection start (will train, as needed, if new data collectors are used) |
Interview Data Collection (Scholar and Health System Advisor Interviews) |
Rolling–within 2 to 3 months of scholar completing the program |
Recruitment for Surveys (Health System Advisors) |
Rolling–within 3 to 4 months of scholar enrolling in the program and 3 to 4 months of scholar completing the program9 |
Survey Data Collection (Health System Advisors) |
Rolling–within 2 to 3 months of scholar enrolling in the program and 2 to 3 months of scholar completing the program |
Recruitment for Interviews (Program Directors) |
Within 7 months before the end of the Contractor’s award |
Interview Data Collection (Program Director Interviews) |
Within 6 months before the end of the Contractor’s award |
Data Analysis and Annual Progress Reports |
Yearly–during months 3 to 5 of each year during data collection |
Draft Report |
Within 4 months before the end of the Contractor’s award |
Final Report |
Within 1 month before the end of the Contractor’s award |
AHRQ does not seek this exemption.
Appendix A1. Scholar Interview Guide
Appendix A2. Scholar Invitation Email
Appendix A3. Scholar Study Description/FAQs
Appendix A4. Scholar Invitation Reminder Email
Appendix A5. Scholar Telephone Recruitment Script
Appendix A6. Scholar Scheduling Email
Appendix A7. Scholar Confirmation Email
Appendix A8. Scholar Reminder Email
Appendix A9. Scholar Reminder Telephone Script
Appendix A10. Scholar Thank-You Email
Appendix B1. HSA Interview Guide
Appendix B2. HSA Invitation Email
Appendix B3. HSA Study Description/FAQs
Appendix B4. HSA Invitation Reminder Email
Appendix B5. HSA Telephone Recruitment Script
Appendix B6. HSA Scheduling Email
Appendix B7. HSA Confirmation Email
Appendix B8. HSA Reminder Email
Appendix B9. HSA Reminder Telephone Script
Appendix B10. HSA Thank-You Email
Appendix C1. PD Interview Guide
Appendix C2. PD Invitation Email
Appendix C3. PD Study Description/FAQs
Appendix C4. PD Invitation Reminder Email
Appendix C5. PD Telephone Recruitment Script
Appendix C6. PD Scheduling Email
Appendix C7. PD Confirmation Email
Appendix C8. PD Reminder Email
Appendix C9. PD Reminder Telephone Script
Appendix C10. PD Thank-You Email
Appendix D1. HSA Survey
Appendix D2. HSA Survey Invitation Email
Appendix D3. HSA Survey Reminder Email
Appendix D4. HSA Survey Reminder Telephone Script
Appendix D5. HSA Survey Thank-You Email
Appendix D6. HSA Survey Screenshots
Appendix E. Federal Register 60-Day Notice
1 Institute of Medicine, Committee on the Learning Health Care System in America. (2012, September). Best care at lower cost: The path to continuously learning health care in America. Smith, M., Saunders, R., & Stuckhardt, L., et al. (Eds.). Washington, DC: National Academies Press (United States).
2 This is the annualized burden for program directors. Interviews with program directors will only be conducted in the final year of the LHS K12 Training Program.
3 Note, this is not the unique number of respondents. It is assumed that the same health system advisor will respond to the survey twice (once at the beginning of a scholar’s tenure and once close to when the scholar is expected to complete the LHS K12 Training Program [in approximately 2 years]) during the data collection period.
4 Bureau of Labor Statistics (BLS), U.S. Department of Labor. (2018). Occupational employment statistics May 2018 national wages. Retrieved from: https://www.bls.gov/oes/home.htm
5 Hourly wages for scholars vary depending on each scholars’ degrees. AHRQ averaged hourly wages using the following occupations code to develop an estimate that represents the mix of medical and academic degrees: 29-0000, 29-1000, 21-0000.
6 AHRQ anticipates that many health system advisors will be C-suite leaders. The hourly wage for BLS’s occupation code 11-1010 (chief executive) was used for this estimate.
7 Program directors hold various roles and responsibilities and, therefore, have varied salaries. For the purpose of this estimate, the hourly wages for the following managerial and post-secondary occupational codes were averaged: 11-3131,11-1021,11-9030,11-9033,11-9039, and 11-9199.
8 Federal employee pay rates are based on the General Schedule of the Office of Personnel Management for 2020 for the Washington, DC, locality and a 2,080-hour work year. https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2020/DCB_h.pdf
9 To streamline the data collection process, health system advisors selected for the survey may be organized into two groups. The groups will be developed based on when the health system advisor’s scholar is expected to complete the LHS K12 training program. The survey will be administered twice a year; however, a respondent will only complete the survey once in the year that they were selected to participate. Each administration of the survey will include both pre-test surveys and post-test surveys administered to the appropriate respondent.
10 Appendices are purposely grouped by data collection activity.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Rich, Curtis B. |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |