SUPPORTING STATEMENT
Part B
The AHRQ Safety Program for Telemedicine: Improving Antibiotic Use
Version: July 11, 2024
Agency for Healthcare Research and Quality (AHRQ)
B. Collections of Information Employing Statistical Methods 3
1. Respondent Universe and Sampling Methods 3
2. Information Collection Procedures 5
The data collection planned under this program is part of a comprehensive evaluation strategy to assess the adoption of the AHRQ Safety Program for Telemedicine (“Safety Program”). Specifically, data collection will be used to assess improvements in antibiotic prescribing among practices that provide telemedicine; measure the effectiveness of the Safety Program among participating practices; and evaluate the characteristics of teams that are associated with successful implementation and improvements in outcomes. A key component of the Safety Program is the recruitment of clinics that utilize telemedicine in their practices that will be supportive of and likely to spread the Safety Program model.
This revision to the currently approved OMB clearance materials includes updates to the improving antibiotic use cohort data collection forms, as informed by pre-recruitment discussions with our Technical Expert Panel (TEP) and potential participants. The improving antibiotic use cohort will take place June 2024 – December 2025. This revision also removes references to the diagnostic process cohort, as the cohort was cancelled prior to its initiation.
This revised data collection request covers planned activities that will occur over the course of 18 months.
Recruitment Methods:
Recruitment will include identifying telemedicine-only practices as well as brick-and-mortar practices that provide telemedicine.
Targets. The program will recruit 300 to 500 practices that provide telemedicine to include practices from across the 10 Department of Health and Human Services (HHS) Geographic Regions.
Practice Characteristics. To meet the recruitment targets, the program will cast a wide net and ensure that eligibility criteria garner broader participation rather than unnecessarily excluding potential practices. In addition to ensuring coverage of all 10 HHS regions, the program will seek to ensure diversity of recruited practices. The improving antibiotic use cohort will include clinic-associated primary and urgent care practices as well as telemedicine-only practices and organizations. We will recruit practices that are both urban and rural; small, medium, and large; and independent and hospital-owned.
Recruitment Strategy. The program will use a multi-pronged approach for recruitment of practices. Our preliminary recruitment plan balances the operational challenges unique to outpatient telemedicine practices with the need for standardization across practices and settings. We will work with AHRQ, Teladoc, and our recruitment partners to enhance and execute our approach. This includes collaborating with federal partners such as the Centers for Disease Control and Prevention (CDC), Centers for Medicare & Medicaid Services (CMS), and Health Resources & Services Administration (HRSA) through AHRQ. To ensure representation from all 10 HHS regions, we will approach national quality organizations and relevant medical specialty societies. Additional recruitment activities will include the following:
Engagement with the TEP members from delivery organizations to gauge their interest in participating in this program.
Leverage the AHRQ patient safety listserv of over 55,000 contacts, the Johns Hopkins Armstrong Institute (JHAI) listserv of 8,000 contacts that have participated in prior patient safety interventions with JHAI, and Teladoc dissemination channels.
Use free social media to recruit ambulatory clinics, including periodically posting on the Johns Hopkins School of Medicine’s (JHM) Facebook (603k likes), Twitter (522k followers), and LinkedIn (37k followers), as well as JHAI’s Twitter (1.6k followers) and social media platforms.
Host informational webinars to garner interest in the program.
Develop a website for the Safety Program that will assist with recruitment.
Collaborate with federal partners including CMS and HRSA to ensure synergistic efforts across related projects.
Respondent Selection and Sample Sizes. The Safety Program will collect data from all participating practices when possible.
For the practice-level measures including the Structural Assessment, the Participant Experience Survey, and EHR data, the program will collect these data from a practice lead/champion at each participating practice. Each leader/champion of brick-and-mortar practices or telemedicine-only organizations will complete the Structural Assessment at pre- and post- intervention to measure changes in infrastructure and policy related to patient safety and quality improvement. A leader/champion from each practice will complete the Participant Experience Survey at post-intervention to assess how practices implemented specific aspects of the Safety Program and how the Safety Program has influenced their practice’s antibiotic prescribing behaviors within telemedicine. For the EHR data, the program will sample all participating practices. This revision includes two different versions of the Structural Assessment and Participant Experience survey—one version for participants in brick-and-mortar and one version for participants in telemedicine-only organizations to account for the significant differences in their infrastructure and operations. This revision also includes modifications to the EHR data collection template to simplify the data collection process for participants.
For individual-level measures including the AHRQ Medical Office Survey on Patient Safety Culture (MOSOPS), the program will collect these data from all eligible staff within each participating practice; because practice size varies, the program estimates the average number of respondents to be seven per practice. This revision includes a modification to the number of anticipated respondents to the MOSOPS. This survey was designed to assess key characteristics of healthcare providers (HCPs) working in-person in a single medical office and therefore the results are unlikely to be reliable or valid if administered among HCPs practicing in telemedicine-only settings. In this revised submission we have proposed administering the MOSOPS to HCPs practicing in brick-and-mortar settings only.
For the semi-structured qualitative interviews that will be conducted, the program will aim to recruit one to two health care providers (HCPs) from approximately 20 participating practices based on a combination of contextual and structural factors (e.g., representation across telemedicine practice size, type, and HHS location) for a total of up to 40 participants from 20 practices. This revision includes changes based on lessons learned from the TEP and conversations with participants from the prior AHRQ Safety Program for Improving Antibiotic Use.
Response Rates. We anticipate a 75 percent response rate from the participating practices to the Structural Assessments and Participant Experience Survey and a 90 percent response rate from participating practices for the submission of EHR data (i.e., antibiotic usage data including antibiotic prescriptions for respiratory tract infection-related telemedicine visits), and response to qualitative interviews.
Completion rates for the AHRQ Survey on Patient Safety Culture, based on previous CUSP interventions focusing on ventilator associated pneumonia (VAP) and central line associated bloodstream infection (CLABSI), are anticipated to range between 30-50 percent. For those interventions, the practice response rates for the AHRQ Patient Safety Surveys varied from baseline to follow-up and from project to project. The response rates averaged 55 percent at baseline and between 16-28 percent at follow-up.
Primary Data Collection
The program will collect a range of data to contribute substantively to the evaluation and facilitate analysis of progress over time. Primary data collection includes surveys and qualitative discussions (see Exhibit 1 below). The program will analyze EHR data submitted from individual practices as secondary data sources.
Exhibit 1: Summary of Primary Data Collection Activities
Primary Data Collection Sources |
Data Collection Tools |
Target Population |
Data Collection Frequency |
Surveys |
Structural Assessment (different versions for brick-and-mortar practices and telemedicine-only organizations) |
Leader/champion at the participating practice/ organization |
intervention) and end of intervention |
|
AHRQ MOSOPS |
All staff at the participating brick-and-mortar practices |
At baseline (pre- intervention) and end of intervention |
Participant Experience Survey (different versions for brick-and-mortar practices and telemedicine-only organizations) |
Leader/champion at the participating practices |
End of the intervention period |
|
Qualitative Interviews (held virtually) |
A semi-structured interview guide |
1 to 2 HCPs from 20 practices (up to 40 providers total) |
End of the intervention period |
Surveys. The program will administer several surveys to practice staff involved in the program’s implementation.
Structural Assessment. The program will administer the Structural Assessment to all practices at baseline (pre-intervention) and at the end of the intervention period. This online assessment tool will ask the lead/champion twelve to thirteen multiple choice questions to understand the practice/organization infrastructure and capacity to carry out the Safety Program. Attachments A and B contain the Structural Assessments for participants in brick-and-mortar settings and telemedicine-only settings, respectively.
AHRQ’s Medical Office Survey on Patient Safety Culture (MOSOPS). The program will administer the MOSOPS to all participating staff at each brick-and-mortar practices only (telemedicine-only organizations excluded) at the beginning and end of the intervention (see Attachment C). The survey asks questions about patient safety issues, medical errors, and event reporting. The program will reach out to all eligible providers and staff to collect data according to AHRQ’s User Guide. All practice staff at brick-and-mortar practices that are involved in the implementation of Safety Program will be asked to complete the survey.
Participant Experience Survey. The program will administer the Participant Experience Survey to all practice leads/champions at the end of the intervention period. This online assessment will ask each practice lead/champion to reflect on their experience participating in the Safety Program including their approach to implementing the Safety Program. Attachments D and E contain the Participant Experience Surveys for participants in brick-and-mortar and telemedicine-only settings, respectively.
As practice size varies, the program estimates the average number of respondents to be seven per practice for surveys assessing all participants at each site. Participating staff should have enough knowledge about the day-to-day activities in the practice and interact regularly with other staff working in the practice in order to provide informed answers. These staff members, who spend all or most of their time at work within the practice, may include the following:
Staff who have direct contact or interaction with patients.
Staff who may not have direct contact or interaction with patients but whose work directly affects patient care.
Qualitative Interviews. The program will recruit one to two HCPs from approximately 20 practices to participate in a qualitative interview to explore practice-level characteristics (e.g., barriers/facilitators, perceived usefulness of program materials) that impact intervention implementation and adoption. The draft interview guide (Attachment F) has been updated from the version included in the original OMB review.
Secondary Data Collection
EHR Data Extracts. The Safety Program will collect visit-level telemedicine visit data to include diagnostic codes and any antibiotic prescriptions associated with the visit from each participating practice. The data will be used to evaluate the effectiveness of the program including: 1) if the Safety Program is effective in improving antibiotic use, and 2) the extent of the improvement in antibiotic use.
Every quarter beginning at baseline through the end of the intervention, each onsite data coordinator will extract monthly data from their EHR system and submit the data electronically to the program evaluation team at NORC. In this revision, we have added three months of baseline data to the initial data request to now include monthly data from March 2024 through August 2024 in the initial submission. The data will include deidentified visit-level information on respiratory tract infection (RTI) related telemedicine visits, including ICD-10 codes and any antibiotics that were prescribed. This will be used by the program team to calculate the overall rates of antibiotic use for RTI-related telemedicine visits by class and diagnosis (using ICD-10 codes). Attachment G contains the EHR data template and Attachment H contains the instructions for conducting the data extract. Exhibit 2 describes the EHR data that will be collected. The EHR template and instructions have been updated from the version included in the original OMB review to simplify data collection for participants.
Exhibit 2: EHR Data Extracts for the Improving Antibiotic Use Cohort
Evaluation Design
We will employ a pre-post longitudinal evaluation study with six months for baseline, 12 months for intervention period, and three months for endline. For the Structural Assessment and AHRQ MOSOPS, we will compare the responses from baseline to endline. For the Participant Experience Survey, we will examine the responses cross-sectionally. Differences in response rates by practice size and type will be explored and reported, if applicable. For the primary outcomes that will be collected through the EHR extracts of antibiotic prescriptions, we will apply a longitudinal analysis at the visit level to assess the change in antibiotic usage over time.
The implementation data collection is designed to be powered at 80 percent to detect a change from baseline to post-intervention period for 2.89 antibiotic prescriptions per 100 RTI-related visits (assuming SD of 20) (see Exhibit 4). The following assumptions were used in the power calculations: 1) 1-sided significance level of 0.05, 2) 500 practices, 3) within-practice correlation of 0.4, and 4) response rate of 90 percent.
Exhibit 4: Power Analysis
Total number of practices |
Correlation within practice |
Antibiotic prescriptions per 100 visits with RTI diagnosis |
500 |
0.2 |
3.34 |
500 |
0.4 |
2.89 |
500 |
0.6 |
2.36 |
400 |
0.2 |
3.74 |
400 |
0.4 |
3.23 |
400 |
0.6 |
2.64 |
300 |
0.2 |
4.31 |
300 |
0.4 |
3.74 |
300 |
0.6 |
3.05 |
Qualitative Analysis (Improving Antibiotic Use Cohort). The program will also conduct semi-structured qualitative interviews to assess practice level characteristics impacting intervention implementation and adoptions. Specifically, the interviews will explore the facilitators and barriers to participating in the Safety Program as well as perceptions of the usefulness of the Safety Program and will be used to gain an understanding of how to operationally measure appropriate antibiotic prescribing in a telemedicine environment. These one-on-one in-depth discussions will take place virtually with an HCP involved in the implementation of the Safety Program. Approximately 20 practices will be selected, with one to two HCPs participating per selected practice. The interviews will occur at endline (months 15-18).
The data collection planned under this program is part of an evaluation to assess the adoption of the Safety Program in practices that provide telemedicine. The results of the evaluation will be used to inform the Safety Program; they will not yield generalizable results or be used for statistical estimation purposes. The Safety Program will recruit practices that have indicated a willingness to participate and meet the inclusion criteria, and who will be supportive of and likely to spread the Safety Program model.
To encourage the participating practices to complete and submit the Structural Assessment, AHRQ’s MOSOPS, Participant Experience Survey, and EHR data, the Safety Program will implement the following strategies:
Offer technical assistance webinars and onboarding calls (including information related to submitting data) to participating practices.
Prompt the on-site data coordinator and practice leader for feedback from each participating practice regarding the data collection activities, and AS-related performance.
Work closely with the program’s Implementation Advisors to encourage timely completion of each assessment.
Work closely with each on-site data coordinator to address any data collection issues and develop site-specific data collection strategies when necessary.
Discuss data management and submission methods with the participating practices early in the process.
Provide sites with a secure portal to submit data. Users will receive a login and password to access the site.
Provide step-by-step instructions on data sharing via the central data collection platform.
Ensure that data collection tools are simple, easy to use, and succinct.
Ensure missing or inaccurate data are identified early and that issues are resolved in a systematic, highly reliable way.
Share data collection strategies/best practices to simplify processes through coaching calls.
The Safety Program data collection procedures and instruments have either been used previously in prior CUSP projects or were developed using existing frameworks such as RE-AIM. The data collection tools proposed for use in the Safety Program were used previously in the AHRQ Safety Program for Improving Antibiotic Use and have been adapted for use in the telemedicine environment. The AHRQ MOSOPS is publicly available and has been widely used to assess patient safety culture in a variety of health care settings. Finally, the Participant Experience Survey was developed using the RE-AIM framework, which has been widely used to assess implementation of healthcare interventions and programs.
NORC at the University of Chicago will serve as the primary consultants for statistical aspects of the design and analysis of the evaluation data. Exhibit 5 provides the names, titles, affiliations, and contact details for the program’s three main statistical consultants.
Exhibit 5: List of Statistical Consultants
Name |
Title and Institution |
Telephone Number |
Roy Ahn, ScD, MPH |
Vice President, Public Health Department, NORC |
312-759-4068 |
Yue Gao, MPH |
Senior Research Scientist, Health Care Evaluation Research Department, NORC |
301-634-9440 |
Ed Mulrow, PhD, MS |
Senior Vice President, Statistics and Methodology Department, NORC |
301-634-9441 |
The data will be collected by NORC at the University of Chicago.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | SChowdhury |
| File Modified | 0000-00-00 |
| File Created | 2024-07-21 |