SUPPORTING STATEMENT
Part B
Guide to Nursing Home Antimicrobial Stewardship
May 5, 2014
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 1
1. Respondent universe and sampling methods 1
2. Information Collection Procedures 3
The purpose of this implementation evaluation is to (1) assess how well the use of the Guide for Nursing Home Antimicrobial Stewardship (the Guide) optimizes antimicrobial (antibiotics and antifungal) use and (2) provide information on how to improve the Guide for future use. The Guide contains toolkits which a nursing home can use for antimicrobial stewardship. The information will include 6-months of pre-intervention information, qualitative information collected during the 6-month intervention period, and information collected post-intervention. The populations of interest include nursing homes, nursing home residents and families, nursing home staff (administrators and nurses), and prescribing clinicians. The units of analysis are the nursing home, resident, and infection. Researchers will purposively select 10 nursing homes based on their characteristics and willingness to participate and then include all eligible residents (i.e., all residents who are listed in the infection log at each nursing home). In each nursing home, interview information will be collected from nursing home staff and medical record data will be collected about residents listed in the infection log.
Nursing Home Level. The 10 nursing homes will be located in Texas, Wisconsin, California, and Pennsylvania, because project staff work with and have access to multiple nursing homes in each of those states. Information on the characteristics of nursing homes (e.g. monthly occupancy, number of beds, for-profit status, independent/chain status) will be used to select nursing homes with varying characteristics. Selected nursing homes will have diverse nursing home industry characteristics so that the best methods for dissemination and implementation for a particular type of nursing home can be identified. These data are publicly available (http://www.medicare.gov/nursinghomecompare/search.html) and do not present a burden to the nursing home staff. Our sampling frame will be limited to nursing homes, located in the study states, that meet both inclusion and exclusion criteria that are known, through previous research experience, to affect implementation of an antimicrobial stewardship program. The inclusion criteria are:
interest in antimicrobial stewardship
interest in participation
ability to have more than two or more champions (staff willing to lead the activity), and
Medicare / Medicaid certification.
The exclusion criteria are:
CMS-designated special focus facility, or
receiving only 1 or 2 stars in Medicare’s Nursing Home Compare indicating below average performance (it is unlikely they would be able to successfully implement the Guide).
Resident-level information. The primary goal of the analysis is to compare the rate of inappropriate prescription of antimicrobials before and after the implementation of the Guide. At the resident-level, the outcome variable is a binary variable indicating whether an antimicrobial was prescribed in the absence of a resident’s physical symptoms. To evaluate the effectiveness of the intervention, we use this binary outcome variable to conduct statistical power calculations and estimate the minimum required sample size from each nursing home for having 80% power to detect a difference before and after the intervention. Since each nursing home is measured repeatedly before and after the intervention and outcomes are more likely to be similar within the same nursing home than across different homes, we use cluster-randomized design for pair-matched samples to account for the correlation between pre-intervention and post-intervention outcomes and the clustering effect among nursing homes. To ensure adequate statistical power for various analyses, we calculated the required minimum sample sizes across different effect size assumptions and across a range of the values of k, the coefficient of variation between nursing homes. The smaller the value of k is, the less correlated the data are within each home, hence, the more power we will have for detecting an effect of the intervention. Other research suggests that k is often smaller than 0.25, and seldom exceeds 0.5 for most health outcomesi.
Exhibit 6 lists the minimum required sample size per nursing home to obtain 80% power in detecting the effect of implementing the Guide. The power calculations, conducted using the formula from Hayes and Bennett1, are based on the following assumptions:
The outcome measure is a binary variable, i.e., whether an antimicrobial was prescribed appropriately.
The proportion of inappropriate antimicrobial prescription before the intervention is 73%, and the reduction in the proportion of inappropriate antimicrobial prescriptions after the intervention is 50%, 30% or 10%, which represents a large, medium or small effect size, respectively. Based on results from a previous AHRQ study, which investigated the effect of training nursing home staff on use of a communication tool, the UTI SBAR form, to reduce inappropriate antimicrobial prescription for urinary tract infections, the proportion of a prescription being written in the absence of physical symptoms decreased in the homes that implemented the intervention from 73% before the intervention to 49% after the interventionii. Assuming results from the current study are similar with those from the previous study, we expect to obtain a small to medium sized intervention effect (e.g., 24%) for implementing the Guide.
Based on results from the previous study2, the estimate of the coefficient of variation, k, from twelve nursing homes during the pre-intervention period is k=0.19. We conducted the power calculation based on both the estimated k=0.19 from the previous study and a more conservative estimate of k=0.25 as suggested in Hays and Bennett1.
The overall Type I error rate is 5% for two-tailed tests.
Exhibit 6. Minimum Sample Size per Nursing Home for the 12-Month Period Required for 80% Statistical Power
Effect Size |
Sample Size for Individuals per Nursing Home |
|
|
k=0.19 (Estimate from previous study) |
k=0.25 |
Large (50%) |
4 |
4 |
Medium (30%) |
14 |
20 |
Small (10%) |
Infinity |
Infinity |
Estimate from previous study (24%) |
30 |
86 |
Note: Sample sizes listed in the table are the total sample sizes needed, combining those needed from the 6-month pre-intervention period and from the 6-month intervention period. |
||
Based on the power calculation results from Exhibit 6, if the effect size and the coefficient of variation are similar as those from the previous study, we will need to abstract information from at least 30*10=300 courses of antimicrobial treatment from the ten nursing homes during the 12-month period (i.e., the 6-month pre-intervention period and the 6-month intervention period) to achieve 80% power. Under a more conservative assumption, if the coefficient of variation equals 0.25, then we will need to abstract information from at least 86*10=860 courses of antimicrobial treatment from the ten nursing homes for the 12-month period to detect an intervention effect of 24% with 80% power. Finally, under the best case scenario of data abstraction (based on the average nursing home size, the occupancy rate, and the percent of individuals who would receive a course of antimicrobials for a suspected infection during a 6-month period, as detailed in the following section, i.e., “Estimated Number of Abstractions”), suppose we are able to abstract as many as 640 courses of antimicrobial treatment, we will be able to detect a small- to medium-sized intervention effect both when k=0.25 and when k=0.19 (i.e., the minimum detectable effect size equals 25% assuming k=0.25, and the minimum detectable effect size equals 20% if k= 0.19). We aim to abstract a total of 640 courses of antimicrobial treatment, but if a smaller sample size is achieved, we might still be able to detect an effect size as small as 24% if the actual coefficient of variation is found to be relatively small (e.g., k=0.19).
Medical Record Review. Data related to changes in, inappropriate use of, and compliance with criteria relating to antimicrobial use will be abstracted during extensive medical chart and minimum data set (MDS) audits and infection control log reviews, using a Web-based platform for chart audit data. The MDS is part of the federally mandated process for clinical assessment of all residents in Medicare and Medicaid certified nursing homes. This process provides a comprehensive assessment of each resident's functional capabilities and helps nursing home staff identify health problems. Care Area Assessments are part of this process, and provide the foundation upon which a resident's individual care plan is formulated. MDS assessments are completed for all residents in certified nursing homes, regardless of source of payment for the individual resident. AHRQ will support data abstraction at all nursing homes. The medical/MDS records and infection log entries for all residents who are listed in the infection log as having an infection will be examined. The universe of infections and residents who meet these eligibility criteria will be sampled. AHRQ’s contractor will abstract data from these data sources; thus, these information collections will place no burden on nursing homes staff.
Estimated Number of Abstractions (>640). Nationally, the average nursing home size is 110 beds with 86 percent occupancy rate, so the daily census should be approximately 94 residents. Based on previous experience with antimicrobial prescriptions in nursing homes, the research teams estimated that 35 percent of these individuals would receive a course of antimicrobials for a suspected infection during a 6-month period (94 x .35=32). This led the research team to estimate that 64 or more episodes of antimicrobial use for a suspected infection during the 12 months (6-month pre-intervention/ 6-month post-intervention; 2 x 32 = 64) of information collection in each nursing home would be tracked, with a 20 percent rate of two episodes occurring within the same individual. Therefore, the team expects to abstract information on at least 512 residents (64 x .8=51.2 x 10 nursing homes=512) and 640 courses of antimicrobial treatment (64 courses x 10 nursing homes=640) in the field test.
Staff/Clinician Level Data
Pre-intervention interviews. This will involve both closed and open ended interviews. The open ended interviews will examine (1) how the staff and the department(s) and/or wider nursing home perceive antimicrobial stewardship; (2) the amount of experience the site has in antimicrobial stewardship and its processes for handling infections; and (3) which toolkit(s) within the Guide the nursing home decided to adopt and why. Individual personnel will also be surveyed in the 10 participating nursing homes using the Absorbtive Capacity for Change Survey, a 25-item survey measuring the individual’s perception of the nursing home’s absorptive capacity (i.e., its capacity to adopt an innovation).
Estimated Number of Interviews (20). Two staff from each participating nursing home (preferably from a wide range of staffing levels, and including the nursing home administrator and the director of nursing) will be surveyed, resulting in a total of 20 unique surveys across nursing homes. Each individual will be surveyed once prior to the intervention.
Technical Assistance (TA) Information and Questions. A form to collect information from nursing home staff will be created for when nursing home staff choose to request assistance. Participating nursing homes will select which tool(s) or toolkits(s) to implement and will choose to request assistance. The number of responses will depend on who is asking for assistance and how many times the requests for assistance are made. The individuals most likely to contact the research team for assistance will be those leading the effort (the “champions”) from the nursing home. Champions may include but are not limited to the administrator, director of nursing and/or the assistant director of nursing. The researchers will collect the amount of time for the call, description of the problem, and whether additional follow-up is required.
Estimated Number of Technical Assistance Collections (60). It is unknown how many individuals will call for technical assistance. Conservatively, we estimate, six contacts per nursing home are estimated (one for each month of the 6-month intervention period) for a total of 60 contacts. The information collected will be mostly open ended, thus statistical power is not applicable. The length of time for the call will be reported descriptively, thus statistical power is not applicable. The data will be used to estimate future TA needs and ways to improve the Guide.
Proactive TA Discussions. Once a month during the 6-month intervention period, the champions at each of the 10 sites will be contacted to provide proactive technical assistance. During this call a semi-structured interview protocol to assess concerns with the selected tool(s) or toolkit(s) as well as identify challenges and facilitators to implementation will be used. These interviews will be open-ended qualitative interviews and analyzed as qualitative data, so statistical power is not an issue.
Estimated Number of Interviews (120). In each proactive TA call, up to two champions from the same nursing home will be interviewed. Therefore, up to 20 individuals will be interviewed across the 10 sites. Each champion will be interviewed 6 times over the course of the 6-month intervention period. These interviews will be open-ended qualitative interviews and analyzed as qualitative data, thus statistical power is not applicable.
Post-intervention interviews. Semi-structured interviews will be conducted at the completion of the 6-month intervention period with purposively selected nursing home personnel from each of the 10 study sites. These will include two champions, two nurses (typically licensed practical nurses or licenses vocational nurses), and if possible, a prescribing clinician and pharmacist. The purpose of these interviews will be to assess facilitators and identify challenges of implementation as well as methods to improve the Guide.
Estimated Number of Interviews (60). Two champions (administrator, director of nursing and/or assistant director of nursing), two nurses, one clinician, and one pharmacist involved in the use of the Guide will be interviewed from each of the 10 nursing homes for a total of 60 individuals. These interviews will occur after the 6-month intervention period. These interviews will be open-ended qualitative interviews and analyzed as qualitative data and statistical power is not applicable.
Information Collection Procedures
In all 10 nursing homes, medical record data will be collected retrospectively for the 6-month period prior to initiation of the intervention (the Guide) and again after the intervention period. Six months of medical record review data will be collected approximately 1 month prior to the start of the intervention. Pre-intervention interviews will be conducted over the phone approximately 1 month prior to the intervention. Passive and proactive TA will occur during the 6-month intervention period. Within approximately 1 month after the intervention, researchers will return to conduct the post-intervention interviews and to conduct the medical record reviews.
Exhibit 7 illustrates the information collection procedures that will be used for the 10 nursing homes prior to the intervention, during the intervention, and after the intervention.
All interviews will be audiotaped and attended by a note taker. The trained and experienced qualitative interviewer will use semi-structured interview protocols.
Exhibit 7. Information Collection Procedures by Stage
Activity (Period) |
Information Collection Procedure (10 Nursing Homes) |
Medical Record Review (6 months of data prior to the initiation of intervention) |
Abstracted from infection log, MDS, and other medical records by researchers |
Pre-intervention interviews (prior to the initiation of intervention) |
Qualitative questions and a survey (25 items) with two staff per participating nursing home responding for 10 responses. |
Technical Assistance (TA) Information and Questions (during the 6-month intervention period) |
Passive Technical Assistance (8 items) with up to one contact per month per site for a total of 60 contacts during the 6-month intervention period. |
Proactive TA discussions (during the 6-month intervention period) |
Proactive Technical Assistance (16 items) with up to two staff per participating nursing home each month for a total of 60 interviews during the 6-month intervention period. |
Medical Record Review (after the 6-month intervention period) |
MDS and medical records abstracted by researchers. |
Post-intervention interviews (after the 6-month intervention period) |
Post-intervention interviews (20 items) with six staff per nursing home, for a total of 60 interviews to be done after the 6-month intervention period is finished. |
The research team will not impute missing data because missing
quantitative data might only exist in the 20 ACAP surveys collected
at the initiation of pre-intervention period. With such a small
sample size, only descriptive statistics will be computed for the
survey items and imputation of missing data is not needed.
Primary data collection will involve interviews with nursing home staff, and if applicable, prescribing clinicians and pharmacists. Staff to be interviewed will be volunteers at nursing homes that have agreed to participate, so no nonresponse is expected. For regular calls (proactive TA discussions), regular monthly conference calls will be set up. Secondary data will be abstracted by researchers from nursing home records for eligible residents.
Pretesting. All information collection tools and procedures will be vetted by a technical expert panel composed of professional experts with experience in nursing home operations and infection control in nursing homes.
Pilot Test: A pilot test is being conducted in three nursing homes. By the conclusion of the pilot test, researchers will have visited each nursing home two times. At the first visit, they will train nursing home staff how to use the Guide, abstract data from the residents’ records, abstract data from the infection log for the 3 months prior to the training, and administer the ACAP survey to two staff (preferably the administrator and director of nursing). Thus, primary data will be collected from six staff members in total at the three nursing homes. During the pilot, researchers will hold guided proactive TA discussions and will collect technical assistance information and questions if nursing home staff ask for assistance. After the pilot, the researchers will collect data from the infection log, MDS, and medical records.
Steven A. Garfinkel, Project Director, American Institutes for Research, (919) 918-2306
Manshu Yang, Ph.D., Statistician, American Institutes for Research, (919) 918-2312
Darcy McMaughan, Principal Investigator, Department of Health Policy and Management, Texas A&M University School of Rural Public Health, (979) 845-0861
Hongwei Zhao, Statistician, Department of Epidemiology and Biostatistics, Texas A&M University School of Rural Public Health, (979) 845-2387.
i Hayes RJ, Bennett S. Simple sample size calculation for cluster-randomized trials. Int J Epidemiol. 1999 Apr;28(2):319-26. Review. PubMed PMID: 10342698.
ii Phillips CD, Adepoju O, Stone N, Moudouni DK, Nwaiwu O, Zhao H, Frentzel E, Mehr D, Garfinkel S. Asymptomatic bacteriuria, antibiotic use, and suspected urinary tract infections in four nursing homes. BMC Geriatr. 2012 Nov 23;12:73. doi: 10.1186/1471-2318-12-73. PubMed PMID: 23176555; PubMed Central PMCID: PMC3534219.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| Last Modified By | erwin.brown |
| File Modified | 2014-09-03 |
| File Created | 2014-09-03 |