SUPPORTING STATEMENT
Part B
Evaluating and Implementing the Six Building Blocks Team Approach to Improve Opioid Management in Primary Care
February 1, 2019
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods
2. Information Collection Procedures
3. Methods to Maximize Response Rates
4. Tests of Procedures
5. Statistical Consultants
Twelve health care organizations will be recruited to implement the Six Building Blocks to Safer Opioid Management (6BBs) intervention to help improve the guidance for the 6BBs Toolkit and better understand the facilitators and barriers to implementing the 6BBs intervention in U.S. health care organizations. There are approximately 327,000 primary care providers in the U.S.—about 70% of those work in physician-owned organizations and about 26% work in organizations owned by insurers, health plans, academic entities, or community health centers.1 Sites will be selected to maximize diversity of health care organization characteristics using a targeted, purposive recruitment approach. The purposive recruitment strategy will also allow us to attract health care organizations whose leadership is already open to improving their opioid prescribing practices and processes. These health care organizations represent the 6BBs Toolkit’s intended audience, as health care organizations resistant to changing their opioid prescribing practices and processes are unlikely to use the 6BBs Toolkit. The recruitment strategy will be conducted as follows:
Step 1: Inform health care systems of the opportunity to participate. Our first strategy to achieve this goal will be to post a description of the project on AHRQ’s website. AHRQ’s website is accessed by hundreds of health care organization representatives on a weekly basis. AHRQ maintains a listserv of health care organizations that also will be used to post a request for participation. The website link and listserv post will include a brief description of the project, the benefits of participation (e.g., opportunity to improve opioid prescribing practices and processes), and the types of health care organizations we seek. Contact information will be provided for interested health care organizations to notify Olivia Bacon at Abt Associates, Inc. of their potential interest. Additionally, we will conduct active outreach through our professional contacts and through experts on the 6BBs intervention listed in Supporting Statement A.
Step 2: Obtain expressions of interest from health care organization representatives that include willingness to meet the participation requirements described in Supporting Statement A (e.g., cooperation with data collection efforts).
Step 3: Exclude health care organizations that do not demonstrate adequate readiness that would affect the likelihood they could engage leaders and staff to implement the 6BBs Toolkit and attempt changes to their opioid prescribing practices and processes (e.g., major change happening concurrently, like a change in electronic health record vendor, inability to implement changes within project time period, limited or no leadership buy-in at the health care organization or from key clinicians within the organization).
Step 4: Select from the remaining health care organizations to ensure diversity on key characteristics. Selection will be made to ensure diversity on key characteristics, including: health care organization type (e.g., academic health center, safety-net or community health center, multi-clinic versus single clinic organization), size, populations served (e.g., racial/ethnic minorities, Medicaid recipients), geographic region, and urban/rural location. Each health care organization will be asked to designate a staff member as a liaison to this project. We suspect an individual involved in the health care organization’s quality improvement efforts might be best suited for organizational liaison; however, we will ultimately allow the organization to decide who would best champion and facilitate changes in its opioid prescribing practices and processes.
We make no claim that the results from this study will be generalizable. Rather, our small sample of information-rich cases will be illustrative of the kinds of barriers, facilitators and results that health care organizations may experience in implementing the 6BBs Toolkit, and will generate insights about needed improvements.
Health care organization characteristics |
Number of target organizations with this characteristic |
Academically affiliated or teaching |
4 |
Safety-net or FQHC |
4 |
Multi-clinic organization |
6 |
Single clinic organization |
6 |
Urban location |
6 |
Rural location |
6 |
Sample Size
Number of health care organizations
Budget constraints will limit the number of participating sites to 12 health care organizations with a range of between 1 and 17 primary care clinics per organization. While it is always preferable to have a larger number of participating sites, we anticipate that 12 sites will provide us with sufficient information to illustrate the types of barriers, facilitators and results that may be expected with 6BBs implementation, and to identify needed revisions to the 6BBs Toolkit content and format.
QI Measures. Aggregate reports for 4 opioid prescribing QI measures will be collected quarterly from each of the 12 health care organizations during 6BBs Toolkit implementation. AHRQ will collect the following 4 measures: the number of patients on long-term opioid therapy and the proportion of those who are on greater than 90 morphine milligram equivalents, co-prescribed a benzodiazepine, had the prescription drug monitoring program checked, and had a urine drug screen. We will ask each organization to report these measures at the clinic level. For example, if an organization has 5 primary care clinics, we will request these measures be reported for each clinic.
Sample sizes within each health care organization
Primary care clinics
Within
selected health care organizations, organization liaisons will work
with the project team to select up
to 33 primary care clinics
for 6BBs Toolkit implementation.
Health care organization staff
Health care professionals and staff within the selected primary care clinics will participate in the project by participating in 6BBs Toolkit implementation. All health care professionals working in the selected clinics who are involved in opioid prescribing processes (e.g., doctors, nurse practitioners, nurses, analysts, social workers) and/or quality improvement efforts for opioid prescribing will be eligible to participate. The number of persons participating is likely to vary considerably across health care organizations and clinics. Sample sizes provided below are rough estimates.
Clinical Staff Survey. A brief survey will be emailed to all clinicians toward the beginning of when their organization receives the Toolkit and approximately 12 months later. We assumed 20 clinical staff per clinical site, and up to 33 clinical sites, for a total of 660 staff across all 12 organizations. We assumed 495 clinical staff will complete the survey based on a 75% response rate. It is expected to take approximately 15 minutes to complete.
Proposed analyses and statistical power calculations
As described in Summary Statement Part A, some of the data collected will be analyzed using quantitative methods. These quantitative analyses include univariate statistics (e.g., average clinician self-efficacy around safe opioid prescribing) and, where appropriate, statistical tests to assess differences in survey responses between the beginning and end of 6BBs Toolkit implementation.
Most data will be analyzed at the health care organization level, and consist primarily of descriptive statistics due to the small sample size (n=12). The quantitative analysis approach will entail running descriptive statistics – mean, median, standard deviation, and plots of distributions for continuous variables, and frequency tables and plots for categorical variables. Likewise, we will descriptively assess organizational-level changes in QI measures over time.
We anticipate receiving a total of 495 completed clinical staff surveys across 12 organizations at both the beginning and end time points. Survey data will be pooled across organizations to perform statistical tests. We will use paired-samples t-tests to compare continuous and Likert-scale outcomes, such as clinician self-efficacy and number of strategies used to manage patients on opioids, between the two time points. We will use binomial tests to compare binary outcomes, including whether or not clinicians report using a patient registry and whether or not clinicians report being waivered to prescribed buprenorphine, between the two time points.
Exhibit 1, below, summarizes for each data source the type of variable used, statistical approach, and required sample sizes assuming 80% statistical power level, significance level of 0.05, and explicit assumptions about the standard deviation (SD). All statistical tests will be computed in SAS.
Exhibit 1. 6BBs Evaluation Sample Size, Statistical Tests and Power Calculations
Instrument |
Variables used |
Statistical tests |
Anticipated Sample size (pooled across organizations) |
Minimum detectable effect size at 80% statistical power level and 5% alpha |
Clinical Staff Survey (Attachment A) |
Average scores (0-5) for several domains:
Average number of strategies:
Binary variables (yes/no):
|
Paired t-tests
Binomial tests (for binary variables)
|
495 toward start 495 toward end
|
0.13 point change on all variables assuming SD=1
Pre-post
change of 13% assuming SD=1
|
QI Measures |
|
Statistics to include mean, median, standard deviation, frequencies, histograms |
33 |
n/a |
*Chronic opioid therapy
Inclusion and selection criteria
Health care organizations will be chosen based on their willingness to participate and readiness, as described above. Since we expect that health care organization characteristics may impact implementation, selection will be made to ensure diversity on key characteristics, including: academic health center, safety-net or community health center, multi-clinic versus single clinic organization), size, populations served (e.g., racial/ethnic minorities, Medicaid recipients), geographic region, and urban/rural location.
Within each health care organization, organizational liaisons in collaboration with leadership will select up to 33 primary care clinics for 6BBs Toolkit implementation. All clinical staff in the participating primary care clinics will be eligible to participate in the clinical staff survey. Clinical staff will be invited to participate in the clinical staff survey by email. We will obtain a list of email addresses for clinical staff from the organizational liaison. Our survey recruitment strategy will include a 3-step email process. First, the Medical Director at each organization will email clinical staff to introduce the survey and stress the importance of completion, informing staff that a follow-up email will be sent from a member of the project team and will contain a link to the online survey. The second email will reintroduce the survey and contain the live survey link. A third, reminder email will be sent to all clinical staff who did not respond about a week after the second email reminding them to complete the survey if they have not done so, and will again contain the live survey link. The online survey is estimated to take no longer than 15 minutes to complete.
Health care organization leaders, for example the Medical Director and QI Lead, will be asked to encourage completion of the Clinical Staff Survey, which should help maximize the response rate. Although clinician surveys (conducted online or using other methods) often suffer from low response rates, AHRQ expects a better than average return rate for this survey because the distribution will be targeted, using the email addresses of clinicians working in clinics that are implementing the 6BBs Toolkit. Also, we will use a recommended, 3-step email approach demonstrated to improve response rates.2
All of the data collection protocols have been reviewed by the project partners (see: Supporting Statement A, Section 8.b. Outside Consultants) with experience obtaining, improving and studying opioid prescribing practices and processes. This review helps establish the face and content validity of the protocols. Additionally, to the extent possible, the data collection protocols use or adapt existing items from previously tested and validated instruments.
The items for the Clinical Staff Survey were developed based on the content in the 6BBs Toolkit. These questions are similar to questions used in other training evaluations. The survey will be pre-tested with several staff from AHRQ’s contractors (Abt Associates and the University of Washington) who are clinicians to identify wording issues, verify the length of time needed to complete the survey, ensure the items reflect the Toolkit content, and identify response options that should be edited.
Abt Associates is the contractor who will facilitate health care organizations’ data collection and analysis on behalf of AHRQ. The professionals from Abt Associates have over 40 years of experience providing high quality, timely and cost effective data collection for federal agencies. Abt Associates employs many statisticians, economists and experienced research methodologists. Statistician and health economist Lauren Olsho, Ph.D., from Abt Associates, was consulted and reviewed the proposed statistical analyses. Dr. Olsho has designed several rigorous, practice-based research studies for AHRQ and other federal agencies. She is available should any questions regarding the statistical analyses for this project arise. The key project contact at Abt Associates is Sarah Shoemaker-Hunt.
Key contact information for Sarah Shoemaker-Hunt is provided below:
Sarah J. Shoemaker-Hunt, PhD, PharmD, Principal Associate/Scientist
Abt Associates Inc.
10 Fawcett Street
Cambridge, MA 02138
Office: 617.349.2472
Email: sarah_shoemaker@abtassoc.com
1 Petterson S, McNellis R, Klink K, Meyers D, Bazemore A. The State of Primary Care in the United States: A Chartbook of Facts and Statistics. January 2018. Accessed at https://www.graham-center.org/content/dam/rgc/documents/publications-reports/reports/PrimaryCareChartbook.pdf
2 Dillman, D.A., 1978. Mail and telephone surveys: The total design method (Vol. 19). New York: Wiley.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | SUPPORTING STATEMENT |
| Author | wcarroll |
| File Modified | 0000-00-00 |
| File Created | 2021-01-15 |