Supporting Statement Part A_REVISED 01.08.20_Clean

SUPPORTING STATEMENT

Part A

Evaluating and Implementing the Six Building Blocks Team Approach to Improve Opioid Management in Primary Care

Version: December 12, 2019

Agency for Healthcare Research and Quality (AHRQ)

Table of contents

A. Justification 3

1. Circumstances that make the collection of information necessary 3

2. Purpose and use of information 10

3. Use of Improved Information Technology 11

4. Efforts to Identify Duplication 11

5. Involvement of Small Entities 11

6. Consequences if Information Collected Less Frequently 11

7. Special Circumstances 12

8. Consultation outside the Agency 12

9. Payments/Gifts to Respondents 12

10. Assurance of Confidentiality 15

11. Questions of a Sensitive Nature 15

12. Estimates of Annualized Burden Hours and Costs 15

13. Estimates of Annualized Respondent Capital and Maintenance Costs 19

14. Estimates of Annualized Cost to the Government 19

15. Changes in Hour Burden 19

16. Time Schedule, Publication and Analysis Plans 20

17. Exemption for Display of Expiration Date 22

List of Attachments 22

A. Justification

1. Circumstances that make the collection of information necessary

The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see https://www.ahrq.gov/policymakers/hrqa99a.html), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:

1. research that develops and presents scientific evidence regarding all aspects of health care; and

2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and

3. initiatives to advance private and public efforts to improve health care quality.

Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.

The project “Evaluating and Implementing the Six Building Blocks Team Approach to Improve Opioid Management in Primary Care” fully supports AHRQ’s mission. The ultimate aim of this project is to further validate and expand the Six Building Blocks to Safer Opioid Management (6BBs) intervention and its associated resources and guidance to support primary care providers in safer opioid prescribing.

Opioid overdose deaths have increased dramatically since 1999¹, and despite recent decreases in the national opioid prescribing rate, prescribing rates remain high in many U.S. counties.² Primary care providers (PCPs) are responsible for about half of all dispensed opioid pain relievers.³

To address the emerging opioid epidemic, the Six Building Blocks to Safer Opioid Management has been developed to support primary care providers in safer opioid prescribing, largely concordant with the Centers for Disease Control and Prevention’s Guideline for Prescribing Opioids for Chronic Pain. The Six Building Blocks to Safer Opioid Management (6BBs) is an evidence-based program that provides a roadmap for primary care clinics to redesign how they provide care to patients with chronic pain who are on long-term opioid therapy. It provides a structured approach to help clinical quality improvement teams revise clinical policies, patient agreements, workflows, develop tracking and monitoring systems, plan for patient-centered visits, identify resources for more complex patients, and measure progress. The six building blocks are:

1. leadership and consensus;

2. policies, patient agreements and workflows;

3. tracking and monitoring patient care;

4. planned, patient-centered visits;

5. caring for complex patients; and,

6. measuring success.

The developers of the 6BBs program also created an accompanying website to describe the elements of the program (https://depts.washington.edu/fammed/improvingopioidcare/), and use of the 6BBs program is described in several publications.⁴

In a prior study of the 6BBs program across 20 rural clinics (see associated publication: Parchman et.al. Ann Fam Med 2019;17:319-325), the rate of decline in both the number of patients on long-term opioids and the proportion of patients on higher dose opioids was greater among intervention clinics that implemented the 6BBs program to make changes in their clinical systems compared to control clinics. A strength of this and other prior 6BBs program studies was that they provided evidence of improvements in both opioid prescribing and staff work life perceptions in clinics that implement the program. On the other hand, a weakness was that the technical assistance provided to intervention clinics-- using a practice coach who provided in-person assistance to clinics to help implement the 6BBs program—was resource-intensive and difficult to scale up.

Therefore, this current project is to develop a How-to-Guide for clinics, and evaluate the How-to-Guide and 6BBs program implementation in clinics that use this Toolkit without an in-person practice coach. The term “Toolkit” in this application refers to both the How-to-Guide and 6BBs program bundled together as a package. The How-to-Guide is a step-by-step manual intended to walk clinics through implementation of the 6BBs program without the need for any additional support, such as from a practice coach.

Table 1 provides an overview of the similarities and differences between the prior study and the current evaluation under review by OMB. 

Table 1. Overview of Prior 6BBs Study vs. Current Evaluation
	Six Building Blocks grant evaluation*	Evaluation of 6BBs + How-To-Guide (ICR Ref. No. 201907-0935-003)
Intervention	Six Building Blocks program Practice facilitator/coach worked with practices, in-person to implement 6BBs	Six Building Blocks program How-to-Guide – a PDF with hyperlinks provided to clinics (self-service model)
Outcomes measured	Number of patients on long-term opioids Proportion of patients on high dosage opioids (MMEs)	Number of patients on long-term opioids Proportion of patients on high dosage opioids (MMEs) Additional QI measures: co-prescribed a benzodiazepine, checked prescription drug monitoring program, checked urine drug screen
Study Design	Quasi-experimental, interrupted time series Comparison group	Implementation-effectiveness design Descriptive comparison between sites
Study Sites	6 health care organizations; 20 clinics 2 states: ID, WA	11 health care organizations; 35 clinics 9 states: WA, CA, AZ, CO, IL, MI, OH, PA, NY

* This evaluation was a research grant from AHRQ and not a contract, i.e. ICR, and as such OMB approval was not required.

Building upon previous work supported by AHRQ to address the opioid epidemic, this research has the following goals:

1. To improve the guidance for the 6BBs Toolkit (How-to-Guide and 6BBs program),

2. To further implement the 6BBs program in primary care practices, and

3. To understand the facilitators and barriers to implementing the 6BBs program for managing patients with chronic pain on long-term opioid therapy.

AHRQ will evaluate the 6BBs Toolkit (How-to-Guide and 6BBs program) to assess:

1. Facilitators of and barriers to implementing the Toolkit.

2. Effectiveness of the Toolkit in improving opioid prescribing practices and relevant outputs and outcomes.

Facilitators and barriers to implementing the Toolkit will be assessed qualitatively through staff interviews and quarterly check-in calls. (See Table 2.) Qualitative data collection will be guided by domains listed in Proctor’s Outcomes for Implementation Research⁵ (Proctor) and the Consolidated Framework for Implementation Research (CFIR).⁶ Regarding the How-to-Guide component of the Toolkit, qualitative data will be collected about the How-to-Guide’s ease of use and completeness of information. Regarding the 6BBs program component of the Toolkit, barriers and facilitators will be identified with respect to the domains outlined in CFIR. These domains include organizational factors such as leadership engagement and available resources.

Effectiveness of the Toolkit will be assessed in the following ways. First, effectiveness of the How-to-Guide will be determined by the extent to which clinics are able to implement the 6BBs program. This will be assessed by: 1) by the proportion of 6BBs program milestones each clinic completes at each of 3 time points throughout the project and 2) implementation progress in Proctor domains assessed qualitatively through staff interviews and quarterly check-in calls. (See Table 2.)

Second, effectiveness of the 6BBs program in improving opioid prescribing practices and relevant outputs and outcomes will be assessed by: 1) the extent to which clinical staff report using recommended clinical practices for managing patients with chronic pain on long-term opioid therapy and 2) by several clinic-level outcomes related to opioid prescribing:

1. Number of patients on long-term opioid therapy

2. Proportion of patients on long-term opioid therapy who are on greater than 90 morphine milligram equivalents (MMEs)

3. Proportion of patients on long-term opioid therapy who are co-prescribed a benzodiazepine

4. Proportion of patients on long-term opioid therapy who had the prescription drug monitoring program (PDMP) checked

5. Proportion of patients on long-term opioid therapy who have had a urine drug screen

The extent to which clinical staff report using recommended clinical practices will be assessed through a clinical staff survey based on the Centers for Disease Control and Prevention (CDC)’s Guideline for Prescribing Opioids for Chronic Pain.⁷

The selected clinic-level outcomes were chosen for several reasons. Two of these (#1 and #2) were used in the original 6BBs study.⁸ Additionally, these are measures released by CDC to support implementation of their evidence-based CDC Guideline for Prescribing Opioids for Chronic Pain. There are actually 16 QI measures from the CDC, but the ones selected above are the ones with the strong evidence (i.e., high MMEs, co-prescribed opioids and benzodiazepines) and well-established practices in the field (i.e., check PDMP and urine drug screen).

Table 2. Evaluation objectives, questions, domains, and data sources
Objective 1: Understand the facilitators and barriers to implementing the 6BBs Toolkit (How-to-Guide and 6BBs program
Questions	Domains	Data Sources
How-to-Guide: How did organizations and clinics use the How-To-Guide? What were their experiences with using the Guide? What were the specific barriers and facilitators to using the Guide?	CFIR domains: Adaptability, complexity, design quality and packaging, executing Proctor domains: Adoption	Staff interviews Quarterly check-in calls Staff interviews Quarterly check-in calls
6BBs program: What were the facilitators/barriers of 6BBs program implementation (e.g. external incentives, available resources, implementation climate)? What were the lessons learned that would be helpful for future clinics to know?	Proctor domains: sustainability CFIR domains: all Determined inductively	Staff interviews Quarterly check-in calls Staff interviews Quarterly check-in calls
Objective 2: Assess the effectiveness of the Toolkit in improving opioid prescribing practices and relevant outputs and outcomes
Questions	Domains	Data Sources
How-to-Guide: Which Building Blocks did practices implement? To what degree? How? How effective was the 6BB guidance and materials in supporting implementation of the 6BBs program? What additional support, guidance, assistance, or information was or would have been needed to support this implementation?	Proctor domains: Adoption, penetration, fidelity 6BBs milestones completed Proctor domains: acceptability, appropriateness, feasibility, adaptability, sustainability potential Proctor domains: acceptability, appropriateness, feasibility, adaptability, sustainability potential	Staff interviews Quarterly check-in calls Secondary data – How-to-Guide worksheet Staff interviews Quarterly check-in calls Staff interviews Quarterly check-in calls
6BBs program: What was the effect of 6BBs program implementation on care processes for patients on chronic opioid therapy? What was the effect of the 6BBs program on clinical staff practices regarding managing patients with chronic pain on long-term opioid therapy? What was the effect of the 6BBs program on clinic-level opioid management outcome measures?	Change in clinical processes: Changes in workflows, patient agreements, dashboards, etc Change in clinical staff practices in managing patients with chronic pain on long-term opioid therapy (determined from CDC Guideline) Change in clinic-level outcomes related to opioid prescribing: Number of patients on long-term opioid therapy Proportion of patients on long-term opioid therapy who are on greater than 90 morphine milligram equivalents (MMEs) Proportion of patients on long-term opioid therapy who are co-prescribed a benzodiazepine Proportion of patients on long-term opioid therapy who had the prescription drug monitoring program (PDMP) checked Proportion of patients on long-term opioid therapy who have had a urine drug screen	Secondary data- organizational documents Clinical staff survey Aggregate reports of QI measures

Evaluation results will be used to inform revisions to the How-to-Guide and disseminate best practices for primary care clinics considering using the 6BBs Toolkit to improve management of patients with chronic pain on long-term opioid therapy. Specifically, qualitative data from interviews and check-in calls will be analyzed using qualitative methods (coded, themes identified, etc.) to evaluate the usefulness of the How-to-Guide, what is not working, and how it could be modified. Findings will be compiled and used to inform modification of the How-to-Guide and prepare it to be publicly-available on AHRQ’s website

The evaluation will take place in 12 health care organizations, in their 35 primary care clinics. Each participating organization will be asked to:

• Use the 6BB Toolkit to implement opioid management improvement initiatives over a period of 15 months based on information presented in the How-To-Guide.

• Conduct and cooperate with assessment activities:

• Collect and report opioid quality measures.

• Collect other metrics to assess the effectiveness of the How-To-Guide.

• Cooperate with project team in the data collection efforts described below.

To achieve the goals of this project the following data collections will be implemented:

1) Clinical Staff Survey (Attachment A). A brief survey will be administered electronically to all clinical staff, including primary care physicians, nurse practitioners, physician assistants, social workers, medical assistants, registered nurses, pharmacists and behavioral health workers, toward the beginning of 6BBs Toolkit implementation and approximately 12 months later. A quality improvement (QI) point person will provide email addresses for the staff who will be invited to complete the survey from each participating organization. These email addresses will be used to send clinical staff the surveys at both time points. The survey will collect information about staff’s self-reported use of evidence-based opioid prescribing practices; procedures in place around opioid prescribing management; self-efficacy regarding safe opioid prescribing; knowledge, beliefs and attitudes regarding opioid prescribing; adaptive reserve; self-reported burnout; and reported implementation experiences. The survey will also collect information about staffs’ background (e.g. clinic role and tenure). The survey will consist largely of closed-ended questions (e.g., scale or Likert response options) with several open-ended questions.

2) Staff Interviews. Interviews will be conducted with 5 staff at each of the 12 participating health care organizations. AHRQ will conduct 2 rounds of interviews, with the first round occurring within several months after the How-To-Guide is distributed to the organization and the second round occurring 12 months later. The evaluation team will conduct in-depth interviews with:

1. The quality improvement (QI) lead (Attachment B) and

2. Four additional staff (Attachment C) who are involved in 6BBs implementation at each organization, that might include a clinician, information technology analyst, social worker, behavioral health specialist, and/or care coordinator.

Staff interviewees will be selected by the QI lead at each organization, who will be asked to nominate a range of staff from those who embraced changes to those who were less willing to implement changes. Interviews will capture qualitative data regarding the organization’s history with efforts to curb opioid prescribing, experiences using the How-To-Guide, implementation of the 6BB intervention and associated opioid management interventions, and lessons learned that can be shared with other health care organizations.

3) Virtual Launch Meeting. A virtual launch meeting will be held for organization liaisons and quality improvement leaders participating health care organizations to launch 6BBs Toolkit implementation. The meeting will be conducted by web-conference, and will last up to 2 hours.

4) Quarterly Check-In Calls. A project team member will hold a quarterly check-in call with organization liaisons and quality improvement leaders to assess the progress of implementation of the 6BBs intervention and improvement initiatives at each organization. Check-in calls will occur quarterly for up to 12 months. Each call will be up to 60 minutes in duration, and notes will be taken by an evaluation team member during each call.

5) QI Measures. Each health care organization will be asked to report quarterly on the number of patients on long-term opioid therapy and the proportion of those who are on greater than 90 morphine milligram equivalents, co-prescribed a benzodiazepine, and had the prescription drug monitoring program checked and a urine drug screen. Organizations may also select other outcome measures aligned to their own goals.

6) Secondary data – organizational documents and worksheets completed by the clinics through their use of the How-to-Guide. Health care organizations will demonstrate their progress on implementing the 6BBs program through completion of worksheets contained in or associated with the How-To-Guide. Since these data collections involve simply submitting worksheets they complete for their own benefit while working through the How-To-Guide, they pose only minimal data collection burden to the health care organization, specifically the person who completes the worksheets (i.e., QI lead). The project team will also obtain relevant organizational documents (e.g., opioid prescribing policies, quality improvement plans, sample patient agreements, relevant practice workflows, screen shots of data dashboards).

This study is being conducted by AHRQ through its contractor, Abt Associates Inc., pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. Purpose and Use of Information

The purpose of the proposed data collection effort is to obtain information needed to modify and enhance the 6BBs Toolkit (How-To-Guide and 6BBs program) and to provide information to health care organizations considering using the Toolkit to improve their ability to manage patients with chronic pain on long-term opioid therapy. Since this is only a study conducted in 12 organizations, outcomes or impacts will not be generalizable.

The data collected will help the project team: 1) understand the facilitators and barriers of using the 6BBs Toolkit (How-to-Guide and 6BBS program) and recommended improvements to processes of care and opioid prescribing practices, and 2) assess the effectiveness of using the 6BBs Toolkit to improve processes of care and opioid prescribing practices. The data collection effort may also provide insights that could guide dissemination of the Toolkit (How-to-Guide and 6BBS program). For example, if it was found that a specific type of organization included in this pilot study (e.g. small, stand-alone clinic in a rural area) particularly benefitted from using the Toolkit, then AHRQ could tailor and target its dissemination of the Toolkit to similar organizations. Once revisions are made based on results of this evaluation, the How-To-Guide will be published on AHRQ’s website. A manuscript describing the pilot study and its results will also be produced for publication in a peer-reviewed journal.

3. Use of Improved Information Technology

A broad range of the data collected during the project will use information technology. The initial launch meeting for health care organizations will be conducted by web-conference to streamline and facilitate participation among the health care organizations. The launch meeting will introduce each element of the 6BBs program, walk through the How-To-Guide, orient participants to the content, and allow organizations to raise questions.

The How-To-Guide will be posted on an active website for the 6BBs intervention (https://www.improvingopioidcare.org/), which participants can access for information at their convenience. The website will also contain links to relevant tools and additional resources health care organizations may find helpful with 6BBs program implementation. This format is ideal for busy health care staff and clinicians who can access the information as they wish. Additionally, the How-To-Guide will contain worksheets in fillable pdf format that health care organizations can use to track their 6BBs program implementation progress.

Also, the clinician survey will be conducted electronically via an online survey application. Staff interviews will be conducted by phone with video conference capability.

4. Efforts to Identify Duplication

The 6BBs Toolkit builds on a previous study supported by the Agency for Healthcare Research and Quality and the Washington State Department of Health, substantially augments the original work, and makes the delivery format more user-friendly. AHRQ has sustained close contact with other organizations doing related work in an effort to identify similar existing information and has not identified any such sources.

5. Involvement of Small Entities

This project does not intend to intentionally involve nor exclude or impact any small entities. However, to the extent an identified and recruited health care organization meets the requirements for participation and is a small entity, we will involve them and expect no greater impact than on other participating health care organizations. The instruments and procedures used to collect data are designed to minimize the burden on all respondents.

6. Consequences if Information Collected Less Frequently

This project is a one-time data collection effort. The data collection period will last for approximately 15 months at each of 12 participating health care organizations.

Not collecting the data, or shortening the data collection period (either by decreasing the study duration or number of sites) places us at risk of not collecting adequate information for the testing of the 6BBs Toolkit. Should we shorten the data collection period, we might not identify potential barriers, facilitators or outcomes of implementing the Toolkit. This would limit the extent to which the final Toolkit would meet health care organizations’ needs related to the improvement of opioid prescribing practices.

7. Special Circumstances

This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register on April 12, 2019 on page 14946 for 60 days (see Attachment D). AHRQ received no substantive comments

8.b. Outside Consultations

The following experts and stakeholders were consulted to identify essential content for the 6BBs Toolkit and to identify appropriate inclusion criteria for health care organizations to be included in the pilot test.

• Michael Parchman, MD, MPH, the MacColl Center for Health Care Innovation at the Kaiser Permanente of Washington Research Institute

• Laura-Mae Baldwin, MD, MPH, University of Washington

• Brooke Ike, MPH, University of Washington

9. Payments/Gifts to Respondents

A range of incentives will be offered as remuneration for clinics’ and individuals’ involvement in the pilot testing of the 6BBs Toolkit in their health care organization:

• Access to a free, high-quality How-To-Guide on implementing change around opioid prescribing practices;

• Opportunity to participate in a opioid prescribing quality improvement project;

• Opportunity to participate in a project funded by AHRQ;

• Potential to improve health care organization’s opioid prescribing practices.

Besides the above-mentioned incentives we propose to offer the following honoraria for clinics and individuals and explain the appropriateness for doing so:

1. An honorarium of $1,000 per clinic is critical to achieve project aims.

Research demonstrates that collecting opioid measure data can be cumbersome for clinics, and providing an honorarium incentivizes clinic participation. As was observed in AHRQ’s EvidenceNOW initiative of nearly 1,500 primary care clinics, there are numerous challenges to pulling measures for quality improvement purposes (Cohen et al., 2018).⁹ CDC-funded projects working with clinics and health systems to improve opioid management practices using QI measures have also reported extensive challenges and time required to produce such measures (Shoemaker-Hunt et al., 2019).¹⁰ The challenges reported by these clinics and health systems included how exactly to operationalize measures (measuring days using calendar days vs. 24-hour periods); accounting for patient populations that should be included and excluded from denominators; accounting for care/services/prescriptions received outside the health system; compiling information contained in unstructured (text) rather than structured fields of the electronic medical record; needing to align measures to state regulations; lacking documentation in the medical record to report on certain measures (e.g. that the clinician counseled the patient on risks and benefits of opioid therapy); and lack of sophisticated information technology staff with the skills and time to pull measures. Furthermore, guidance on Primary Care Practice-Based Research Networks (PBRNs)—many of which AHRQ has supported—indicates that honoraria for participating practices or clinics are critical.¹¹

Regarding the current project, clinics are asked to obtain IRB approval, provide clinic documents, participate in regular check-in calls, and facilitate contacting health care staff for interviews and surveys in addition to collecting and reporting the measures for this evaluation. As such, $1,000 was deemed the appropriate level of honoraria, without undue inducement.

While participating clinics as well as future clinics will need to use resources to implement the 6BBs program, there are several requests of practices participating in this study that are necessary for the purposes of evaluation, but not necessarily borne by future clinics who wish to use the 6BBs. These requests are described below:

• The secondary data refers to health care organizations’ documents that report on their progress implementing the 6BBs program and their use of the How-to-Guide for the evaluation. This could include their QI action plan, meeting notes, new policies or workflows, etc. This also includes, for example, tracking and reporting back on their milestones for each of the building blocks to help the evaluation team understand the extent to which they implemented the 6BBs program.

• The quarterly check-in calls are meant to be quarterly check-in calls with organization liaisons and quality improvement (QI) leaders to assess the progress of implementation of the 6BBs intervention and improvement initiatives at each organization using the How-To-Guide.

• QI measures. While future clinics should ideally use QI measures to monitor their improvements in opioid prescribing over time, the participating practices in this study are asked to build or produce 5 different QI measures in a short period of time in service of the outcome evaluation component. For future clinics, they would likely not prioritize this many measures at one time given the complexity of producing these with electronic health record (EHR) data.

Given these informational needs from participating clinics to support the evaluation, we believe that a $1,000 honorarium per clinic is appropriate, while still requiring time that clinics should bear given the benefits they might accrue from implementing the 6BBs program with guidance from the How-to-Guide.

2. Honoraria for each interview ($50 each for n=60, 2 time points), and each survey ($20 each for n=525, 2 time points) ensure target response rates and appropriate incentive for busy, clinical staff to take the time to complete the interviews and surveys. The honoraria amount are comparable to those used in similar studies.

The team’s belief is that health care organizations and clinicians will be motivated to participate primarily because of their interest in improving their opioid prescribing practices or because of the opportunity to participate in a research project with AHRQ, not because of monetary incentives.

Still, participation in this project and associated data collection activities will place a burden on health care organizations and individuals, and research demonstrates that incentives that compensate for the added burden and costs associated with participation are viewed favorably and are seen as affirming participants’ value and the importance of their participation in research.^12,13

10. Assurance of Confidentiality

Individuals and organizations will be assured of the confidentiality of their replies under Section 944(c) of the Public Health Service Act.  42 U.S.C. 299c-3(c).  That law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

11. Questions of a Sensitive Nature

The data collection protocols do not contain any questions concerning political affiliations and attitudes; respondents’ mental or psychological problems; illegal, antisocial, self-incriminating or demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships; legally privileged relationships; or records describing how an individual exercises First Amendment rights. Nor do they contain questions related to sexual behavior and attitudes, religious beliefs, income or proprietary business information. However, surveys may elicit sensitive information that reflects negatively on staff or health care organization performance related to opioid prescribing. Respondents to the survey will be explicitly informed that their participation is voluntary, information they provide is confidential to the extent provided by law, and they may choose to withdraw from the study or not respond to specific items without penalty. We will also remove individual staff and health care organization names from written interview records and reports to maintain respondent confidentiality.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 presents estimates of the reporting burden hours for the data collection efforts. Time estimates are based on prior experiences and what can reasonably be requested of participating health care organizations. The number of respondents listed in column A, Exhibit 1 reflects a projected 75% response rate for data collection efforts 2a and 2b below.

1. Clinical Staff Survey (Attachment A). A brief survey will be emailed to all clinicians both toward the beginning of 6BBs Toolkit implementation and approximately 12 months later. We assumed 20 clinical staff per clinical site, and 35 clinical sites overall (with a range from 1 clinic to 17 per organization), for a total of 700 staff across all 12 organizations. We assumed 525 clinical staff will complete the survey based on a 75% response rate. It is expected to take up to 15 minutes to complete.

2. Staff Interviews. In-depth interviews will occur with 5 staff at each health care organization, for a total of up to 60 individuals. The evaluation team will conduct these interviews, each lasting up to 1-hour, at 2 points in time with:

   1. One QI lead per organization (toward the start of and at the end of the project). (Attachment B)

Four additional staff (e.g. clinician, information technology analyst, social worker) per organization (midway through and at the end of the project). (Attachment C)

3. Virtual Launch Meeting. The meeting will occur with the quality improvement (QI) leads at participating health care organizations to launch 6BBs Toolkit implementation. The meeting will be conducted by web-conference, and will last up to 2 hours.

4. Quarterly Check-In Calls. Calls will occur with QI leads, clinical champions, and other relevant staff the QI lead identifies, for a total of no more than 3 individuals per organization. These calls will assess progress with the organization’s use of the Toolkit and implementation of associated practice changes, and will occur quarterly over 15 months, for a total of 5 quarterly check-in calls. Each call will take up to 60 minutes.

5. QI Measures. Aggregate reports of the specified quality measures will be provided on a quarterly basis over the course of an 18-month period by a data analyst at each organization, for a total of 12 individuals across all 12 organizations. We assume 40 hours total for each data analyst to collect and provide these data. The QI measures are measures of opioid prescribing that are critical for understanding the potential improvements in opioid prescribing in implementing the 6BBs program using the How-to-Guide. The prioritized measures to monitor improvements in recommended prescribing practices include:

• Number of patients on long-term opioid therapy

• Proportion of patients on long-term opioid therapy who are on greater than 90 morphine milligram equivalents

• Proportion of patients on long-term opioid therapy who are co-prescribed a benzodiazepine

• Proportion of patients on long-term opioid therapy who had the prescription drug monitoring program (PDMP) checked

• Proportion of patients on long-term opioid therapy who have had a urine drug screen

Each health care organization is asked to report quarterly on the QI measures. Clinics may obtain these measures from electronic health record (EHR) data, or they may not have the sophistication or capacity to do that and may track these measures using Excel files or other methods. The method of pulling these measures will vary by clinic, and we will charge clinics will developing a system for collecting these measures that works best for them. We assume it will take each clinic up to 20 total hours to develop a system for pulling these measures, and then subsequently 5 hours to pull and submit these reports each quarter. This would result in a total of 40 hours over the course of the project.

6. Secondary data – organizational documents and worksheets completed by the clinics through their use of the How-to-Guide. These secondary data will be provided by the QI lead at each organization, for a total of 12 individuals across all 12 organizations. We assume 1 hour per month for 12 months for a total of 12 hours for each QI lead to collect and provide these data. The secondary data refers to health care organizations’ documents that report on their progress implementing the 6BBs Toolkit (How-to-Guide and 6BBs program) in service of the evaluation (see description on pg. 10).

Exhibit 1. Estimated annualized burden hours

Data Collection Method or Project Activity	A. Number of respondents	B. Number of responses per respondent	C. Hours per response	D. Total burden hours
1. Clinical Staff Survey*	525	2	15/60	263
2a. Staff Interview – QI Lead	12	2	1	24
2b. Staff Interview – Additional Staff	48	2	1	96
Virtual Launch Meeting	12	1	2	24
Quarterly Check-In Calls	36	5	1	180
5. QI Measures – develop system	12	1	20	240
5. QI Measures – pull reports	12	4	5	240
6. Secondary data	12	12	1	144
TOTAL	669	n/a	n/a	1,211

*Number of respondents (Column A) reflects a sample size assuming a 75% response rate for this data collection effort.

Exhibit 2, below, presents the estimated annualized cost burden associated with the respondents’ time to participate in this research. The total cost burden is estimated to be about $88,857.

Exhibit 2. Estimated annualized cost burden

Data Collection Method or Project Activity	Number of respondents	Total burden hours	Average hourly wage rate*	Total cost burden
1. Clinical Staff Survey	525	263	$98.04	$25,785
2a. Staff Interview – QI Lead	12	24	$109.36	$2,625
2b. Staff Interview – Additional Staff	48	96	$78.84	$7,569
3. Virtual Launch Meeting	12	24	$109.36	$2,625
4. Quarterly Check-In Calls	36	180	$78.84	$14,191
5. QI Measures – develop system	12	240	$42.32	$10,157
5. QI Measures – pull reports	12	240	$42.32	$10,157
6. Secondary data	12	144	$109.36	$15,748
Total				$88,857

The average hourly rate of $98.04 for the clinical staff survey was calculated based on the 2018 mean hourly wage rate for health diagnosing and treating practitioners, $49.02 (occupation code 29-1000), doubled to account for employer overhead and fringe benefits.

The average hourly rate of $109.36 for QI lead interviews was calculated based on the 2018 mean hourly wage rate for medical and health services managers, $54.68 (occupation code 11-9111), doubled to account for employer overhead and fringe benefits. The average hourly rate of $78.84 for staff interviews was calculated based on the 2018 mean hourly wage rate for healthcare practitioners and technical occupations, $39.42 (occupation code 29-0000), doubled to account for employer overhead and fringe benefits.

The average hourly rate of $109.36 for the virtual launch meeting was calculated based on the 2018 mean hourly wage rate for medical and health services managers, $54.68 (occupation code 11-9111), doubled to account for employer overhead and fringe benefits.

The average hourly wage rate of $78.84 for quarterly check-in calls was calculated based on the 2018 mean hourly wage rate for healthcare practitioners and technical occupations, $39.42 (occupation code 29-0000), doubled to account for employer overhead and fringe benefits.

The average hourly rate of $42.32 for QI measures was calculated based on the 2018 mean hourly wage rate for medical records and health information technicians, $21.16 (occupation code 29-2071), doubled to account for employer overhead and fringe benefits.

The average hourly rate of $109.36 for secondary data was calculated based on the 2018 mean hourly wage rate for medical and health services managers, $54.68 (occupation code 11-9111), doubled to account for employer overhead and fringe benefits.

Mean hourly wage rates for these groups of occupations were obtained from the Bureau of Labor & Statistics on “Occupational Employment and Wages, May 2018” found at the following URL: http://www.bls.gov/oes/current/oes_nat.htm#b29-0000.htm

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

Exhibit 3.  Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Toolkit Development	$381,656	$190,828
Data Collection Activities	$137,000	$68,500
Data Processing and Analysis	$137,000	$68,500
Publication of Results	$87,000	$43,500
Project Management	$294,221	$147,111
Total	$1,036,877	$518,439

Exhibit 4. Government Personnel Cost

Tasks/Personnel	Annual Salary	% of Time	Cost
PRE OMB Approval Costs
Government Personnel Costs
Social Science Analyst – GS15*, Step 9	$164,200	1%	$1,642
POST OMB Approval Costs
Government Personnel Costs
Social Science Analyst – GS15*, Step 9	$164,200	2%	$3,284
Grand Total			$4,926

*Based on 2019 OPM Pay Schedule for Washington/DC area: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2019/general-schedule/

15. Changes in Hour Burden

This is a new information collection.

16. Time Schedule, Publication and Analysis Plans

Exhibit 5 Project Timeline

Description (in chronological order)	Due Date
Identify potential health care organizations	November-January 2019
Final list of recommended health care organizations	January 2019
Finalize 6BBs How-To-Guide	March 2019
Practices implement 6BBs Toolkit	March 2019 – September 2020
Virtual launch meeting	April 2019
Quarterly check-in calls	April 2019 – June 2020
Secondary data collection	June 2019 (upon OMB approval) – March 2020
Organizations report QI measures quarterly	June 2019 (upon OMB approval) – March 2020
First round of clinical staff surveys	June 2019 (upon OMB approval)
First round of interviews with QI leads	June 2019 (upon OMB approval)
First round of interviews with additional staff	December 2019
Second round of clinical staff surveys	May 2020
Second round of interviews with QI leads and additional staff	May 2020
Complete analysis	June 2020 - July 2020
Draft report	August 2020
Revised 6BBs Toolkit	August 2020
Final report	September 2020
Final 6BBs Toolkit	September 2020

Publication Plan:

Study results will be disseminated through a peer-reviewed publication. The final How-To-Guide will be posted on the appropriate section of the AHRQ web site and disseminated via AHRQ’s Office of Communication and Knowledge Transfer (e.g., e-mails to relevant professional associations and postings on listservs).

Analysis Plan:
As described above, the purpose of this data collection is twofold: 1) Understand the facilitators and barriers of using the 6BBs Toolkit and recommended improvements to processes of care and opioid prescribing practices, and 2) Assess the effectiveness of using the 6BBs Toolkit to improve processes of care and opioid prescribing practices. AHRQ has proposed to use multiple data sources to triangulate findings to meet both of these goals.

The data analysis strategies therefore differ, each of which are described below:

Goal 1: Understand the facilitators and barriers of using the 6BBs Toolkit and recommended improvements to processes of care and opioid prescribing practices.

Data collection strategy: Staff interviews, quarterly check-in calls, secondary data.

Data analysis strategy: Qualitative synthesis complemented with quantitative data from clinical staff surveys.

Data will be primarily analyzed qualitatively, identifying themes of facilitators and barriers to implementing the 6BBs Toolkit and improvements to health care organizations’ processes of care and opioid prescribing practices. Qualitative analysis software (NVivo) will be used to synthesize and analyze the data as well as to allow for qualitative comparisons and synthesis by each organization, organization type (e.g., single versus multi-clinic organization), staff type (e.g., QI lead, clinical provider), geographic location, and in aggregate. Qualitative comparisons of data from the first to the second round assessments of opioid prescribing processes will be similarly analyzed. The insights regarding facilitators and barriers to implementation will be used in the final 6BBs Toolkit to describe how other health care organizations might implement changes more smoothly, and may also inform How-To-Guide content (e.g., how to obtain buy-in from leadership).

Goal 2: Assess the effectiveness of using the 6BBs Toolkit to improve processes of care and opioid prescribing practices.

Data collection strategy: Clinical staff survey, QI measures, secondary data.

Data analysis strategy: Quantitative analyses complemented with qualitative data from staff interviews.

Quantitative data will be collected through two rounds of the clinical staff survey, QI measures, and secondary data sources describing the extent of 6BBs implementation. AHRQ will analyze these data using descriptive statistics (i.e., frequencies, averages) by health care organization and health care organization type to allow for comparisons. Additional analysis will include comparisons of survey data, QI measures, and extent of 6BBs implementation to measure changes over time. Analysis sub-goals for each set of instruments and analysis plans are summarized in Exhibit 6, below.

Exhibit 6. 6BBs Toolkit Implementation Data Collection and Analysis Plans

Instrument	When administered and to whom	Analysis sub-goal	Analysis Plan
Clinical Staff Survey (Attachment A)	Toward the beginning of implementation and 12 months after Clinical staff	Assess opioid prescribing practices and readiness to change; reactions to and evaluation of 6BBs Toolkit (post only)	Descriptive statistics (i.e., frequencies, average) Qualitatively analyze open-ended comments Paired t-test to compare rates of favorable prescribing practices from round 1 to round 2.
QI measures	Collected quarterly throughout implementation QI leads/data analysts	Assess change in opioid prescribing practices and processes of care	Descriptive statistics (i.e., frequencies, average) Scatterplot of measures over time by organization and overall
Secondary data and other outcome data (e.g., surgical cancellation and delay rates)	Toward the start, middle, and end of implementation	Assess extent of 6BBs implementation	Descriptive statistics (counts, frequencies, averages); comparison of implementation milestones completed at each time point by organization and average across all organizations

17. Exemption for Display of Expiration Date

AHRQ does not seek this exemption.

List of Attachments:

Attachment A -- Clinical Staff Survey

Attachment B -- Staff Interview Guide – QI Lead

Attachment C -- Staff Interview Guide – Additional Staff

Attachment D -- 60-Day Federal Register Notice

1 Seth P, Scholl L, Rudd RA, Bacon S. Increases and Geographic Variations in Overdose Deaths Involving Opioids, Cocaine, and Psychostimulants with Abuse Potential – United States, 2015-2016. MMWR Morb Mortal Wkly Rep. ePub: 29 March 2018.

2. ² Centers for Disease Control and Prevention. U.S. County Prescribing Rates, 2017. https://www.cdc.gov/drugoverdose/maps/rxcounty2017.html Published July 2017. Accessed December 14, 2018.

3 Daubresse M, Chang HY, Yu Y, Viswanathan S, Shah ND, Stafford RS, Kruszewski SP, Alexander GC. Ambulatory diagnosis and treatment of non-malignant pain in the United States, 2000–2010. Medical care. 2013 Oct;51(10).

4 Parchman ML, Von Korff M, Baldwin LM, Stephens M, Ike B, Cromp D, Hsu C, Wagner EH. Primary care clinic re-design for prescription opioid management. The Journal of the American Board of Family Medicine. 2017 Jan 1;30(1):44-51.; Parchman ML, Penfold RB, Ike B, Tauben D, Von Korff M, Stephens M, Stephens KA, Baldwin LM. Team-Based Clinic Redesign of Opioid Medication Management in Primary Care: Effect on Opioid Prescribing. The Annals of Family Medicine. 2019 Jul 1;17(4):319-25.; Ike B, Baldwin LM, Sutton S, Van Borkulo N, Packer C, Parchman ML. Staff and Clinician Work-Life Perceptions after Implementing Systems-Based Improvements to Opioid Management. The Journal of the American Board of Family Medicine. 2019 Sep 1;32(5):715-23.

5 Proctor, Enola, Hiie Silmere, Ramesh Raghavan, Peter Hovmand, Greg Aarons, Alicia Bunger, Richard Griffey, and Melissa Hensley. "Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda." Administration and Policy in Mental Health and Mental Health Services Research 38, no. 2 (2011): 65-76.

6 Damschroder, Laura J., David C. Aron, Rosalind E. Keith, Susan R. Kirsh, Jeffery A. Alexander, and Julie C. Lowery. "Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science." Implementation science 4, no. 1 (2009): 50.

7 Available at: https://www.cdc.gov/drugoverdose/pdf/prescribing/CDC-DUIP-FactSheet-At-A-Glance_Opioid-Measures-508.pdf

8 Parchman et.al. Ann Fam Med 2019;17:319-325

9 Cohen DJ, Dorr DA, Knierim K, DuBard CA, Hemler JR, Hall JD, Marino M, Solberg LI, McConnell KJ, Nichols LM, Nease Jr DE. Primary care practices’ abilities and challenges in using electronic health record data for quality improvement. Health Affairs. 2018 Apr 1;37(4):635-43.

10 Shoemaker-Hunt, SJ. (June 2, 2019). Observations from a QI Collaborative of Healthcare Systems Implementing the Opioid QI Measures for Improvement Purposes. Podium presentation as part of the panel session on: Opioid Measurement for Improving Care: Findings from the CDC Opioid Quality Improvement Collaborative. AcademyHealth Annual Research Meeting. Washington, DC.

11 Please see Tables 2 & 3 in Dolor, Schmit & Baldwin et al (2014) Guidance for Researchers Developing and Conducting Clinical Trials in Practice-based Research Networks (PBRNs) available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4297606/#!po=41.6667

12 Russell, M.L., Moralejo, D.G. and Burgess, E.D., 2000. Paying research subjects: participants' perspectives. Journal of medical ethics, 26(2), pp.126-130.

13 Groth, S.W., 2010. Honorarium or coercion: use of incentives for participants in clinical research. The Journal of the New York State Nurses' Association, 41(1), p.11.