SUPPORTING STATEMENT
Part A
TeamSTEPPS 2.0 Online Master Trainer Course
OMB CONTROL NO. 0935-0224
Version: April 6, 2017
Agency for Healthcare Research and Quality
Table of Contents
1. Circumstances That Make the Collection of Information Necessary 3
2. Purpose and Use of Information 5
3. Use of Improved Information Technology 5
4. Efforts to Identify Duplication 5
5. Involvement of Small Entities 6
6. Consequences if Information Collected Less Frequently 6
8. Federal Register Notice and Outside Consultations 6
9. Payments/Gifts to Respondents 7
10. Assurance of Confidentiality 7
11. Questions of a Sensitive Nature 8
12. Estimates of Annualized Burden Hours and Costs 8
13. Estimates of Annualized Respondent Capital and Maintenance Costs 9
14. Estimates of Total and Annualized Cost to the Government 9
16. Time Schedule, Publication, and Analysis Plans 10
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, the Healthcare Research and Quality Act of 1999 (see www.ahrq.gov/hrqa99.pdf), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
Research that develops and presents scientific evidence regarding all aspects of health care;
The synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policymakers, and educators; and
Initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) elderly people, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
As part of its effort to fulfill its mission goals, AHRQ, in collaboration with the U.S. Department of Defense’s TRICARE Management Activity, developed TeamSTEPPS® (aka, Team Strategies and Tools for Enhancing Performance and Patient Safety) to provide an evidence-based suite of tools and strategies for training teamwork-based patient safety to health care professionals. TeamSTEPPS includes multiple toolkits, which are all tied to or are variants of the core curriculum. TeamSTEPPS resources have been developed for primary care, rapid response systems, long-term care, and patients with limited English proficiency.
The main objective of the TeamSTEPPS program is to improve patient safety by training health care staff in various teamwork, communication, and patient safety concepts, tools, and techniques and ultimately helping to build national capacity for supporting teamwork-based patient safety efforts in health care organizations. Since 2007, AHRQ’s National Implementation Program has produced (and continues to produce) Master Trainers who have stimulated the use and adoption of TeamSTEPPS in health care delivery systems. These individuals were trained during two-day, in-person classes using the TeamSTEPPS core curriculum at regional training centers across the U.S. AHRQ has also provided technical assistance and consultation on implementing TeamSTEPPS and has developed various channels of learning (e.g., user networks, various educational venues) for continued support and the improvement of teamwork in health care. Since the inception of the National Implementation Program, AHRQ has trained more than 6,000 participants to serve as TeamSTEPPS Master Trainers.
Despite the success of the National Implementation Program and the availability of training through this initiative, AHRQ had been unable to match the demand for TeamSTEPPS Master Training. Wait lists for training at times exceeded 500 individuals.
To address this prevailing need, AHRQ launched an effort to develop and provide TeamSTEPPS training online. This program, known as TeamSTEPPS 2.0 Online Master Trainer course, mirrors the TeamSTEPPS 2.0 core curriculum and provides equivalent training to the in-person classes offered through the National Implementation Program.
As part of this initiative, AHRQ seeks to continue to conduct an evaluation of the TeamSTEPPS 2.0 Online Master Trainer program. This evaluation seeks to understand the effectiveness of TeamSTEPPS 2.0 Online Master Training and what revisions might be required to improve the training program.
This research has the following goals:
1) Conduct a formative assessment of the TeamSTEPPS 2.0 Online Master Trainer program to determine what improvements should be made to the training and how it is delivered, and
2) Identify how trained participants use and implement the TeamSTEPPS tools and resources.
To achieve this project’s goals, AHRQ will train participants using the TeamSTEPPS 2.0 Online Master Trainer program and then survey these participants six months post-training. Each activity is briefly described below.
TeamSTEPPS 2.0 Online Master Trainer Course. This training program, which includes 13 accredited hours of training, is based on the TeamSTEPPS 2.0 instructional materials and will be delivered online to 3,000 participants. The training will cover the core TeamSTEPPS tools and strategies, coaching, organizational change, and implementation science.
TeamSTEPPS 2.0 Online Post-Training Survey. This online instrument will be administered to all participants who complete the TeamSTEPPS 2.0 Online Master Training. The survey will be administered six months after participants complete the training program. (see Attachment A – TeamSTEPPS 2.0 Online Post-Training Survey)
The TeamSTEPPS 2.0 Online Master Trainer program is led by Reingold, Inc. (Reingold). This study is being conducted by Reingold’s subcontractor, IMPAQ International (IMPAQ). This study is being conducted pursuant to AHRQ’s statutory authority to conduct and support research on health care and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness, and value of health care services and with respect to quality measurement and improvement, 42 U.S.C. 299a(a)(1) and (2).
This is a continuation of data collection for the purpose of conducting an evaluation of the TeamSTEPPS 2.0 Online Master Trainer program. The evaluation will be primarily formative in nature as AHRQ seeks information to improve the delivery of the training.
To conduct the evaluation, the TeamSTEPPS 2.0 Online Post-Training Survey will be administered to all individuals who completed the TeamSTEPPS 2.0 Online Master Trainer program, six months after completing training. The purpose of the survey is to assess the degree to which participants felt prepared by the training and what they did to implement TeamSTEPPS. Specifically, participants will be asked about their reasons for participating in the program; the degree to which they feel the training prepared them to train others in and use TeamSTEPPS; what tools they have implemented in their organizations; and resulting changes they have observed in the delivery of care.
The TeamSTEPPS 2.0 Online Post-Training Survey will be administered via the Web to participants.
In order to reduce respondent burden, the training participant questionnaire will be administered via the Web. The participant information acquired by Reingold when participants enroll in the TeamSTEPPS 2.0 Online Master Trainer program will be used to develop the questionnaire’s distribution lists. Each potential respondent will receive up to five e-mail communications to encourage participation (i.e., an advance notice of the questionnaire, an initial invitation to complete the questionnaire, and three follow-up e-mails to remind respondents to complete the questionnaire).
Using an online system for data collection, rather than administering a paper-based questionnaire, makes completing and submitting the questionnaire less time-consuming for respondents. Any skip patterns included in the questionnaire (i.e., questions that are appropriate only for a subset of the respondents) will be automatically programmed into the Web-based form of the questionnaire, thereby eliminating any confusion during questionnaire completion. In addition, the contractors can also ensure that important items are not inadvertently skipped or ignored by setting software requirements to ensure proper completion of questionnaires based on specific respondent selections.
AHRQ is aware of two other evaluations of the TeamSTEPPS suite of training programs, each requiring an Office of Management and Budget (OMB) clearance package. Each project is evaluating a different TeamSTEPPS training program and will collect data from distinct populations of individuals who participate in training. Each of these two other evaluations is described below.
Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program. This effort provides in-person training to participants using the TeamSTEPPS 2.0 core curriculum. AHRQ is administering a version of the post-training survey to these participants. The survey used for this effort has obtained initial and renewed OMB clearance (#0935-0170).
Implementation of TeamSTEPPS in Primary Care Settings. This effort provides online training to participants using the TeamSTEPPS for Office-Based curriculum. AHRQ proposes to continue to conduct a formative evaluation of the TeamSTEPPS for Office-Based Care program using a version of the post-training survey tailored to this group of training participants. The survey obtained OMB approval (#0935-0222) and will be undergoing review for OMB renewal.
Besides the two evaluation efforts listed above, AHRQ intends to continue to collect similar data on the TeamSTEPPS 2.0 Online Master Training project. This effort (described in this renewal submission) provides online training to participants using a newly developed online version of the TeamSTEPPS 2.0 core curriculum. AHRQ proposes to continue to conduct a formative evaluation of the TeamSTEPPS 2.0 Online Master Trainer program. The survey approved for the Assessing the Impact of the National Implementation of TeamSTEPPS Master Training Program effort was modified slightly to align with this program’s target participants and curriculum.
The information collected may involve small entities, as TeamSTEPPS 2.0 Online Master Trainer course participants may work for very large or small hospitals or practices. All participants will be required to complete the same number of items as large practices. For this study, only items that provide critical information for answering the study questions will be included.
This is a one-time collection to answer specific questions about the TeamSTEPPS 2.0 Online Master Trainer program.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on May 18th, 2017 on Page 22828 for 60 days (see Attachment C). There were no substantive comments received for this notice.
This project is part of AHRQ’s patient safety portfolio and supports AHRQ’s TeamSTEPPS program. AHRQ consulted with the Health Research & Educational Trust (HRET), the contractor leading AHRQ’s National Implementation of TeamSTEPPS project. HRET is also a contractor on the TeamSTEPPS 2.0 Online Master Training project. HRET confirmed that this is a unique, one-time data collection focused on the TeamSTEPPS 2.0 Online Master Trainer curriculum.
There are no unresolved issues or conflicts associated with this initiative.
No remuneration of respondents is planned.
Individuals will be assured of the confidentiality of their replies is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify any individual respondent will not be disclosed outside of the Department of Health and Human Services unless the respondent has consented to that disclosure.
Information that can directly identify the respondent, such as a name and/or social security number, will not be collected. Only basic demographic information related to the individual’s organization and position will be collected for the purpose of describing the respondents. Participation will be voluntary and participants will be informed that their responses will be confidential. The following statement of confidentiality will appear on the initial screen of the Web-based survey and in email correspondence:
“This survey is authorized under 42 U.S.C. 299a. The confidentiality of your survey responses is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)]. Information that could identify you will not be disclosed outside of the Department of Health and Human Services without your consent. If needed, AHRQ may contact you in the future to request additional information about your experiences with TeamSTEPPS.”
Reingold’s subcontractor, IMPAQ, will collect all information. Response data will be stored on password-protected secure servers at IMPAQ. Only members of the IMPAQ project team will have access to the password-protected secure server. Only aggregated data will be included in reports submitted to AHRQ. A data file containing individual-level responses to the survey will be delivered to the AHRQ Task Order Officer at the end of the data collection effort.
Questionnaire items do not require respondents to provide information of a sensitive nature as defined by OMB and DHHS or to provide information such as social security numbers or Medicare/Medicaid numbers. The TeamSTEPPS 2.0 Online Post-Training Survey includes an introduction that addresses aspects of informed consent such as a description of the research objectives, a discussion of the importance of respondents’ input and experiences, details concerning how the data will be used, and confidentiality of personal information. The introduction will be positioned at the beginning of the questionnaire. Continuation to complete the questionnaire will indicate the respondent’s consent.
No questions of a sensitive nature will be asked on the TeamSTEPPS 2.0 Online Post-Training Survey. Although written consent will not be required, all respondents will be informed about the nature of the study and that their participation is voluntary. There are no known consequences of participation, and all confidentiality procedures will be described.
Exhibit 1 shows the estimated annualized burden hours for the respondent’s time to participate in the study. The TeamSTEPPS 2.0 Online Post-Training Survey will be completed by approximately 3,000 individuals. We estimate that each respondent will require 20 minutes to complete the survey. The total annualized burden is estimated to be 1,000 hours.
Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in the study. The total cost burden is estimated to be $45,320.
Exhibit 1. Estimated Annualized Burden Hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Training participant questionnaire |
3,000 |
1 |
20/60 |
1,000 |
Total |
3,000 |
N/A |
N/A |
1,000 |
Exhibit 2. Estimated Annualized Cost Burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Training participant questionnaire |
3,000 |
1,000 |
$45.32 |
$45,320 |
Total |
3,000 |
1,000 |
N/A |
$45,320 |
*Based on the mean of the average wages for all health professionals (29-0000) and wages for medical and health services managers (11-9111) for the training participant questionnaire presented in the National Compensation Survey: Occupational Wages in the United States, May 2016, U.S. Department of Labor, Bureau of Labor Statistics (https://www.bls.gov/oes/current/oes_nat.htm).
Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.
The total contractor cost to the government to conduct the one-time questionnaire, as well as to analyze and present all results is estimated to be $52,210. As shown in Exhibit 3a, this amount includes costs for collecting the data ($36,222), analyzing the data ($11,687), and reporting the findings ($4,301).
Exhibit 3a. Estimated Annualized Contractor Cost
Cost Component |
Estimated Annualized Contractor Cost |
Data Collection Activities |
$36,222 |
Data Processing and Analysis |
$11,687 |
Publication of Results |
$4,301 |
Total |
$52,210 |
A Health Communications Specialist (the Project Officer) will be responsible for project management and oversight. This will include oversight of the data collection approach and review of interim and final reports of summarized results. The estimated cost to the Federal Government for these activities is provided in Exhibit 3b. The average annual salary for the GS-13, Step 6 level is $110,595. Federal salary information based on the 2017 OPM Pay Schedule for the Washington/DC area is available on the OPM website at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2017/DCB.pdf.
Exhibit 3b. Federal Government Personnel Cost
Activity |
Federal Personnel |
Average Annual Salary |
Estimated Hours Per Year |
Annual Cost |
Total Cost* |
Data Collection Oversight |
Health Communications Specialist (GS-13, Step 6) |
$110,595 |
6 |
$319.02 |
$691.21 |
Review of Interim and Final Results |
Health Communications Specialist (GS-13, Step 6) |
$110,595 |
4 |
$212.68 |
$460.81 |
Total |
$531.70 |
$1,152.02 |
|||
*Total cost based on the 26-month period of performance.
The estimated total annualized cost for this activity is $52,742. This cost includes annual contractor costs ($52,210) and Federal personnel costs ($532).
This is a new collection of information.
The time schedule for the data collection, data analysis, and final report preparation is presented in Exhibit 4.
Exhibit 4. Timeframe for Data Collection, Analysis, and Preparation of Final Report
Data Collection and Analysis |
Timeframe |
Administer training participant questionnaire |
Immediately on OMB approval |
Analyze data |
60 days from end of data collection |
Prepare final report |
90 days from end of data analysis |
The contractor team will analyze the survey data to identify trends in use of the TeamSTEPPS 2.0 Online Master Trainer curriculum, as well as the perceived impact of the program on organizational outcomes. To that end, a three-phase analysis is proposed. This analysis includes (1) ensuring the quality of the data collected, (2) conducting descriptive analyses, and (3) conducting comparisons of specific cohorts. These phases of analyses are described below.
To ensure maximum integrity of the results, several data-screening and checking procedures (Tabachnick & Fidell, 1996) will be conducted. Specifically, data quality checks will be performed by searching for deviant response ranges, anomalous response patterns, excessive missing data, extreme outliers, and highly skewed or irregular distributions. From these analyses, faulty data or data of poor measurement quality will be flagged, corrected, and/or eliminated. For example, excessive missing data is an indicator of poor data quality. Respondents who fail to respond to more than 10 percent of the protocol questions in the Web survey will be identified; for these cases, the pattern of responses will be reviewed more carefully to determine if, for example, a given respondent’s data should be eliminated. Any strategies that result in the elimination of data will be jointly discussed by the contractor and AHRQ representatives and then fully documented in the final report.
For the descriptive analyses, IMPAQ will employ the following approach: (1) compute a number of descriptive statistics for each variable measured by the survey; (2) develop early warning data protocols (specific statistical analyses to indicate significant variability, low response rates, or error in the data); (3) conduct item analyses; and (4) conduct comprehensive group and subgroup analyses.
IMPAQ will calculate frequency distributions, means, and standard deviations for each closed-ended item included in the survey and combinations of related items that focus on a particular variable or issue. In addition, Reingold will calculate these statistics for each subgroup represented in the sample (e.g., year of training attendance) and conduct analyses to identify subgroup differences. Frequency distributions will show the percentage of people who responded to each response option for each item included in the protocol. Means and standard deviations will be used to examine the relative importance of different items and item combinations that measure specific issues associated with each survey. Finally, standard deviations will be used to examine the level of agreement among respondents regarding issues that are identified as important.
A few items included will be open-ended in nature as a means of following up on closed-ended items to obtain richer detail on unique activities being conducted post-training. The individual responses to the open-ended items will be recorded, compiled, and examined for any themes or patterns of interest. If appropriate, codes will be defined based on the themes identified, and the open-ended responses would be coded into closed-ended categories, which would then be tabulated. Otherwise, the results will be summarized in a memo.
The data gathered from the survey will allow the identification of differences between roles that Master Trainers may fulfill. For example, the analyses will include comparisons of each TeamSTEPPS tool and strategy based on the ratings of usefulness by averaging the usefulness ratings of each tool across all participants. Then, t-tests will be conducted to assess the magnitude of any subgroup differences. In addition to conducting comparisons of tool usefulness, the analyses will compare tool use and perceived impact, and will analyze variations in these characteristics by training participant type. T-tests will be conducted to identify the magnitude of any differences by training participant type. For example, these analyses will demonstrate how useful a specific tool is for direct implementers as opposed to those Master Trainers whose roles are primarily to facilitate implementation or training of TeamSTEPPS.
AHRQ does not seek this exemption.
Attachment A – TeamSTEPPS 2.0 Online Post-Training Survey
Attachment B – Advance Notice, Invitation, Reminder Notice, and Thank You Letters for Training Participant Questionnaire
Attachment C – Federal Register Notice
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |