SUPPORTING STATEMENT
Part A
Assessing the Impact of the Patient Safety Improvement Corps (PSIC) Training Program
Version: October 28, 2008
Agency of Healthcare Research and Quality (AHRQ)
Table of Contents
A. Justification - 3 -
1. Circumstances that make the collection of information necessary - 3 -
2. Purpose and Use of Information - 3 -
3. Use of Improved Information Technology - 3 -
4. Efforts to Identify Duplication - 3 -
5. Involvement of Small Entities - 3 -
6. Consequences if Information Collected Less Frequently - 3 -
7. Special Circumstances - 3 -
8. Federal Register Notice and Outside Consultations - 3 -
8.a. Federal Register Notice - 3 -
8.b. Outside Consultations - 3 -
9. Payments/Gifts to Respondents - 3 -
10. Assurance of Confidentiality - 3 -
11. Questions of a Sensitive Nature - 3 -
12. Estimates of Annualized Burden Hours and Costs - 3 -
13. Estimates of Annualized Respondent Capital and Maintenance Costs - 3 -
14. Estimates of Annualized Cost to the Government - 3 -
15. Changes in Hour Burden - 3 -
The mission of the Agency for Healthcare Research and Quality (AHRQ) set out in its authorizing legislation, The Healthcare Research and Quality Act of 1999 (see Attachment A), is to enhance the quality, appropriateness, and effectiveness of health services, and access to such services, through the establishment of a broad base of scientific research and through the promotion of improvements in clinical and health systems practices, including the prevention of diseases and other health conditions. AHRQ shall promote health care quality improvement by conducting and supporting:
1. research that develops and presents scientific evidence regarding all aspects of health care; and
2. the synthesis and dissemination of available scientific evidence for use by patients, consumers, practitioners, providers, purchasers, policy makers, and educators; and
3. initiatives to advance private and public efforts to improve health care quality.
Also, AHRQ shall conduct and support research and evaluations, and support demonstration projects, with respect to (A) the delivery of health care in inner-city areas, and in rural areas (including frontier areas); and (B) health care for priority populations, which shall include (1) low-income groups, (2) minority groups, (3) women, (4) children, (5) the elderly, and (6) individuals with special health care needs, including individuals with disabilities and individuals who need chronic care or end-of-life health care.
As part of their effort to fulfill their mission goals, AHRQ, in collaboration with the Veteran’s Administration’s (VA) National Center for Patient Safety (NCPS), developed and implemented the Patient Safety Improvement Corps (PSIC) training program, which has been operating since 2003. The main objective of this program is to improve patient safety by training a select group of stakeholders such as state health department and hospital staff in various patient safety concepts, tools, and techniques and ultimately helping to build a national infrastructure for supporting patient safety efforts in healthcare organizations and at the state level.
To understand the extent to which AHRQ has successfully created this infrastructure of patient safety knowledge and skills, AHRQ has contracted with the American Institutes for Research (AIR) to conduct an evaluation of the PSIC training program to examine the extent to which training participants have been able to implement the patient safety concepts, tools, and techniques in their home organizations and the extent to which participants have spread that training, knowledge, and skills to their organizations, local areas, regions, and states.
Participants applied as teams with each member representing a different type of organization (i.e., State Health Department, hospital or provider organization, and Quality Improvement Organization, QIO). Due to the differences among the types of organizations participating in the program, each participant has a different potential to apply tools and concepts within and/or beyond their home organizations. For example:
In their regulatory role, State Health Department participants will be more likely than some other participants to use PSIC materials as a means of influencing practices across a range of organizations (e.g., setting new State policies on standards for conducting RCAs, determining the type of acceptable or required follow-up). As a result, they are more likely to have a broad impact overall. State Health Departments may also have a more direct impact to the extent that they work closely with hospitals to disseminate PSIC material and information and/or to investigate patient safety events.
Hospital participants are more likely than other participants to implement the PSIC material on a daily basis and will be more likely to affect specific work processes being conducted within an organization. As a result, hospital participants are likely to have a focused and specific impact within that organization only.
Similar to State Health Department participants, QIO participants will be more likely to have both an in-depth and broad impact assuming that they use the PSIC materials to assist a particular organization in their patient safety activities, as well as to provide general patient safety guidance to a large number of organizations.
To clarify the differences among the participants, we developed a logic model (see Attachment B) that highlights the roles of the different types of participants, the types of activities in which they are likely to engage post-training, and the potential outcomes that may stem from these activities. The logic model served as a guide for developing questions for the web-based questionnaires and qualitative interviews to ensure that we could capture participant and leadership feedback as thoroughly and accurately as possible.
Two Web-based questionnaires are planned to examine post-training activities and patient safety outcomes as a result of training from multiple perspectives. One questionnaire is directed to all PSIC training participants, and the other questionnaire is directed to leaders of the organizations from which the training participants were selected. Items will cover post-training activities, implementation experiences, and perceived outcomes as a result of these activities. The participant and leadership questionnaires are presented in Attachments C and D, respectively1. Advance notice, invitations to participate, reminder e-mails, and thank you letters to respondents are included in Attachments E and F for the participant and leadership questionnaires, respectively.
In addition, we will conduct a series of semi-structured qualitative interviews across six states with PSIC trainees, as well as with other patient safety professionals who did not attend the PSIC training but who are employed by or partner with organizations that were part of the training. Information gathered from these semi-structured interviews will be analyzed and used to draft a “lessons learned” document that will capture additional detail on the issues related to participants’ and organizations’ abilities to implement and disseminate the PSIC material post-training. Each of the six states will be distinct from the others and vary in terms of degree of training spread and type of post-training activities, among other aspects. Because many years have elapsed since the advent of the PSIC program, both AHRQ and the NCPS have had continued contact and communications with many participants. This anecdotal information, in concert with relevant participant characteristics (e.g., types of organizations represented by the participating teams), serves as the basis for identifying potential sites as having either a perceived high or low degree of spread post-training. The interviews themselves will more accurately reveal the degree of training spread for the teams included.
Interviewees for the qualitative interviews will be drawn from qualified individuals serving in a variety of roles (i.e., policy maker, trainer or facilitator, front-line implementer) among PSIC trainees and non-trainees, yielding a total of six different types of respondent groups defined by role and PSIC training experience. Up to nine individuals per each of the six respondent groups across the six states selected will be interviewed, for a maximum of 54 individuals. A semi-structured interview protocol will be used as a guide for the interviews and not read verbatim (see Attachment G2). It will be adapted for each role based on the respondent group and to some degree, for each individual, based on their training and patient safety experience. Attachment H contains the informed consent form that each participant will be required to sign prior to beginning the interview.
Some of the limitations of this study, which will also be noted in the final report, include the following:
Retrospective analysis of post-training experiences. This is not an empirical study; therefore, neither causality nor generalizability can be established;
Survey non-response, which can limit our understanding of the concerns and issues participants encountered when trying to apply PSIC tools and concepts in their home organizations or organizations that they support;
Lack of access to contact information for non-participants, except on the rare occasion when our site visit contacts identify non-participants to include in the interviews during the site visits;
Inclusion of only six out of 52 states and territories for the site visits and qualitative interviews due to budgetary constraints. As a result, we will not be able to generalize findings beyond these individual cases;
Limited number of interviews that can be conducted at each site visit due to time and budgetary constraints; and
Inability to interview participants who have changed jobs since PSIC. Because up to five years have passed for some of the PSIC participants, they may no longer work for the organization they were employed by at the time of participation or may no longer work in the field of patient safety; therefore, we may have fewer than anticipated participants.
Despite the limitations in this study, the data we collect via the questionnaires and qualitative interviews will provide AHRQ with valuable information that will help them as they develop tools and training that effectively support organizations’ efforts to improve patient safety. By understanding the factors that facilitate or inhibit the use of tools or the spread of knowledge, AHRQ will better understand the needs of these organizations and will be better prepared to address their future concerns, issues, and needs for improving patient safety.
The PSIC program represents a new approach to training for AHRQ. This program focused on disseminating patient safety information, building skill sets, and ultimately fostering a national network of individuals who support, promote, and speak a common language of patient safety. To meet the objective of creating a national infrastructure, AHRQ first required that each team’s application include a letter of endorsement and support from each team member’s Chief Executive Officer (CEO) or equivalent regarding the team member’s participation in the program. In addition, participants were selected, in part, on the expectation that they would later disseminate their newly acquired knowledge and skills to others in their home organizations or in other organizations locally, regionally, or state-wide. The PSIC program involved training teams of participants, led by representatives from either State health departments or Quality Improvement Organizations (QIOs) and included members of individual hospitals or healthcare systems.
As a result of the time and commitment invested over the four years of program implementation, AHRQ seeks to learn the extent to which they have succeeded in spreading the PSIC material and forming the requisite infrastructure to support on-going and future patient safety efforts.
The final product for this evaluation will be a report to AHRQ that documents the background, methodology, results including any patterns or themes emerging from the data, limitations of the study, and recommendations for future training programs and tool development. The results of this evaluation will help AHRQ understand the extent to which participants and participating organizations have been able to employ various PSIC tools and concepts and the barriers and facilitators they encountered. This information will help guide AHRQ in developing and refining other patient safety tools and future training programs for patient safety and other areas.
In order to reduce respondent burden, both the training participant and leadership questionnaires will be administered via the Web. The contact lists acquired by AHRQ and NCPS will be used to develop the questionnaire distribution lists. Each potential respondent will receive a minimum of four e-mail contacts to encourage participation (i.e., an advance notice of the questionnaires, an initial invitation to complete the questionnaire, and two follow-up e-mails to remind respondents to complete the questionnaire).
Using an on-line system for data collection rather than a paper-based questionnaire makes completing and submitting the questionnaire less time-consuming for respondents. Any skip patterns included in the questionnaire (i.e., questions that are only appropriate for a proportion of the respondents) will be automatically programmed into the Web-based form of the questionnaire, thereby eliminating any confusion during questionnaire completion. In addition, the contractors can also ensure that important items are not inadvertently skipped or ignored by setting software requirements to ensure proper completion of questionnaires based on specific respondent selections.
AHRQ’s interagency agreement with the NCPS included conducting the training sessions and training evaluation. As part of this agreement, NCPS and AHRQ have collected traditional participant feedback via end-of-course questionnaires and feedback discussions conducted during the training program3. These efforts were used to make revisions and improve the program and are not redundant with the proposed evaluation effort. In addition, AHRQ contracted with the RAND Corporation to conduct a formative evaluation of AHRQ’s patient safety initiative of which the PSIC training program is one small component. Through this evaluation, RAND conducted interviews with a sample of training participants to obtain their feedback on the PSIC program and their ability to use this material back on the job.
This study takes the next step and builds upon the information already collected by AHRQ, NCPS, and RAND by first enabling the development of a logic model that depicts the relationship between participants’ patient safety roles, possible activities in which they may engage post-training, and resulting outcomes. Second, this information provides the basis by which questions to participants and their organizational leaders could be streamlined to enhance AHRQ’s understanding of the linkages between roles, activities, and outcomes. For example, participant responses to previous interview questions have been leveraged to reformulate open-ended questions into closed-ended response options, further reducing the burden on the respondents. Items have been designed to minimize redundancy with the data already collected and delve more deeply into post-training issues.
No small businesses will be involved in this study.
This request is for a one-time data collection effort.
This request is consistent with the general information collection guidelines of 5 CFR 1320.5(d)(2). No special circumstances apply.
As required by 5 CFR 1320.8(d), notice was published in the Federal Register on April 16, 2008 for 60 days (see Attachment I). No comments were received.
AIR has consulted with the RAND Corporation on their evaluation study components relating directly to the PSIC and obtained clarification on their methodology. RAND reported having good success soliciting participation in their study and that this population has been eager to share their feedback.
Respondents to the Web-based questionnaire will not receive any gifts or payment in exchange for their participation.
Individuals and organizations will be assured of the confidentiality of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). They will be told the purposes for which the information is collected and that, in accordance with this statute, any identifiable information about them will not be used or disclosed for any other purpose.
Individuals and organizations contacted will be further assured of the confidentiality of their replies under 42 U.S.C. 1306, and 20 CFR 401 and 4225 U.S.C.552a (Privacy Act of 1974). In instances where respondent identity is needed, the information collection will fully comply with all respects of the Privacy Act.
AHRQ’s authority will withstand a FOIA request of information that would identify a person or establishment.
Names and business contact information for PSIC training participants will be taken from a pre-established records system belonging to the AHRQ and the NCPS. This records system was developed through participating organizations’ applications to the PSIC program and subsequent communications related to the training program. Names and business contact information for the CEOs or equivalent for each participating organization will be identified through their initial application packets to the PSIC program. The list provided will include training participant’s name, e-mail address, organizational affiliation, state, and job title.
This information will be used solely to recruit and contact respondents. An AIR staff member will contact those individuals by e-mail to inform them of the study, its purpose, and request their participation. The invitation e-mail and questionnaire responses will be stored on a secure server at AIR.
Identifiable questionnaire responses and qualitative interview responses will be accessible only to members of the AIR project team all of whom have signed affidavits of nondisclosure. Responses will be de-identified using ID numbers corresponding to each respondent. AIR project staff will be instructed that any downloaded or print versions of the questionnaire results or interview notes are only to be shared within the AIR project team, and are to be de-identified (using ID numbers instead of names and/or organizational or state affiliations). All de-identified data files will be stored on AIR’s secure server, which is password-protected. Only AIR project staff will be able to access the data files and server.
The data files containing identifiable information will be destroyed at the end of the project (March 2009). All remaining data files will include de-identified information and will be provided to AHRQ at the end of the contract.
Results will be reported in the aggregate, and no information collected by AIR will be shared with persons outside the project. Reports on questionnaire results or qualitative interviews will not reveal the identity of the respondents unless they provide specific, written permission authorizing its release.
The explanations regarding confidentiality provided to respondents are included in the e-mail correspondences to request their collaboration. The explanations are also included in the introductory page of the questionnaire. These statements assure respondents that the information they provide will be “treated in a confidential manner” by AIR researchers and AHRQ. The text for both of these communications is shown below.
Text for both the initial screen for the Web-based questionnaire and e-mail correspondences: Please note that all of your information will remain confidential and that all information provided to AHRQ as a result of this questionnaire will be reported at the aggregate level to ensure your confidentiality.
Questionnaire items and qualitative interviews do not require respondents to provide information of a sensitive nature as defined by OMB and DHHS or to provide information such as social security numbers or Medicare/Medicaid numbers. As required by AIR’s Internal Review Board (IRB), AIR has developed an introduction to the questionnaire that includes aspects of informed consent such as a description of the research objectives, a discussion of the importance of their input and experiences, details concerning how the data will be used, and aspects regarding confidentiality. The introduction will be positioned at the beginning of the questionnaire. Continuation to complete the questionnaire will indicate the respondent’s consent.
Exhibit 1 shows the estimated annualized burden hours for the respondent’s time to participate in the study. Questionnaires will be distributed to all training participants and the leaders from the participant’s organization who nominated them for participation in the PSIC program. The training participant questionnaire is estimated to require 30 minutes to complete and the organizational leader questionnaire is estimated to require 15 minutes to complete. Qualitative interviews will be conducted with a maximum of 54 individuals and will last about one hour each, resulting in a total burden of 223 hours.
Exhibit 2 shows the estimated annualized cost burden based on the respondents’ time to participate in the study. The total cost burden is estimated to be $7,453.06.
Exhibit 1. Estimated annualized burden hours
Form Name |
Number of respondents |
Number of responses per respondent |
Hours per response |
Total burden hours |
Qualitative interview |
54 |
1 |
60/60 |
54 |
Training participant questionnaire |
300 |
1 |
30/60 |
150 |
Organizational leader questionnaire |
75 |
1 |
15/60 |
19 |
Total |
429 |
NA |
NA |
223 |
Exhibit 2. Estimated annualized cost burden
Form Name |
Number of respondents |
Total burden hours |
Average hourly wage rate* |
Total cost burden |
Qualitative interview |
54 |
54 |
$35.19 |
$1,900.26 |
Training participant questionnaire |
300 |
150 |
$32.18 |
$4,827.00 |
Organizational leader questionnaire |
75 |
19 |
$38.20 |
$725.80 |
Total |
429 |
223 |
|
$7,453.06 |
* Based upon the mean of the average wages for health professionals for the training participant questionnaire and for executives, administrators, and managers for the organizational leader questionnaire presented in the National Compensation Survey: Occupational Wages in the United States, June 2005, U.S. Department of Labor, Bureau of Labor Statistics.
There are no direct costs to respondents other than their time to participate in the study.
The total cost to the government for this activity is estimated to be $127,442 to conduct the two one-time questionnaires and to analyze and present its results. This amount includes costs for developing the data collection tools ($50,976); collecting the data ($25,488); analyzing the data and reporting the findings ($44,605); and administrative support activities ($6,373).
This is a new collection of information.
The time schedule for the data collection via Web-based questionnaires and case studies, data analysis, and final report preparation is presented in Exhibit 3.
Exhibit 3. Timeframe for data collection, analysis, and preparation of final report
Data Collection and Analysis |
Timeframes |
Conduct qualitative interviews |
Immediately upon OMB approval |
Administer Web-based questionnaires |
Immediately upon OMB approval |
Analyze data |
60 days from end of data collection |
Prepare final report |
90 days from end of data analysis |
AHRQ does not seek this exemption.
Attachment A: AHRQ's Authorizing Legislation
Attachment B: Logic Model of Post-PSIC Patient Safety Activities and Outcomes by Participant Role
Attachment C: Training participant questionnaire
Attachment D: Leadership questionnaire
Attachment E: Advance notice, invitation, reminder notices, and thank you letters for training participant questionnaire
Attachment F: Advance notice, invitation, reminder notices, and thank you letters for leadership questionnaire
Attachment G: Semi-structured interview guide
Attachment H: Interview informed consent form
Attachment I: 60-Day Federal Register Notice
1 Attachments C and D were revised on October 28, 2008, to reflect our response to OMB’s questions on October 22, 2008.
2 Attachment G was revised on October 28, 2008, to reflect our response to OMB’s questions on October 22, 2008.
3 This data collection effort did not require OMB clearance, as noted in our response to OMB’s questions on October 22, 2008.
| File Type | application/msword |
| File Title | OMB Clearance Application |
| Author | hamlin-ben |
| Last Modified By | wcarroll |
| File Modified | 2008-11-04 |
| File Created | 2008-11-04 |