10 Appendix J: Pilot Test Evaluation Protocol for Administr

Appendix J – Pilot Test Evaluation Protocol for Administrators with TeamSTEPPS® Experience

AHRQ – Building Diagnostic Safety Capacity

Administrators – TeamSTEPPS® Course Pilot Test Evaluation

MedStar Health Research Institute (MHRI) will conduct interviews with administrators with TeamSTEPPS® experience from up to ten (n=10) practices. Evaluation will be completed within 3-6 months after implementation of the TeamSTEPPS® Course.

• 10-30 cognitive interviews with practice administrators (1-3 practice administrators per practice x 10 practices); each interview will last approximately 60 minutes

Recruitment Criteria

We will aim to recruit staff members and practice staff in the following manner:

• Administrators involved in the decision making around implementation of the TeamSTEPPS® Course or who have had experience using the TeamSTEPPS® Course

• Administrators who have prior experience using TeamSTEPPS® strategies

• Diversity among practice staff and staff members

MHRI staff will work with the practice coordinators to identify individuals to participate in the interviews.

Interview Goals

The goals of the interviews will be to:

• Obtain feedback on the intervention Course materials

• Obtain feedback on the barriers and facilitators encountered during implementation

• Obtain feedback on satisfaction with instructions and materials

• Obtain feedback on receptivity and enhancements to the Course to improve adoption and implementation

Interview Materials

• Copies of the Course materials

• Informed consent documents

• Digital recorder

Location

Interviews will be conducted at a location within the practice or within the setting’s community. Interviews may also be conducted virtually via the telephone.

Participant Stipends

Upon arriving at the interview location and after the completion of the informed consent process, all participants will complete the required paperwork to receive the stipend for participation.

Informed Consent Procedures

None.

Each interview will take no more than 60 minutes.

AHRQ – Building Diagnostic Safety Capacity

Practice Administrators – TeamSTEPPS® Course Pilot Test Evaluation

Form Approved
OMB No. xxxx-xxxx
Exp. Date xx/xx/20

WELCOME AND INTRODUCTION

• Thank you for agreeing to participate in the interview about your experiences with the Agency for Healthcare Research and Quality’s TeamSTEPPS® Course to improve communication among providers related to diagnosis!

• My name is [ INSERT NAME OF INTERVIEWER] and I am the facilitator for today’s conversation. I am here with [INSERT NAME OF PROJECT TEAM] and they will be taking notes of our conversation.

• With your permission we will also be audio recording the session. This will help make sure that we don’t miss anything that you say and can share with other people who are working on this project. The recording will be deleted after we have the notes transcribed and are sure we have captured all your comments accurately.

• TODAY/TONIGHT we will be asking you questions about your experiences with using the TeamSTEPPS® Course and your setting’s experience implementing it.

• Nothing that you say or share today will be shared in an identifiable way with your practice but represented together along with nine other practices in a report to the Agency for Healthcare Research and Quality. We will not share your name or anything that you say with them in a personally identifiable way.

• Do you have any questions before we begin?

• So let’s get started.

This survey is authorized under 42 U.S.C. 299a. The confidentiality of your responses to this survey is protected by Sections 944(c) and 308(d) of the Public Health Service Act [42 U.S.C. 299c-3(c) and 42 U.S.C. 242m(d)].  Information that could identify you will not be disclosed unless you have consented to that disclosure. Public reporting burden for this collection of information is estimated to average 60 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (0935-XXXX) AHRQ, 5600 Fishers Lane, Room #07W42, Rockville, MD 20857.

I would like to begin by asking you all some questions about your experience with implementing TeamSTEPPS® strategies.

TeamSTEPPS Experience

1. Has your setting attempted to implement a TeamSTEPPS® training course/strategies before?

   1. Which of the TeamSTEPPS® programs have you previously implemented? How did that experience influence your decision to adopt the TeamSTEPPS for diagnosis course?

• TeamSTEPPS® 2.0

• TeamSTEPPS® for Office-Based Care

• TeamSTEPPS® for Long-Term Care

• TeamSTEPPS® Dental Module

• TeamSTEPPS® Rapid Response Systems Guide

• Training Guide: Using Simulation in TeamSTEPPS® Training

• Other: Please name

2. How did your experience with the present TeamSTEPPS® differ from your prior experiences?

   1. How do you think practices who do not receive technical assistance would fare in their implementation?

   2. Are there strategies we should include to help novice practices adopt this TeamSTEPPS® course?

   3. From your perspective, can you describe how the TeamSTEPPS® Course changed communication among providers related to diagnosis?

Diagnosis

2. Prior to the signing up for the pilot test, what was your practice doing in terms of improving diagnosis and diagnostic error mitigation?

   1. Was diagnostic improvement a strategic priority for your organization?

   2. What were the key factors that underpinned your decision to join the pilot test?

   3. What information could we have provided to make this intervention more attractive to you as an administrator and leader?

3. Can you describe for me how you made the decision about which of the TeamSTEPPS® modules/communication strategies your practice decided to implement?

4. How would you describe the impact of the TeamSTEPPS® Course on safety of the diagnostic process in your setting?

1. What modules within the Course (introduction, team structure, communications, leadership, situation monitoring, mutual support,) do you feel had the greatest impact on the diagnostic process within your setting?

   1. Can you elaborate on why you think it had that impact?

2. What tools within the Course do you feel had the greatest impact on the diagnostic process within your setting?

   1. Can you elaborate on why you think it had that impact?

3. What tools within the Course do you feel had the least impact on the diagnostic process within your setting?

   1. Can you elaborate on why you think it had minimal impact?

4. What was the most effective part of the TeamSTEPPS® Course within your practice? Can you describe why that is?

Patient Engagement

Thank you for sharing. I now want to speak with you a little more about the impact of this training on communications with patients.

5. Do you think the TeamSTEPPS® course impacted patient’s engagement in diagnosis? Why do you think this is?

Implementation of the TeamSTEPPS® Course

I would like to focus on the practice’s implementation of the TeamSTEPPS® Course.

6. Can you describe your implementation process for me? Does your organization rely on any standard implementation or quality improvement approaches that were used to help implement the TeamSTEPPS course?

   1. Can you describe for me how that may have impacted your success?

7. Can you describe for me how your setting advised the team about implementation?

   1. What was the reaction of clinicians to the program? What about the staff? Can you describe any positive or negative feedback you have received?

      1. How can we improve the program to better address these concerns?

8. What about costs of implementing the TeamSTEPPS® Course? Was that a challenge at all for you?

   1. What advice might you have for other administrators considering adopting the program?

9. How much staff time and resources did the implementation of the TeamSTEPPS® Course take? Is this sustainable?

Those were all the questions I had today. Are there any questions that I should have asked that I did not?

Thank you for your time and participation in this interview. Your comments will be very helpful to this project and will help us to make important improvements to the TeamSTEPPS® Course!