84.421F Disability Innovation Fund (DIF) Evidence Building Support (EBS)

Attachment F

84.421F Disability Innovation Fund (DIF)

Project Director Interview Protocol

NOTE: Interviewer instructions are in ALL CAPS and square brackets. Before the interview, interviewers will consult with the project liaison to ensure they have a broad understanding of the project design, staffing, population, goals, evaluation, challenges, and progress to date. 

Thank you for agreeing to participate in this interview. My name is [NAME], and I’m also joined by [INTRODUCE OTHER WESTAT STAFF]. We work for Westat, the national research organization that the Rehabilitation Services Administration (RSA) has contracted with to provide evaluation technical assistance support for 84.421F DIF grantees. As part of our work, we are also tasked with synthesizing evaluation findings across all 27 84.421F DIF grantees. The information gathered from this interview will help us do that. Can you please confirm for me your name and role on [PROJECT]? [CHECK AND DOCUMENT NAME AND ROLE IN INTERVIEW TRACKING SHEET.] 

• As described in the information that we previously sent you, this conversation will focus on the design and goals of the 84.421F DIF project you direct, project recruitment and enrollment, evaluation progress and challenges, implementation barriers and facilitators, and project adaptations and lessons learned. 

• [IF YEAR 3-5 INTERVIEW, ADD: During our conversation, I will ask you to think about project and evaluation activities that have taken place in the last year, since our last interview.] 

• The information you provide will be used in the Annual Evidence Report, which summarizes grantee project and evaluation progress. 

• Your participation is an opportunity to highlight your project’s accomplishments, share lessons learned, contribute to a broader understanding of DIF implementation, and inform future DIF funding and evaluation improvements. 

You received an informed consent information sheet outlining the purpose of the interview and your rights. You can ask questions about it at any time. I will review some key points of that information before we begin. 

• This is a research study, and your participation is voluntary. You may stop participating at any time and decline to answer any question. 

• Your participation or responses will not affect RSA’s funding decisions. 

• The interview will last no longer than ninety minutes. 

• With your permission, we will audio and video record the interview for accuracy. Only the Westat research team will have access to the recordings and transcripts. 

• As noted, your responses will help us draft the DIF Annual Evidence Report. That means we may quote you or other DIF project directors in the report. You will not be identified by name if we quote you. 

Now I’ll pause to give you the opportunity to ask any questions you might have about that information. [ANSWER QUESTIONS.] 

Let me just confirm, do you agree to participate in this interview? [PAUSE FOR RESPONSE.] Do you agree to have the interview recorded? [PAUSE FOR RESPONSE.] 

• [IF PARTICIPANT DECLINES TO BE RECORDED]: Okay, that’s fine, we will take notes during our conversation. Let’s begin. 

• [IF PARTICIPANT AGREES TO BE RECORDED]: Thank you. I’m going to start recording now. [START RECORDING.] For the purposes of the recording, can you please confirm: Are you willing to participate in the interview? [PAUSE FOR RESPONSE.] Are you willing to have the interview recorded? [PAUSE FOR RESPONSE.] Okay, let’s begin. 

A. Project Overview 

To start our conversation, I will review some notes about your project, and then the evaluation, to help make sure I have the correct, up-to-date background. First, I will review the project background. [PROVIDE A BRIEF OVERVIEW OF THE PROJECT DESIGN AND PROGRESS BASED ON NOTES FROM THE PROJECT LIAISON.] 

1. Is there any other important background about the project design or its status right now that I missed? 

PROBE: Have I missed any important information about partners, recruitment and enrollment so far, or service delivery? 

Now let’s move on to the current progress of the evaluation. [PROVIDE A BRIEF OVERVIEW OF THE EVALUATION DESIGN AND PROGRESS BASED ON NOTES FROM THE PROJECT LIAISON.] 

2. Is there any other important background about the evaluation design or its status right now that I missed? 

PROBE: Have I missed any important information about data sources, instrument development progress, data collection progress, or anything else? 

B. Recruitment and Enrollment 

Now I will ask some questions about project recruitment and enrollment. [IF YEAR 3-5 INTERVIEW, ADD: Please think about recruitment and enrollment activities that took place in the last year, since our last interview. REVIEW PROJECT NOTES FOR DETAILS TO TAILOR QUESTIONS ON PARTICIPANTS AS NEEDED. FOR EXAMPLE, THEY MAY INCLUDE BOTH INDIVIUALS RECEIVING EMPLOYMENT SERVICES AND EMPLOYERS.] 

1. What partnerships has the project built to help with recruitment and enrollment of [PROJECT PARTICPANT GROUP OR GROUPS]? 

PROBE: What help have you received from local community organizations, professional networks, personal networks, or others? 

PROBE: What challenges did you encounter in partnering with other agencies or organizations to support project recruitment? 

2. What strategies or activities have you or your partners used to recruit project participants, including [PROJECT PARTICPANT GROUP OR GROUPS]? 

PROBE: Did any of the program participants belong to existing client groups? [IF YES]: Please tell me about that. 

PROBE: How did you advertise the opportunity? What organizations or groups helped get the word out? 

Let’s discuss the challenges and successes you have encountered with recruitment. I will start with the background information about recruitment shared by your project liaison. [PROVIDE A BRIEF REVIEW OF CHALLENGES AND SUCCESSES DESCRIBED IN PROJECT NOTES.] 

3. Are there other challenges the project team has encountered with recruiting participants? 

PROBE: How did you address these challenges? 

4. Are there other noteworthy successes with recruitment you would like to highlight? 

PROBE: Overall, what recruitment strategies or activities have worked best? What made these approaches successful? 

5. How does your current recruitment approach compare to what was initially planned for the project? 

PROBE: What changes have been made to how you recruit participants over time? Why did you make these changes? 

6. How do the current project participants compare to the intended population? 

C. Implementation 

The next questions ask about experiences with providing the project services. [IF YEAR 3-5 INTERVIEW, ADD: Please think about services the project has provided in the last year, since our last interview.] Let’s start by reviewing my notes on the services described in the project logic model. [PROVIDE AN OVERVIEW OF THE SERVICES DESCRIBED IN THE PROJECT LOGIC MODEL. AS NEEDED, DESCRIBE SERVICES PROVIDED TO DIFFERENT TYPES OF PARTICPANTS.] 

1. Are there other services the project provides that are missing from this list? 

PROBE: What services are those? What participants have they been provided to? 

2. Sometimes unforeseen challenges or barriers to providing services arise between the planning and implementation stages of a project. Are there services I named that the project is not currently providing? 

PROBE: What services are those? What are some challenges, barriers, or other reasons why the project is not currently providing them? 

3. What services have been most popular or well-received by participants? 

PROBE: What makes these services popular? 

PROBE: What has made the project’s approach to delivering these services especially successful? 

4. It can be challenging to predict what services will be appealing and helpful to specific service populations. What project services have had lower than expected enrollment or engagement? 

PROBE: What do you think has contributed to lower enrollment and engagement with these services? 

5. What other challenges have project staff encountered in delivering the services? 

PROBE: For example, have there been challenges with staffing, capacity, training, keeping participants interested in the program, or anything else? 

6. What trainings did the project conduct? 

PROBE: Who participated in the trainings? 

PROBE: What was successful about the trainings? 

PROBE: What was challenging about the trainings? 

7. How do the trainings provided compare to what was proposed or intended? 

PROBE: Did you provide extra or additional trainings? What trainings were those, and why were they provided? 

PROBE: Are there any trainings that you intended to hold but did not? What barriers prevented staff from carrying out those trainings? 

D. Evaluation Activities 

Now let’s turn to your independent evaluation activities. [IF YEAR 3-5 INTERVIEW, ADD: Please think about evaluation activities that have taken place in the last year, since our last interview.] 

1. What data is your evaluator using to inform the evaluation? 

PROBES: Is your evaluator collecting data? Are they using data collected by the grant project for the evaluation? 

2. [IF THE EVALUATOR IS COLLECTING DATA AS PART OF THE EVALUATION]: What strategies or methods has your evaluator used to collect data for the evaluation? 

PROBE: For example, how is the evaluation measuring participation, receipt of services, and outcomes? 

3. What has helped the evaluation be successful this year? 

PROBE: For example, what partnerships, strategies, or circumstances have helped facilitate the evaluation? 

4. What are some key challenges or barriers that have been encountered in the process of conducting the evaluation? 

PROBE: Are these challenges still present? If not, how have you addressed them? 

5. What adaptations have been made to help the evaluation be successful? 

6. [IF THE EVALUATOR IS COLLECTING DATA AS PART OF THE EVALUATION]: What concerns, if any, do you have about the quality of the data being collected? 

7. Does the evaluation plan include an assessment of project fidelity? 

PROBE: In other words, does the evaluation measure how closely the project has stuck to a certain model, stated goals, or something else? Please tell me about that. 

E. Outcomes 

Now I will ask you to reflect on project progress and outcomes. [IF YEAR 3-5 INTERVIEW, ADD: Again, please think about project outcomes in the last year, since our last interview.] 

1. Overall, how do you think the success of services provided by the project compare to other service models or activities you and your staff have used before? 

2. What has helped the project successfully achieve positive outcomes? 

PROBE: For example, what partnerships, strategies, or circumstances have helped the project be successful? 

3. What challenges have acted as barriers to achieving positive outcomes? 

PROBE: For example, what partnerships, strategies, or circumstances have created difficulties with project success? 

4. What unintended or unplanned outcomes have resulted from the project activities? 

F. Closing Questions 

We will end the interview with a few big picture questions. 

1. Are there any other current challenges or facilitators to implementation of the DIF project itself that we haven’t yet discussed? 

2. What about other challenges or facilitators to the project evaluation that we didn’t cover?

3. Have staff discussed efforts to make the project sustainable after the DIF grant program ends? 

PROBE: What kinds of strategies have you discussed? 

4. What are your current thoughts about how the project could be replicated or reproduced in other populations or contexts? 

PROBE: What populations or contexts do you think the project would be most successful in? 

5. [UNLESS YEAR 5 INTERVIEW, ASK]: What are some key lessons you have learned this year that you will carry over into next year’s project implementation? 

6. [UNLESS YEAR 5 INTERVIEW, ASK]: What about lessons learned that you will carry into next year’s evaluation? 

7. Is there anything else about your program or evaluation that you think it is important for me to know? 

Thank you for your participation! Those are all the questions I have for you today. We will use the answers given by Project Directors in these interviews to help us draft the Annual Evidence Report that we are submitting to RSA on [PROJECT FFY DUE DATE]. The report will be made public after they review it, and we will reach out to share it with you. 

Do you have any last questions for me? [ANSWER QUESTIONS].

Notice of Confidentiality

Information collected for this study comes under the confidentiality and data protection requirements of the Rehabilitation Services Administration (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for research purposes. Neither the participating grant programs nor individual respondents will be identified by name or affiliation. All of the information you provide may only be used for research purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). 

PRA Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is 18xx-xxxx. Public reporting burden for this collection of information is estimated to average 1.5 hours per response, including time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. The obligation to respond to this collection is voluntary. If you have any comments concerning the accuracy of the time estimate, suggestions for improving this individual collection, or if you have comments or concerns regarding the status of your individual form, application or survey, please contact Diandrea Bailey, PhD, U.S. Department of Education, Office of Special Education and Rehabilitative Services, Rehabilitation Services Administration, 400 Maryland Avenue SW, Room 4B238, Washington, DC 20202-2800 directly.

Attachment F Project Director Interview Protocol

F-0