Instrument 6C – Site Visit 3 Interview Guide for Administrators_ Provider Agency Administrators

Provider Agency Administrator Interview Site Visit #3

Instructions:

Thank you for us joining today. We would like to invite you to participate in an interview that will last about 60 minutes. We are conducting an evaluation of LifeSet, and today we’d like to learn more about how the program has been operating over the last year. Specifically, we will talk about your agency’s relationship with other agencies, performance monitoring, and aspects of the program’s implementation among other topics. We will use this information to better understand how the LifeSet program operates and serves young adults in New Jersey.

Your participation in this interview is voluntary. You can choose not to answer any question or not participate in the interview at all. There will be no consequences to you if you choose not to participate. We will keep the information you provide private and will not share it with anyone except for research staff working on the study. Additionally, federal law states that an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this data collection is 0970-0577 and the expiration date is XX/XX/XXXX.

Background and Role

Thank you for taking the time to talk with me today.

[Note to Interviewer: Only ask #1-8 if this is the first time interviewing this respondent, otherwise go to #9] I’d like to start by learning a little bit about you.

1. What is your current position and how long have you been in it?

2. What are your main responsibilities?

3. Is there a degree/credential or experience requirement for your position? What is it?

4. How long have you worked at [agency]?

5. What other positions have you held at [agency]?

6. What role do you play as part of the LifeSet program?

7. Have you worked with former foster youth or other vulnerable youth populations before working for [agency]? If so, in what capacity?

8. Have you held other positions – client facing or management – in social services before? If so, what were they?

[Note to interviewer: Only ask if this is a follow-up interview with this respondent] First I’d like to ask if anything changed regarding your position or your agency since we last spoke on [date].

9. Has anything changed regarding your position or main responsibilities? Please describe any changes in your position or main responsibilities. When the changes occur and why?

CQI Processes

I am interested in learning about any continuous quality improvement efforts that may support LifeSet services.

10. Does your agency have a formal continuous quality improvement (CQI) process for LifeSet?

    1. [if yes] Can you please briefly describe your CQI process?

11. [If applicable] In what ways does the CQI process support the implementation of LifeSet? [If needed] For instance, how does the CQI process support fidelity or outcomes?

12. [If applicable] What LifeSet topics or issues have been, or are being, addressed through your CQI process?

    1. Can you provide an example of how the CQI process is addressing one of the topics or issues you mentioned?

Collection and use of feedback

13. Does your agency collect feedback from LifeSet supervisors and specialists to understand program needs?

    1. If so, how, and how often is this feedback collected? 

    2. How, if at all, does this input change LifeSet service delivery?   

14. Does your agency collect feedback from LifeSet youth? 

    1. If so, how, and how often is this feedback collected? 

    2. How, if at all, does this input change LifeSet service delivery? 

Performance Monitoring

The next few questions are about performance monitoring related to LifeSet. We are defining performance monitoring broadly to include not just processes or activities but also outcome monitoring and reporting.

15. Can you describe the DCF performance monitoring requirements you must meet in order to provide LifeSet services?

    1. What outcomes does DCF expect you to achieve for youth in LifeSet? What happens if you don’t achieve those outcomes?

16. Can you describe the Youth Villages performance monitoring requirements you must meet in order to provide LifeSet services?

    1. What outcomes does Youth Villages expect you to achieve for youth in LifeSet?

    2. What happens if you don’t meet Youth Village’s performance requirements?

Impact of the evaluation

The next few questions are about the impact of the evaluation on implementation of LifeSet.

17. In what ways has the external evaluation by the Urban Institute impacted the way LifeSet is implemented, both positive and negative? [If needed] You may know the evaluation by the title the Young Adult Services Study, or YASS.

    1. [Probes] For example, impact on things like how you receive referrals, the amount of contact with DCF, the length of the program, how specialists engage youth.

18. What agency resources have been required to support the external evaluation?

Partner Relationships

Now, I’d like to learn a bit about your relationships with DCF, Youth Villages, and other agencies.

Relationship among implementing agencies

19. How would you describe your agency’s relationship with DCF?

    1. [If needed] Would you say this relationship is positive, negative, or neutral? Why do you say that?

    2. Does the relationship between your agency and DCF need to change in order to better serve youth in LifeSet? If so, can you please tell me what changes are needed?

20. What information do you share with DCF about the youth that you are both serving? What information do they provide to you?

    1. Is this information sharing helpful or unhelpful in meeting youths’ needs? Why do you say that? Could you give an example?

21. How would you describe your agency’s relationship with Youth Villages?

    1. [If needed] Would you say this relationship is positive, negative, or neutral? Why do you say that?

    2. Does the relationship between your agency and Youth Villages need to change in order to better serve youth in LifeSet? If so, can you please tell me what changes are needed?

Relationships with other entities/systems

22. In what ways does your agency collaborate with other agencies, organizations, or systems around LifeSet? For example, mental health, judiciary/courts, other non-profits, or community groups.

    1. [If applicable] How effective are these collaborations in helping to meet the needs of youth in LifeSet? Can you give an example?

Fidelity

The next couple questions are about whether LifeSet is being delivered with fidelity. By fidelity, I mean consistently following the procedures from the LifeSet manual for things like what activities occur, how, and when.

23. Do you believe LifeSet is being delivered with fidelity in your agency?

    1. If yes, how do you know?

       1. [If not mentioned] Are there model fidelity requirements you must meet? If so, what are they?

    2. If no, what LifeSet components are not being delivered with fidelity?

24. Do the staff who are implementing LifeSet believe in the model?

    1. [Alternative wording, if needed] Have you found that LifeSet staff in your agency tend to follow the model, even if a staff member might prefer a different approach? Do you have an example you can share?

25. Would you say it is easy or difficult to maintain fidelity to the LifeSet model over time? Please tell me why you think that.

    1. [probe if needed] Can you give an example of what makes it easy or difficult?

    2. What resources are needed to sustain fidelity to the model? [If needed: For example, maintain or increase funding, more or different training, retaining qualified staff]

Program Sustainability

Next, I’d like to discuss how LifeSet may be implemented in the future, once the evaluation ends.

26. What resources have been required to implement LifeSet?

27. What resources and supports are needed to sustain LifeSet after the evaluation?

    1. [Probe] Are any supports needed from Youth Villages to sustain LifeSet?

    2. [Probe] Are any supports needed from DCF to sustain LifeSet?

28. What factors would impact your agency’s decision to continue providing LifeSet program?

    1. [Probes] evaluation results, funding, political will

29. How will LifeSet’s implementation change when the external evaluation is over?

Staff Requirements

The next couple questions are about turnover within the LifeSet program.

30. What has been the rate of turnover in LifeSet Supervisors and Specialists?

    1. [If applicable] What do you think is causing staff to leave the program or agency?

31. In what ways has staff turnover affected the delivery of LifeSet?

    1. [Probes]Ability to take new referrals, staff able to devote time to youths’ needs

Service Delivery

Difference between LifeSet and SAU

Now I’d like to ask you some questions about LifeSet services compares to services as usual.

32. In what ways is LifeSet similar or different from other services usually provided to youth leaving foster care?

    1. [Probes if needed] Types of services offered, intensity or duration, program philosophy or values

33. From your perspective, do older youth who have been in LifeSet have different outcomes than youth who received other programs or services?

    1. [if yes] What outcomes are different for youth who have been in LifeSet?

    2. [if no] What youth outcomes are common regardless of the program?

34. Are some youth a better fit than others for LifeSet? If so, what makes youth a good fit for LifeSet?

Cultural responsiveness and disparities in services

Next, I’d like to learn about whether LifeSet is responsive to youths’ unique backgrounds and any inequities in the types of services youth receive.

35. Do you consider the LifeSet program to be culturally responsive? [If needed] In other words, is the program able to take into account youths’ unique demographic and cultural backgrounds, such as youths’ race/ethnicity, sexual orientation, gender identity, language, religion, or citizenship status?

    1. If so, in what ways? Could you give some examples?

    2. In what ways are LifeSet services not responsive to youths’ demographic and cultural backgrounds?

36. Are there demographic disparities or inequities in the services youth receive or are referred to?

    1. If so, can you describe these disparities?

    2. What actions can be, or are being, taken to reduce service disparities?

Reflection on LifeSet Services

Finally, I’m interested in your opinion about the LifeSet program.

37. Is there anything about LifeSet that you would like to change? If so, what is it?

38. What does the LifeSet program does well?

39. What does the LifeSet program could improve upon?

Closing Questions 

Thank you for taking the time to talk with me today.

40. Is there anything that I did not ask about that you think I should know about LifeSet or your experience? 

41. Do you have any final questions for me about the study, or about the research team? 

The Paperwork Reduction Act Statement: This collection of information is voluntary and will be used to evaluate the programs and services provided to young adults who are currently or were previously in foster care. Public reporting burden for this collection of information is estimated to average 60 minutes per response, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0577, Exp: XX/XX/XXXX. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to Michael Pergamit at mpergamit@urban.org.