Phase II Evaluation Activities for Implementing a Next Generation Evaluation Agenda for the Chafee Foster Care Independence Program

Formative Data Collections for Policy Research and Evaluation

Generic Appendices 4.19.16_revised

Phase II Evaluation Activities for Implementing a Next Generation Evaluation Agenda for the Chafee Foster Care Independence Program

OMB: 0970-0356

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0970-0356 can be found here:

Document [docx]

Download: docx | pdf

Appendix A: Initial Screening Materials 2

Appendix B-1: Discussion Guide Materials for Specialized Case Management Interviews 6

Appendix B-2: Discussion Guide Materials for College Success Program Interviews 10

Appendix B-3: Discussion Guide Materials for Employment Program Interviews 16

Appendix A: Initial Screening Materials

Initial Screening Email for Specialized Case Management Discussions: To Learn If State Uses Specialized Case Management

Dear [Name]

My name is [NAME], and I am writing on behalf of a team of researchers at the Urban Institute and Chapin Hall at the University of Chicago. We are part of a research team working with the Department of Health and Human Services, Administration for Children and Families (ACF) to help develop an evaluation agenda to learn more about the effectiveness of programs for youth in foster care. The project is described in more detail in the attached document.

We’re interested in the different ways child welfare jurisdictions in states that have opted to extend federally funded foster care to age 21 may structure case management for transition-age youth who are 17 ½ and older. We’re especially interested in “specialized case managers,” which we’re defining as case managers who work with older adolescents and young adults in recognition of the unique needs of transition-age youth.

As part of this process, we are contacting child welfare agencies in states or jurisdictions that have Federal approval to extend foster care to age 21, to learn if the jurisdiction also provides specialized case management services to transition age youth in extend foster care.

We are emailing to ask:

1) Does your state or jurisdiction provide case management to youth 17 ½ and older? Yes/No

2) Does your state or jurisdiction use “specialized” case managers as we’re defining them above? If so, what are they called?

If your state or jurisdiction does provide specialized case managers, we would greatly appreciate an opportunity to speak with you at a time that is convenient to learn more about the youth you serve, the approach you use and the way the services are funded. Please note this is not an evaluation of your specialized case management services. The interview will be conducted by a pair of researchers and will last no more than an hour. Participation is voluntary and responses will be kept private (in that they will not attribute responses to a specific name). We may contact you following the interview to request clarification.

If there is another person who you think we should be speaking with to learn about the program, we would appreciate you providing us with their name and contact information. Please let me know if you have any questions. I look forward to hearing from you soon.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.

Outreach email for Specialized Case Management Discussions: States We Know Use Specialized Case Management

Dear [NAME],

As part of this process, we are talking with representatives from a number of states that provide specialized case management services to transition age youth. [STATE] has been identified as one such state. We would greatly appreciate an opportunity to speak with you at a time that is convenient to learn more about the youth you serve, the approach you use, the way the services are funded, and any administrative data your agency might collect about its specialized case management services. This is not an evaluation of your specialized case management services, but an opportunity for ACF to learn more about the range of approaches to specialized case management that are being used by states. The interview will be conducted by a pair of researchers and will last no more than an hour. We may contact you following the interview to request clarification. Participation is voluntary and responses will be kept private (in that they will not attribute responses to a specific name).

Sincerely,

[Researcher Name and contact info]

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.

Employment or College Success Initial Outreach Email

Dear [NAME],

My name is [NAME], and I am writing on behalf of a team of researchers at the Urban Institute and Chapin Hall at the University of Chicago working with the Administration for Children and Families (ACF) of the US Department of Health and Human Services to help develop an evaluation agenda to learn more about the effectiveness of programs for youth in foster care. The project is described in more detail in the attached document.

As part of this process, we are talking with representatives from a number of programs aimed at improving the [EMPLOYMENT/EDUCATION] outcomes of youth transitioning out of foster care to learn more about their operation. [PROGRAM NAME] has been identified as one such program. We would greatly appreciate an opportunity to speak with you at a time that is convenient to learn more about how [PROGRAM NAME] functions, the number and characteristics of the youth you serve, and the overall goals of your program. You can expect the conversation to last no more than an hour. Participation is voluntary and responses will be kept private (in that they will not attribute responses to a specific name).

If there is another person at [PROGRAM NAME] who you think we should be speaking with to learn about the program, we would appreciate you providing us with their name and contact information.

Please let me know if you have any questions. I look forward to hearing from you soon.

Sincerely,

[Researcher Name and contact info]

Shape3

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 03/31/2018.

One-Page Project Overview

Policymakers have long been concerned about the poor outcomes experienced by youth transitioning out of foster care into adulthood. The John H. Chafee Foster Care Independence Act (the Chafee Act) provides $140 million in federal funding annually to help states provide independent living services to youth making this transition and allocates 1.5 percent of those funds to rigorous evaluation of such programs. Although the first round of evaluations of independent living programs conducted under the Chafee Act, the Multi-Site Evaluation of Foster Youth Programs, found limited evidence of the effectiveness of the specific programs studied, the project demonstrated that rigorous evaluation of independent living services is possible, and that experimental methods can be used.^¹ Yet, few programs are sufficiently ready for rigorous evaluation. The reasons include:

A team of researchers at the Urban Institute and Chapin Hall at the University of Chicago who has been contracted by the Administration for Children and Families (ACF) of the US Department of Health and Human Services to plan a next-generation evaluation agenda for the John H. Chafee Foster Care Independence Program. This is an opportunity to engage the field in a collaborative process of identifying promising independent living programs and exploring what these programs would need to do to become ready for rigorous evaluation.

The project team has been speaking with individuals who operate different types of programs of particular interest to ACF, including college success, Education and Training Vouchers (ETVs) employment services, housing, and specialized case management programs. The purpose of these exploratory interviews is to help the research team better understand the types of programs that currently exist in each of these areas, learn more about how those programs operate on the ground, and identify next steps for readying and scaling-up some of these programs for potential future evaluation activities. Participation is voluntary and responses will be kept private (in that they will not attribute responses to a specific name). Each interview will last about one hour.

Shape4

Appendix B-1: Discussion Guide Materials for Specialized Case Management Interviews

Discussion Guide for Specialized Case Management Interviews

Thank you for talking with us today. I’m ___________ and this is my colleague __________. We’re from the Urban Institute, a nonprofit research organization in Washington DC / [Chapin Hall at the University of Chicago]. We are part of a research team working with the Department of Health and Human Services, Administration for Children and Families (ACF) to help them develop an evaluation agenda to learn more about the effectiveness of programs for youth in foster care.

[Note: Consent will be completed just prior to beginning the interview.]

We’re interested in the different ways child welfare jurisdictions in states with extended foster care may structure case management for transition-age youth, by which we mean youth 17 ½ and older. We’re especially interested in “specialized case managers,” which we’re defining as case managers who work with older adolescents and young adults in recognition of the unique needs of transition-age youth.

Background:

We’d like to start with a little information about the case management services your state/county/jurisdiction offers to older youth.

What types of case management are older youth eligible for in your state/county/jurisdiction?
Do you use “specialized” case managers as we’re defining them? If so, what are they called?

Jurisdictional issues:

Now we have some questions about your state/county/jurisdiction structures and provides the specialized case management.

What areas of the state provide specialized case managers for transition-age youth? Are services only provided in some areas, and if so, why? [In county-administered states] Are only some counties providing specialized case management?
Is specialized case management a function of the public child welfare agency, is it a contract service, or both? What is the division of labor between the public and private sector in provision of case management for transition-age youth? Does this differ by placement type?

Is specialized case management provided as part of a specialized placement service, such as transitional or supported housing?

Youth characteristics:

What is the age range of youth served by specialized case managers? Are only young adults served, older adolescents, or both? At what age(s) are youth transferred from non-specialized to specialized case managers?
Are all transition-age youth eligible to work with a specialized case manager?
Are there specialized case managers for transition-age youth with special needs (e.g., pregnant and/or parenting youth; youth with serious mental health problems)?

Role(s) of case managers:

Is specialized case management for transition-age youth the only case management provided by the public child welfare agency, or is the specialized case management provided in addition to standard case management? Does this differ for those under 18 and those age 18 or older?
What kinds of services and supports do specialized case managers provide? Do the case managers follow a particular practice model? If so, what model do they follow? Probe on the following:

Life skills training?
Employment support and career exploration?
Educational support? (high school completion; post-secondary enrollment and support)
Mentoring?
Financial assistance?
Health care and/or Reproductive Health information?
Counseling and/or behavioral interventions?
24-hour crisis intervention?
Transportation?
Referrals to other specialized services (e.g., housing; employment training; parent training and child care; mental and behavioral health services)?
Other? Specific tasks related to IVE (court oversight, transition planning, home visits, credit report provision, certification of eligibility…)

[If they haven’t answered this already] Of the services and supports you’ve just described, does one specialized case manager provide them to the youth or do the youth have more than one case managers for different services/functions? (e.g., case worker, housing program case manager, contracted case manager, educational mentor etc.)

What is the caseload size for specialized case managers? How frequently are specialized case managers required to have contact with youth on their caseloads? What kinds of direct interaction (e.g., in person, by phone, email and text messaging) do case managers have with youth on their caseload?
How long is a specialized case manager expected to work with the youth? Is it a specified timeframe, when certain goals are met, or open until a youth leaves care?
What kind of interactions with foster parents or residential placement staff are specialized case managers expected to have?

Hiring, Training and supervision:

Are specialized case managers expected to have special training or background – i.e. different from other case managers? If so, what type of background or training is preferred?
What specialized initial and ongoing training do specialized case managers receive?
What kind of supervision do specialized case managers receive? How does the supervision of specialized case managers differ from that provided to other case managers, if at all?

Program Data

What data does your agency/program collect about the provision of specialized case management? This includes, but is not restricted to, the caseload size of case managers, frequency of contact between case managers and youth, and the nature of services provided by case managers to youth.

General Impressions:

We appreciate your time today and have one final question.

In your opinion, how well does specialized case management work?

Do you see particular benefits to youth? What about challenges for youth?
Do you see particular benefits to the agency or the child welfare system generally? What about challenges?

Informed consent for Specialized Case Management Discussions

Before we begin, I want to tell you a few things about this study and your participation in it. Please feel free to ask me any questions you might have. We will also email you a copy of this information.

A team of researchers from the Urban Institute and Chapin Hall at the University of Chicago is working with the U.S. Department of Health and Human Services, Administration for Children and Families (ACF) to identify programs for transition age foster youth that could potentially be included in a federally funded evaluation. ACF is particularly interested in programs that provide current and former foster youth with specialized case management services.

As part of this process, we are talking with representatives from a number of states that provide specialized case management services to transition age youth. We will ask you some questions about the youth you serve, the approach you use, the way the services are funded, and any data that your agency might collect about specialized case management services. However, this is not an evaluation of your specialized case management services.

Your participation in the study is completely voluntary. You can choose not to participate, skip any questions you would prefer not to answer or end the interview at any time without penalty. The interview will be conducted by a pair of researchers and will last no more than an hour. We may contact you following the interview to request clarification.

We will share what we learn about different approaches to providing specialized case management with ACF and use this information to decide if any of these programs might be good candidates for a rigorous evaluation based on ACF priorities.

The risks associated with participating in this study are minimal. Although any reports or other publications will not identify you by name, they may include the name of your program. Because of your position, we cannot promise that your participation in this research will be kept private.

The data we collect will be stored on an Urban Institute SharePoint drive or on secure servers at Chapin Hall to which only research team members have access and will be kept no more than five years after the project ends in September 2017.

In addition, if you were to disclose an intent to harm yourself or someone else during the course of the interview, we would need to notify law enforcement authorities.

DO YOU HAVE ANY QUESTIONS ABOUT THE STUDY?

DO YOU AGREE TO PARTICIPATE IN THE STUDY?

If you have questions or concerns about the study, please contact:

Mike Pergamit Mark Courtney

Urban Institute University of Chicago

202-261-5276 773-702-1219

mpergamit@urban.org markc@uchicago.edu

If you feel that your rights have been violated or that you have not been treated fairly, contact:

SSA-Chapin Hall IRB Coordinator

University of Chicago School of Social Services Administration

969 E. 60^th Street, Chicago, IL 60637

773-834-0402

Shape5

Appendix B-2: Discussion Guide Materials for College Success Program Interviews

Discussion Guide for College Success Program Interviews

Thank you for talking with us today. I’m ___________ and these are my colleagues ___________ and ______________. We’re from the Urban Institute, a nonprofit research organization in Washington DC / [Chapin Hall at the University of Chicago]. We are part of a research team working with the Department of Health and Human Services, Administration for Children and Families (ACF) to help them develop an evaluation agenda to learn more about the effectiveness of ETV programs for youth in foster care. As part of this research project, we are reaching out to programs aimed at helping foster youth succeed in college. In our conversation with you, we would like to gather detail on [NAME OF PROGRAM].

[Note: Consent will be completed just prior to beginning the interview.]

We’d like to start by hearing more about your program’s history and goals, then talk about who you serve, next we’ll focus on your program’s different components and features, and lastly we want to spend a little time hearing about what you might already be measuring or would like to measure.

Overview of Program

In what year did [INSERT NAME OF PROGRAM] begin operating?

How would you describe your program?
1. What are the main program components?

[Ask each of the following questions below if not specifically mentioned already in question 2].

Who is eligible to participate in [INSERT NAME OF PROGRAM]?
1. How does your program identify eligible students?
2. Is there an application process?

How many students are currently participating in your program?
1. How does your program define participation?
2. Are students counted as participants if they are involved in at least one program components or is there a minimum number of program components in which they need to be involved to be counted?
3. How many students does your program currently have the capacity to serve?

What can you tell me about the characteristics of the students in your program?

How many staff does your program employ and what are their respective roles?

What are your primary sources of funding? (e.g., college/university, private foundations or other philanthropic organizations, individual donors, state or local child welfare system)

For campus-based programs: Does your program work closely with any college/university offices or departments? Which ones? In what ways do you work together?

For non-campus-based programs: Does your program work closely with the colleges/universities in which your students are enrolled? With which college/university offices or departments does your program typically work?

What, if any, relationship does your program have with the state or local child welfare agency?

Program Components

Does your program provide students with a scholarship? What can you tell me about that scholarship?

Do students in your program receive year round housing?

Does your program provide academic advising or supports? Can you tell me more about the academic advising and supports your program provides?

Does you program include a mentoring component? What can you tell about the mentoring?

Does you program provide opportunities for leadership development? What can you tell about those opportunities?

Are there services or supports other than the ones I have already mentioned that your program provides? What are those services or supports?

Are students required to meet with program staff? If so, how frequently?

What opportunities, if any, does your program provide for students to interact with one another?

Evaluation Readiness

Does your program have a logic model?

Does your program track students’ academic progress? How? What academic outcomes do you track?

Does your program track any post-graduate outcomes such as employment? Which outcomes? How?

Are there other outcomes you wish your program was able to track but are limited in capacity to do so? What outcomes?

Does your program track the services or supports students receive?

Are there are eligible students who are not being served by your program due to capacity limitations?

What potential, if any, do you see for program expansion? What are the main barriers to program expansion?

Do you anticipate any major changes in the program during the next few years? What are those changes likely to be?

What distinguishes your program from the other college success programs for students who are or were in foster care?

Is there anything else that we should know about your program that we have not already talked about?

Verbal Informed Consent for Employment and College Success Discussions

A team of researchers from the Urban Institute and Chapin Hall at the University of Chicago is

working with the U.S. Department of Health and Human Services, Administration for Children and

Families (ACF) to identify programs for transition age foster youth that could potentially be included

in a federally funded evaluation. ACF is particularly interested in programs that aim to help current

and former foster youth succeed in [college/the labor market].

As part of this process, we are talking with representatives from a number of [college success/ETV/

employment] programs to learn more about their operation. We will ask you some questions about the [INSERT NAME OF PROGRAM] including questions about the students/youth it serves, the supports it provides and the way it is funded. However, this is not an evaluation of your program.

You can expect the interview to last no more than an hour, although we may need to contact you following the interview to request clarification. Your participation in the study is completely voluntary. You can choose not to participate, skip any questions you would prefer not to answer or end the interview at any time without penalty.

We will share what we learn about different [college success/ETV/employment] programs with ACF

and use this information to decide if any of these programs might be good candidates for a rigorous

evaluation based on ACF priorities.

The risks associated with participating in this study are minimal. Some of the information we gather may be included in a report to ACF, but not released in a public or published document. Any report will not identify you by name but may include the name of your program, and it is possible that your identity could be inferred given your position. We do not intend to quote any of the representatives we talk with, but if for any reason we wanted to quote you in a report, we would ask for your permission first.

DO YOU HAVE ANY QUESTIONS ABOUT THE STUDY?

DO YOU AGREE TO PARTICIPATE IN THE STUDY?

If you have questions or concerns about the study, please contact:

Mike Pergamit Amy Dworsky

Urban Institute Chapin Hall at the University of Chicago

202-261-5276 773-256-5164

mpergamit@urban.org adworsky@chapinhall.org

If you agree to participate in this study and feel that your rights have been violated or that you have

not been treated fairly, contact:

SSA-Chapin Hall IRB Coordinator

University of Chicago School of Social Services Administration

969 E. 59th Street

Chicago, IL 60637 / (773-834-0402)

Shape6

Appendix B-3: Discussion Guide Materials for Employment Program Interviews

Discussion Guide for Employment Program Interviews

Thank you for talking with us today. I’m ___________ and these are my colleagues ___________ and ______________. We’re from the Urban Institute, a nonprofit research organization in Washington DC / [Chapin Hall at the University of Chicago]. We are part of a research team working with the Department of Health and Human Services, Administration for Children and Families (ACF) to help them develop an evaluation agenda to learn more about the effectiveness of employment programs for youth in foster care. As part of this research project, we are reaching out to employment programs for foster youth. In our conversation with you, we would like to gather detail on [employment program].

[Note: Consent will be completed just prior to beginning the interview.]

We’d like to start by hearing more about your program’s history and goals, then talk about who you serve, next we’ll focus on your program’s different components/features including any involvement with employers or other agencies etc., and last we want to spend a little time hearing about what you might already be measuring or would like to measure.

Program background and context:

Could you start by giving us a quick overview of your employment program and the community in which you operate?

PROBE: program history, location, services offered in program, community characteristics, availability of similar services in area

What is the staffing structure? How many staff do you employ and what are their various roles?
Who are your program partners and what’s the nature of the partnership (e.g., is there a formal agreement or MOU, do they participate in decision making and program development)?
How long are youth typically in the program? If relevant: how many week/months does the program typically run?

What is/are your primary sources of funding for your employment services?
What, if any, relationship does your program have with the local or state child welfare agency?
What are the overall employment goals of your program (e.g. general work experience, career exploration, skill building, certifications)?

Program Participants:

How many youth are you serving currently? And how many do you serve over the course of a year?
1. How does your program define participation?
2. Are students counted as participants if they are involved in at least one program components or is there a minimum number of program components in which they need to be involved to be counted?
3. How many students does your program currently have the capacity to serve?
Could you tell us more about the characteristics of the youth population you serve?

PROBE: ages, demographic characteristics, placement type

Do you have specific criteria for eligibility? If so, could you describe those criteria? What are the eligibility requirements with respect to youth in foster care? Do youth need to currently be in foster care to be eligible? Do you serve youth who have aged out of foster care?
Do you have an application process for your program? What are the application components?
How do youth learn about your program? Do you do any recruiting or marketing to particular groups? Do you have any feeder programs or schools? Who can make referrals to your program and how do they identify eligible youth? (Probe: How does the referral process work – from identifying to referring youth etc.)
Do all youth in your program need to have had some involvement in the foster care system?
[If no] About what proportion of the total youth population that you serve is currently or has been in foster care?
What is the educational background of the youth you serve (high school dropouts, high school completers, college-bound, college students)?
What are some of the educational goals or aspirations (if any) that participants in your program may have?
What are some of the career goals (if any) that participants in your program have?

Do you think there are eligible youth who are not being served due to capacity limitations?

PROBE: Do you see potential for program expansion? How would you go about expanding your program – what are the main challenges or considerations you’d need to make in order to expand?

Understanding program components:

Now we’d like to hear a little more about the specific program components. Let’s start with the employment piece. What does it involve? And specifically, who does what? If the program involves building connections to jobs who develops those connections etc?

[If hasn’t come up already] Does your program include paid/unpaid work? hard skills training? Soft skills training? Job search skills training? Job Coaching/ mentoring? Peer support? Other aspects we haven’t mentioned?

Ask about each – see probes below for more detail.

[If soft skills training] How and in what setting are these services offered (e.g. classroom-based, one-on-one)? Do you use a specific curriculum for soft-skills training? If so, can you share it with us?
1. Does your program provide participants with clothing for interviews or work?
[If hard skills training] How and in what setting are these services offered (e.g. classroom-based, one-on-one)? Do you use specific curricula for these trainings? If so, can you share them with us?
1. Do participants receive any kind of certification for completion of training? If so, is the certification recognized in the field?
[If job search skills training] What job search skills are provided (resume writing, on-line job search, fill out applications)? By whom? Does the program provide computers for youth to use in job search?
[If job coaching/mentoring] Who provides the mentoring in your program? Are youth in the program coached or mentored by individuals within the organization, other youth, employers, a combination, other? How and in what setting is job coaching/mentoring offered?
[If peer support] Who organizes and runs the peer support component? What’s offered? How often do youth meet? What do they do/discuss/ work on etc?
Do you offer employment services with broader case management?
[If job placement] Could you describe the kinds of jobs in which you aim to place participants?
1. Are they jobs within the program or with outside employers?
2. Are they jobs that require specific skills and credentials (e.g. IT or construction) or are they jobs that require more general skills (e.g. clerical)
3. [If with local employers] Could you describe the partnership that you have with local employers?

PROBE: Workforce Investment Board, nonprofit agencies, community colleges

How do you identify and work with potential employers? What are your main goals in working with employers (probe: do they try to align youth and employer interests? Do they train for particular types of work? Do they provide particular training and then look for employers who may want youth with those skills)

[If with local employers] Do you work with local nonprofit agencies, community colleges, or other organizations?

[If haven’t mentioned already] Do you offer any residential programming or housing assistance with your employment program? If so, tell us how it works and who is eligible for the assistance, including how it is paid for.
Do you combine employment services with any other program components (e.g. financial literacy, asset-building, education, etc.)?
[If haven’t mentioned already] Is there an educational component to your program – and if so, what is it?
[If haven’t mentioned already] Is there a vocational exploration component to your program – and if so, what is it?
[If hasn’t come up already] Does your program include a stipend or any other incentive for participation. Please tell us about it [amount, timing of payments, requirements for receiving payments]. How important is a stipend or other incentive for participation in the program?

Assessing evaluation readiness – what is measured, what could be measured

Have you ever undertaken any program evaluation? What types of adjustments did you make to your program to prepare for the evaluation? (Probes: e.g, around staffing, increased referral, change eligibility criteria etc)? What was your program’s experience of doing an evaluation [lessons learned about program, data management, and/or partnering with external researchers]?
Does your program have a logic model and/or theory of change? If so, can you share a copy with us? How did it get developed and how do you use it? [If ever undertaken evaluation] Did you use the logic model before the evaluation started? If so, in what ways did the logic model change?
Do you do any assessments of participants’ career interests or skills? Do you track participants’ use of services? If so, what do you track?
What do you see as your key program outcomes? What program outcomes are you currently measuring? Are there any outcomes you’re interested in that aren’t currently being measured? If so, which ones?
1. What is the process for measuring and tracking key outcomes?
2. Are there key outcomes that you wish you were able to measure and track but are limited in capacity to do so?
Do you know of any other employment programs for youth who are or were in foster care? What distinguishes your program from other employment programs for youth who are or were in foster care?
Is there anything else that you think we should know about your program that we have not already talked about?

Shape7