Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Supporting Youth to be Successful in Life (SYSIL) Study
OMB Information Collection Request
0970-0574
Supporting Statement
Part A
July 2024
Type of Request: Revision
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officer: Mary Mueggenborg
Part A
Executive Summary
Type of Request: This Information Collection Request seeks to (1) to extend the previously approved data collection for the Supporting Youth to be Successful in Life (SYSIL) Study (OMB #0970-0574) and (2) to revise previously approved interview and focus group protocols, as well as an additional round of interviews and focus groups.
Progress to Date: OMB approved the impact and implementation studies and instruments for the OPRE Study, Supporting Youth to be Successful in Life (SYSIL) on July 27, 2021 (OMB #0970-0574). Baseline data collection with youth for the impact study began in September 2021. Data collection for the 6-month and 12-month follow-up surveys began in April 2022 and October 2022, respectively. The first round of check-in calls began late 2022 and is complete. The second round of check-in calls began February 2024 and is complete. The first round of interviews and focus groups began May 2022 and is complete.
Timeline: There were no challenges to meeting the original timeline. An extension for three additional years is requested to allow OPRE to continue collecting these data. OPRE requests approval of the extension before the current approval expires in July 2024 to avoid the confusing and costly process of updating the expiration date on currently approved materials each month until full approval is received.
Previous Terms of Clearance: The previous clearance did not identify any terms.
Summary of Changes Requested: OPRE is requesting approval for proposed changes to the previously approved interview and focus group protocols, as well as an additional round of interviews and focus groups. The purpose of the proposed revisions is to better understand staff and youth experiences in delivering and receiving services. We are requesting three additional years of approval of this study.
Time Sensitivity: OPRE requests approval of the extension before expiration of the current approval on July 31, 2024.
A1. Necessity for Collection
Preventing homelessness among young people involved in the child welfare system remains an urgent issue for child welfare policymakers and practitioners. Housing stability is essential for achieving self-sufficiency and promotes health and well-being, particularly during the transition to adulthood. Unstable housing can initiate a negative cycle of poor health, limited employment, and continued housing instability (Dion et al. 2014). A combination of disadvantages places youth with a history of foster care, especially those aging out of care, at greater risk of experiencing homelessness than their peers.
The current body of evidence does not provide sufficient guidance to child welfare practitioners and other interested parties about programs to support transitions for older youth and young adults in foster care. For example, evidence of effectiveness exists for some programs that promote positive youth development, but these programs generally have not been tested with youth who are aging out of foster care (Courtney et al. 2017). Moreover, not enough is known about the best ways to implement programs for youth transitioning out of foster care (Courtney et al. 2017).
The Supporting Youth to be Successful in Life (SYSIL) study is building evidence on how to end homelessness among youth and young adults with experience in the child welfare system by continuing work with an organization (Colorado Department of Human Services) that was part of Phases I and II of the Youth At-Risk of Homelessness (YARH) project (OMB Control Number: 0970-0445). The Administration for Children and Families (ACF) has contracted with Mathematica to conduct a summative evaluation of Colorado’s Pathways to Success comprehensive service model (Pathways). Pathways is an intensive, coach-like case management model for youth at risk of homelessness with child welfare involvement. The summative evaluation is assessing the impact of Pathways on key outcomes related to housing stability and a successful transition to adulthood for youth and young adults with experience in the child welfare system. The summative evaluation includes an impact study and an implementation study.
This request is for an extension of our data collection, which is needed to complete the work. We propose edits in two instruments – the protocols for interviews with staff and focus groups with youth and young adults. The overarching purpose for the data collection, and the use of the data, have not changed. Rather, we are proposing a limited set of additional questions to support a deeper understanding of some of the data we collected to date.
There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.
A2. Purpose
Purpose and Use
The purpose of SYSIL is to determine whether and how the Pathways program affects the targeted outcomes for youth and young adults at risk of homelessness and to provide a rich understanding of what is required of organizations to implement Pathways. The data collected through the surveys will provide evidence on targeted outcomes while data collected through the interviews and focus groups will provide more detailed information about what is required for successful program implementation and service delivery. ACF is using the data collected through SYSIL to provide important information to the field around the effectiveness and implementation of Pathways, which could inform decisions by other states on their Chafee services.
The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.
Dissemination Efforts
Findings from the impact and implementation studies are and will be of interest to federal and state agencies providing child welfare and homelessness services, program providers, and youth and young adults. We are releasing traditional reports that will be supplemented by other dissemination efforts – such as briefs, infographics, videos, and podcasts – to ensure the findings are shared broadly. We are and will continue to work with experts to identify appropriate products and strategies to reach the full range of intended audiences. We hope findings will be of interest to both those who participate in the evaluation, live or work in Colorado, and more generally who work with youth and young adults.
Research Questions
Impact Study
The impact study is designed to answer two types of research questions: (1) impact research questions about the magnitude of the effect that Pathways has on participant outcomes and (2) exploratory research questions that link features of implementation to participant outcomes.
Answers to two impact research questions will provide evidence of the effect of Pathways for the full study population and for key subgroups:
What is the impact of Pathways on key outcomes, including but not limited to housing, educational attainment, employment, permanency, and well-being?
What are the impacts after the first six months of Pathways (about halfway into the average length of participation in Pathways)?
What are the impacts immediately following participation in Pathways (12 months after entry)?
What are the impacts 12 months after the end of participation in Pathways (24 months after entry)?
Is Pathways particularly effective for key subgroups of the target population? Specifically, how do findings differ for the following:
Youth approaching age 17.5 who are able to decide to remain in foster care or leave foster care
Youth with varying foster care backgrounds (for example, age at entry, time in care, second-generation child welfare status, permanency status)
Youth by gender identity
Youth who have mental health or substance abuse challenges (potentially stemming from trauma)
Youth by race and ethnicity
Youth by sexual orientation
Youth by level of connectedness at program entry
Youth by the experience of the site1 implementing Pathways (for example, new implementers or seasoned implementers)
Implementation Study
The implementation study will explore research questions related to five topics that will provide information necessary for interpreting the impacts of Pathways and understanding its implementation in different settings:
Topic: Differences between the Pathways service model and comparison services. The question on this topic addresses the ways in which the Pathways service model differs from services offered to youth and young adults in the comparison condition. The research question is:
How are services under the Pathways service model distinct from services available in the comparison condition?
Topic: Barriers to and facilitators of Pathways implementation. Questions on this topic address how Pathways leadership and staff put components of the Pathways service model into operation and the factors that contributed to or inhibited implementation. Research questions are:
What did the site do to support initial service delivery (that is, start-up activities) in Pathways sites?
To what extent did Pathways sites use continuous quality improvement (CQI) to support implementation? How did CQI support implementation?
What factors (facilitators and barriers) contributed to or hindered initial and ongoing service delivery in Pathways sites?
Topic: Fidelity to the Pathways service model. Questions on this topic address the extent to which sites delivered core services in the comprehensive service model as intended and factors that might have contributed to or hindered fidelity. Research questions are:
To what extent did the Pathways sites deliver Pathways services with fidelity?
To what extent did levels of fidelity vary across Pathways sites?
What factors (facilitators and barriers) contributed to or hindered achieving and sustaining fidelity?
Topic: Service, resource, and policy environment surrounding Pathways implementation. Questions on this topic address the services generally available to youth and young adults in the evaluation sample (both treatment and comparison groups) and the child welfare policies that may affect youth and young adults in the evaluation sample. Research questions are:
Which services and resources are available to youth and young adults in the Pathways and comparison groups in the sites where this summative evaluation occurs?
What does site leadership perceive to be key child welfare policies and regulations that affect youth and young adults in the Pathways and comparison groups in the locations where this summative evaluation occurs?
Topic: Youth and young adults’ responsiveness to the Pathways service model. Questions on this topic address youth and young adults’ acceptance of Pathways services and their perceptions of the services. Research questions are:
What were the characteristics of the population served by Pathways?
What services were delivered to youth and young adults in Pathways?
What strategies did Navigators use to promote and maintain youth and young adult engagement?
How did engagement vary among youth and young adults participating in the Pathways service model?
What were youth and young adults’ perceptions of Pathways services? How did they describe their experience in Pathways?
Study Design
Impact Study
The impact study approach consists of a cluster QED that will use survey data as the primary data source for key outcomes of interest and be supplemented by administrative data elements.2 Forty-nine counties within Colorado will participate in the impact study.3 Data is being collected from youth in participating counties at baseline, and at 6-, 12-months after study enrollment. By comparing youth in treatment sites to youth in comparison sites, the study will be able to show the effectiveness of Pathways on a large number of outcomes collected in the survey and available in the administrative data.
Currently4, the approved study includes the following information collections:
Three 30-minute web-based surveys: one at baseline (enrollment), and then post-enrollment at 6-, and 12-months.
Administrative data from Linked Network of Colorado (LINC) to look at additional youth outcomes.
Two contact information update web-based surveys at 3- and 9-months.
One virtual and one in-person visit to sites to interview staff and youth.
Two brief check-in calls with staff.
We are requesting an additional virtual site visit to interview staff and youth as part of this information collection revision.
Request for exemption from Statistical Policy Directive No. 15
The YARH/SYSIL study team discussed OMB’s Statistical Policy Directive No. 15: Standards for Maintaining, Collecting, and Presenting Federal Data on Race and Ethnicity (SPD 15) and its possible implication on this data collection. After careful consideration, the study team recommends that YARH / SYSIL not implement SPD 15. The following information informed our decision.
YARH/SYSIL began data collection in September 2021. We will enroll youth until 09/30/2024.
Our baseline, 6-, and 12-month surveys currently collect information on race and ethnicity, using the categories in place in 2021.
As of May 2, 2024, 429 youth and young adults enrolled in the evaluation. Since December 2023, enrollment averages 14 youth per month. Based on this information, we anticipate we will enroll an additional 70 youth before enrollment closes on 09/30/2024. This will bring our final enrollment to 499 youth and young adults.
Focusing on the 429 youth and young adults who enrolled by May 2, 2024, 380 completed baseline surveys.
Our current analytic plan related to race and ethnicity is to use the baseline survey responses, unless it is missing. If missing, we will utilize race and ethnicity responses provided on a subsequent survey or data from the Pathways administrative data system.
Given the SPD 15 categories would be used for only 3 of a 36-month enrollment period and we may need to collapse SPD 15 categories for analytic reasons, we do not feel implementing SPD 15 would be logical or advisable at this time.
Given the available data from the baseline survey and from the administrative data systems, we are confident that we can demonstrate the equivalence of the treatment and comparison groups on a large number of potential confounders. Demonstrating equivalence at baseline helps the study to produce strong, internally valid impact estimates. The demonstration of baseline equivalence helps address the limitation of all QEDs that it is impossible to know whether the youth are well-matched on unmeasured characteristics. A second potential limitation is that the study may not be well powered to detect program impacts on several outcomes of interest, unless the observed impacts are quite large. We will therefore conduct an additional impact study design that uses the administrative data sources to more fully supplement the main impact study. Specifically, we will use a larger pool of potential sample members to expand the comparison group and conduct an analysis that obviates the need to do a clustering correction. This change will address the chief limitation of the main study design: statistical power. We will acknowledge these limitations in all publications that describe the findings of this study.
Implementation Study
The implementation study will adapt the Consolidated Framework of Implementation Research (CFIR) to guide clear conceptualization and systematic assessment of the range of contextual factors that facilitate or hinder implementation of the Pathways service model.
We will conduct three rounds of site visits to 12 sites (6 sites delivering Pathways and 6 sites delivering comparison services) for the implementation study. One virtual visit is complete. We hope to conduct an in-person visit in fall/winter 2024. The final proposed site visit would be virtual, occurring in fall/winter 2025 if approved in this information collection revision. The additional proposed site visit will allow us to understand implementation for the youth and young adults who enrolled nearer the end of our enrollment period.
During these site visits, we will speak with a range of people involved in Pathways services and comparison services. We will also conduct focus groups with youth and young adults who receive services in both the Pathways and comparison sites. The site visits will occur at about 12-, 36-, and 48-months after enrollment begins. We will collect and analyze program administrative data to assess patterns of service delivery and describe the extent to which Pathways services are delivered with fidelity.
We conducted two “check-ins” by telephone with program directors from all 16 sites to ask about current service delivery. Both check-ins were completed by April 2024, and are not included in this information collection request.
Table A.2 provides an overview of the data collection instruments.
Table A.2. Information Collections
Data Collection Activity |
Instruments |
Respondent, Content, Purpose of Collection |
Mode and Duration |
Baseline Data Collection |
Youth Survey* |
Respondents: Youth with consent to participate
Content: Demographics, attitudes about and outlook toward the future, experiences with the child welfare system, education and employment history and goals, involvement with the criminal or juvenile justice system, access to available system resources, connections with adults and peers, and parenting
Purpose: To describe the target population and allow for a demonstration of baseline equivalence |
Mode: Web and phone
Duration: 25 minutes |
Follow-up data collection (6-months and 12-months) |
Youth Survey |
Respondents: Youth with consent to participate
Content: Demographics, attitudes about and outlook toward the future, experiences with the child welfare system, education and employment history and goals, involvement with the criminal or juvenile justice system, access to available system resources, connections with adults and peers, and parenting
Purpose (6-months): To examine short-term impacts across outcome domains of interest Purpose (12-months): To examine interim impacts across outcome domains of interest
|
Mode (6-months): Web and phone Mode (12- months): Web, with phone and field non-response follow-up
Duration: 25 minutes
|
Interviews |
Interview Guide for Pathways Sites (Treatment Sites) |
Respondents: Key partners and program staff in treatment sites
Content: Youth recruitment and enrollment; program service components; partnerships that support the program; service, resource, and policy context; implementation experience; program resources; and continuous quality improvement
Purpose: To understand the contextual information that may influence implementation and fidelity and provide context for findings from the impact study |
Mode: In-person or virtual+
Duration: 1.5 hours |
Program Director Check-ins |
Subset of questions from the interview guides for Pathways (Treatment) Sites** |
Respondents: Program Directors from all Pathways sites (treatment sites)
Content: Details around the services provided to youth, including any updates or changes since the site visit
Purpose: To understand the breadth of services provided to youth in Pathways sites |
Mode: Phone
Duration: 30 minutes |
Interviews |
Interview Guide for Comparison Sites |
Respondents: Key partners and program staff in comparison sites
Content: Youth recruitment and enrollment; program service components; partnerships that support the program; service, resource, and policy context; implementation experience; program resources; and continuous quality improvement
Purpose: To understand the contextual information that may influence implementation and fidelity and provide context for findings from the impact study |
Mode: In-person or virtual+
Duration: 1.5 hours |
Program Director Check-ins |
Subset of questions from the interview guides for Comparison Sites** |
Respondents: Program Directors from all comparison sites
Content: Details around the services provided to youth, including any updates or changes since the site visit
Purpose: To understand the breadth of services provided to youth in comparison sites |
Mode: Phone
Duration: 30 minutes |
Focus groups |
Focus Group Discussion guide for Pathways Youth (Treatment Youth) |
Respondents: Youth in treatment sites with consent to participate in the focus group
Content: Experiences with program services and staff, including services the youth received, how they accessed the services, if they found the services helpful and why or why not, and suggestions on how to improve them
Purpose: To understand youth’s experiences with services |
Mode: In-person or virtual++
Duration: 1.5 hours |
Focus groups |
Focus Group Discussion Guide for Comparison Youth |
Respondents: Youth in comparison sites with consent to participate in the focus group
Content: Experiences with program services and staff, including services the youth received, how they accessed the services, if they found the services helpful and why or why not, and suggestions on how to improve them
Purpose: To understand youth’s experiences with services |
Mode: In-person or virtual++
Duration: 1.5 hours |
Contact Information Update (3 and 9 months) |
Contact Information Update Request |
Respondents: Youth enrolled in the study.
Content: Address, phone number and alternate contact information.
Purpose: To obtain updated contact information for youth and continue to engage and familiarize youth with the study. |
Mode: Email and text
Duration: 5 minutes |
*The youth survey is the same instrument administered at three different points in time: (1) baseline, (2) 6-month follow-up, and (3) 12-month follow-up.
** The program director check-ins do not require a unique instrument; they were conducted using a subset of items from the interview protocol.
+ Mode of administration for the interviews and focus groups will be determined based on existing travel restrictions and public health guidelines concerning COVID-19 at the time of data collection.
++ During a webinar for YARH-3 experts, we received feedback on youth recruitment and engagement activities for evaluation activities. Experts suggested that virtual focus groups are more effective than in-person focus groups for engaging youth participants. We will have the flexibility to conduct in-person focus groups, if necessary.
Other Data Sources and Uses of Information
The study will also use administrative data from the child welfare system, homelessness management information system, and program providers. Administrative data from the Linked Information Network of Colorado (LINC) will provide additional outcomes on child welfare, public assistance, and employment, among others. We will use the administrative data from LINC as an alternative outcome data source for estimating impacts on the outcomes. We expect that some outcomes measured in the administrative data will not be captured in the survey data (in particular, long-term outcomes). LINC data will also be used to validate the subset of constructs measured in both the survey and administrative data (for example, by comparing youth self-reports on recent child welfare status with administrative data on recent status). We will also collect and analyze program administrative data from the Pathways Management Information System (PMIS)5 to assess patterns of service delivery and describe the extent to which Pathways services are delivered with fidelity. Administrative data from the LINC and PMIS will be used in their existing format. Use of these data will not impose any new information collection or record-keeping requirements on respondents.6
A3. Use of Information Technology to Reduce Burden
The SYSIL data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. The youth surveys are web-based surveys. Trained staff provide participants with smartphones or tablet computers, along with a unique URL to access the survey from the device. Youth can also complete the survey on their own device.
Web-based surveys are an attractive option for surveys for adolescents and young adults, and in particular for surveys that ask sensitive questions and present various pathways based on responses to those questions. Web-based surveys can decrease respondent burden and improve data quality. Unlike paper instruments in which respondents must determine the question routes themselves, the web-based application will include built-in skips and route respondents to the next appropriate question based on their answers. The web-based program automatically skips them out of any questions that are not relevant to them, thus reducing the burden on respondents having to navigate through various paths. In addition, data checks can be programmed into the survey to eliminate responses that are out of range or that conflict.
Mathematica has and will continue to conduct semi-structured interviews with program administrators, supervisors, and direct service staff and focus groups with participating youth and young adults. Mathematica conducted brief check-ins with program directors. The information to be collected during interviews, focus groups, and check-ins is not conducive to the use of information technology, such as computerized interviewing. In-person administration, or virtual administration via a secure web platform such as WebEx offers the best opportunity to tailor the data collection to staff and youth participants with minimal burden on respondents. These recordings, with participant approval, can assist in minimizing burden as verbatim transcripts will be made, decreasing the need for the interviewer to ask the respondent to repeat themselves to ensure the notes are accurate.
A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency
The information collection requirements for the SYSIL study have been carefully reviewed to avoid duplication and to maximize opportunities to use existing data, including administrative data. By using the administrative data from the PMIS, we will avoid requesting this information through a separate data collection instrument specifically for the SYSIL study, thus reducing the potential burden on case managers. Additionally, as noted earlier, survey questions that request information that can be answered by data items available through administrative data sources have been removed from the survey, thus reducing burden on youth and minimizing duplication of information.
Our decision to rely on the LINC administrative data r minimizes burden while maintaining SYSIL’s ability to look at longer term outcomes.
A5. Impact on Small Businesses
No small businesses will be involved with this information collection.
A6. Consequences of Less Frequent Collection
Data collected as part of this study are essential to conducting a rigorous evaluation of the Pathways program. Without outcome data collected through the impact study, we could not estimate the short- term effects of the program (using the 6-month follow-up survey) or determine whether those effects are sustained in the long-term or translate to the expected outcomes (using the 12-month follow-up survey and LINC administrative data). Data collected through the implementation study will be essential for understanding the results of the impact study and assessing the implementation of Pathways.
A7. Now subsumed under 2(b) above and 10 (below)
A8. Consultation
Federal Register Notice and Comments
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on April 10, 2024, (89 FR 25270) and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.
The following experts in their respective fields were consulted on the data collection instruments for which clearance is requested:
Tym Belseth, Research Coordinator with Texas Institute for Child and Family Wellbeing, University of Texas at Austin
Mark Courtney, Samuel Deutsch Professor in the School of Social Service Administration, University of Chicago
Lanae Davis, Senior Research Associate, Center for Policy Research
Jan DeCoursey, Senior Research Scientist and Program Area Co-Director, Child Welfare, Child Trends
Amy Dworsky, Research Fellow, Chapin Hall, University of Chicago
Jennifer Haight, Supervisory Children and Family Program Specialist, Children’s Bureau
Chereese Philips, Child Welfare Program Specialist, Children’s Bureau
Kaylene Quinones, LMSW, BraveLife Coordinator
Cassandra Simmel, Associate Professor and Director, MSW Certificate in Promoting Child and Adolescent Well-Being, Rutgers University
Nancy Thoennes, Associate Director, Center for Policy Research
As previously approved, we are offering tokens of appreciation to youth for (1) completion of each 25-minute survey, and (2) participation in 1.5-hour focus groups. Additionally, youth will receive an item such as a dry bag, water bottle, document portfolio, or other item of similar value when they enroll in the study.7 We offer a $5 token of appreciation for youth who complete a contact information update request. In our discussions with experts in the field, program staff, and other partners, several stressed the importance of providing tokens of appreciation to youth that reflect the value of their input, as no one can replace them for this data collection. The data collection requires input from the small, specialized population of youth and young adults with experience in the child welfare system. The surveys contain introspective and potentially sensitive questions that only youth with their experiences can speak to. The tokens of appreciation reflect the value of the specific experiences these youth provide in responding to the surveys. The following proposed amounts for the surveys were determined based on consultation with experts in the field, other partners, and program staff:
$40 gift card for baseline survey
$45 gift card for the 6-month follow-up survey
$50 gift card for the 12-month follow-up survey
$5 gift card for each contact information update request
Participating youth are asked to participate in multiple waves of data collection. The proposed amounts for the tokens of appreciation serve to develop an on-going relationship with youth to encourage their continued participation for each survey, which is essential for assessing program impacts on long-term outcomes. We value youth’s participation in the data collection and the proposed tokens of appreciation reflect the value of their participation in data collection activities and contributions to the data collection that only they can provide. Estimates of program impacts may be biased if respondents differ from non-respondents and those differences are correlated with group assignment. The risk of biased impact estimates increases with lower overall responses or larger differences in survey response rates between research groups (What Works Clearinghouse 2013). Concern about the potential for low overall response rates are particularly relevant to this study because the target population is youth and young adults with experience with the child welfare system who are at risk of homelessness. A number of factors could complicate tracking participants over time including:
System fatigue
Unstable housing
Fewer permanent connections with others
Less use of leases, public utility accounts, cell phone contracts, credit reports, memberships in professional associations, licenses for specialized jobs, and appearance in publications such as newspapers and blogs
Use of an alias to get utility accounts because of poor credit and prior payment issues
Use of pay-as-you-go phones
Youth and young adults may be in the custody of the state (foster care) at time of early data collection efforts. If they are not in the custody of the state, we anticipate they will be low-income given their age, likely education level, and employment opportunities. We increased the value of the token of appreciation as time passed to account for the decreased contact between the study and youth once they are no longer receiving services and the increased value of money to help with bills once living on their own. Mathematica offers gift cards for the baseline survey either as a physical gift card or as an electronic gift card. All other tokens of appreciation are provided as electronic gift cards.
Additionally, we expect that as participating youth get older and age out of the foster care system, they will be more transient and harder to locate for data collection. We utilize field staff to locate and complete surveys with these hard-to-reach youth, however we anticipate that offering tokens of appreciation will create cost savings for the field tracking efforts. We expect that the tokens will encourage youth to complete the surveys, resulting in a smaller proportion of youth who will need to be located by field staff.
The tokens of appreciation are designed to boost overall response rates for this low-income, hard-to-reach population as well as to minimize differential response rates between the treatment and comparison youth. Participants assigned to the comparison group may be less motivated to participate than those assigned to the treatment group because they are not receiving the intervention and may not feel that the surveys are relevant to them. Additionally, youth in the comparison group may not have developed the strong connections with their Chafee workers that treatment youth may have developed with their Navigators and therefore may be less likely to feel connected to the study and participate in the data collection. Pejtersen (2020) showed incentives can increase the response rates in surveys of youth, and higher incentive amounts can have some beneficial impacts on response rates (Brown and Calderwood, 2014). Oh et al (2021) found larger incentives may encourage youth to complete study activities.
We offer the choice of a water bottle or a portfolio folder to the Chafee workers as a way to engage them and obtain buy-in to the study and facilitate data collection. The token shows our appreciation for the additional work related to the study, mainly screening and obtaining consent/assent for participation. Additionally, the token will provide a feeling of being part of a team.
We are currently conducting a test of a two-part payment option for the 6-month follow-up survey token of appreciation8. For this test, cohorts of youth (youth enrolled in a month) are randomly assigned to either receive a $5 pre-payment for the 6-month survey, or no pre-payment. Those who receive the pre-payment will get the remainder of the token of appreciation ($40) upon completing the 6-month survey. Those who do not receive the pre-payment will get the full token of appreciation ($45) upon completion of the 6-month survey. We will examine whether response rates vary based on their pre-payment status.
For the 12-month survey, all youth receive the $5 pre-payment for the 12-month survey with the remainder of the token of appreciation ($45) upon completion. We will not include a test of this option as there is no variation in whether youth receive the pre-payment.
We offer youth who participate in the 1.5-hour focus groups a $40 gift card. Focus group data are not intended to be representative in a statistical sense, in that they will not be used to make statements about the prevalence of experiences youth and young adults with experience in the child welfare system. However, it is important to secure participants with a range of background characteristics to capture a variety of possible experiences. All participants are expected to be low-income youth and young adults. Without offsetting the direct costs incurred by respondents for attending the focus groups, such as child care, additional use of data plans or minutes or phones, or transportation if focus groups are able to be held in-person safely, the research team increases the risk that only individuals able to overcome financial barriers to attend will participate in the study. The $40 gift card will help offset these incidental expenses that may otherwise prevent their participation.
A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing
Personally Identifiable Information (PII)
PII is collected on consent forms and through the youth surveys. Each youth is assigned a unique study ID for the duration of the study. This ID is linked to the user-specific URLs that youth will use to access the web-based surveys. The unique ID is used to link survey responses by a single respondent across surveys. PII will be stored in secure files, separate from survey and other individual-level data.
Table A.3 below lists the PII that will be collected and its use.
Table A.3. Personally Identifiable Information (PII) To Be Collected and Its Use
PII |
Intended Use |
First and last name |
Contact information collected through the consent forms and surveys will be used to contact respondents about completing the follow-up surveys. |
Address (street, city, state, and zip code) |
|
Phone numbers (cell, home) |
|
Email address |
|
|
|
First and last name for alternate contacts |
Youth participants will be asked to provide contact information for up to three people who will know their location should they move during the study. This information will be used only to help locate respondents for follow-up data collection. We will not reveal anything about the respondents or the focus of the study when reaching out to alternate contacts. |
Address (street, city, state, and zip code) for alternate contacts |
|
Phone numbers (cell, home) for alternate contacts |
|
Email address for alternate contacts |
|
Date of birth |
Date of birth will be used to verify respondent at follow-up rounds of data collection. |
Race |
These variables will be used to assess baseline equivalence and examine whether Pathways is particularly effective for key subgroups of the target population. They will include at a minimum a baseline measure of the outcome and demographic characteristics, because these variables are likely to be strongly predictive of the outcomes of interest.
|
Ethnicity |
|
Employment status |
|
Estimated monthly income |
|
History of arrests/convictions |
|
State child welfare identification (ID) number |
We will request the youth’s state child welfare ID number from the Chafee workers when they enroll youth. We will use this ID to help communicate with Chafee workers about youth throughout the study. |
PII is not kept in the same location as any data collected. Access to respondents’ contact information is restricted to those working on the SYSIL evaluation. Any files containing PII are stored on Mathematica’s network in a secure project folder whose access is limited to select project team members. Only the principal investigator, project director, and key study staff have access to this folder. Furthermore, approved study team members can only access this folder after going through multiple layers of security. A secure FTP site (Box.com) will be used to transfer administrative data, which will contain as limited PII as possible.
Assurances of Privacy
Information collected is and will be kept private to the extent permitted by law. Respondents are and will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law (see Appendix A). At least some of the information collected under this ICR will likely be retrieved by an individual’s personal identifier in a way that triggers the Privacy Act of 1974, as amended (5 U.S.C. 552a). The system of records notice (SORN) for this collection is OPRE Research and Evaluation Project Records, 09-80-0361. Each individual will be provided with information that complies with 552a(e)(3) prior to being asked for information that will be placed into that system of records. This means respondents will receive information about the authority, the purposes for use, the routine uses, that the request is voluntary, and any effects of not providing the requested information. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.
Due to the sensitive nature of this research (see A.11 for more information), the evaluation obtained a Certificate of Confidentiality. The study team applied for this Certificate and will provide it to OMB if requested. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.
The project team has Institutional Review Board (IRB) approval from the Health Media Lab IRB.
Data Security and Monitoring
As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.
As specified in the evaluator’s contract, the contractor shall use Federal Information Processing Standard-compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard. The contractor shall ensure that this standard is incorporated into the contractor’s property management or control system and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with National Institute of Standards and Technology (NIST) requirements and other applicable federal and departmental regulations. In addition, the contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents with sensitive information or PII that ensures secure storage and limits on access.
Interviews and focus groups were and will be recorded only with permission from participants. Before the discussions begin, participants will be informed that we would like to record the discussion and will be asked to give their permission. Discussions will not be recorded if the participants do not give their permission. Participants will be asked to keep each other’s information private (see Appendix A).
Recordings from the interviews and focus groups are deleted as soon as they have been transcribed. We have and will continue to wait to begin recording the discussion until after everyone has introduced themselves. The transcribed notes do and will not include any names. All recordings are and will be stored on Mathematica’s secure network and destroyed per the contract requirements.
The survey data will be archived with the National Data Archive on Child Abuse and Neglect (NDACAN) (hhs.gov).
A11. Sensitive Information 9
This study includes questions on sensitive topics. It is necessary to include these questions in order to describe the population and determine baseline equivalence between the treatment and comparison groups, to measure the impacts of Pathways on the targeted outcomes (Research Question 1), and to determine whether Pathways is particularly effective for key subgroups of the population (Research Question 2).
Table A.4 includes the sensitive topics found in the youth survey, along with the justification for their inclusion.
The items were carefully selected based on experience and consultation with outside experts. We have considered whether the benefits and utility of the measures outweigh concerns about the heightened sensitivity among sample members and program staff to specific issues. We obtained feedback from youth during the pre-test on the proposed measures and asked specifically about questions that they may consider to be too sensitive. Items that pre-test respondents raised as being too sensitive were removed from the current version of the survey.
Table A.4. Summary of Sensitive Questions To Be Included in the Youth Survey and Their Justification
Topic |
Justification |
Race |
Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses. |
Ethnicity |
Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses. |
Gender identity |
Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses. |
Sexual orientation |
Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses. |
Ever experienced homelessness |
Will use for baseline equivalence. |
Risk behaviors (including unprotected sex, drug and alcohol use, domestic violence) |
Will use to assess baseline equivalence between groups; identify subgroups for subgroup analyses and assess social-emotional well-being, a key outcome domain. |
Income |
Related to employment and self-sufficiency, which may affect housing stability, both key outcome domains for the study. |
Criminal justice history |
This is a key outcome domain for the study. Involvement in the criminal justice system makes it harder to find employment and to secure stable housing, both of which are key outcomes of the study. |
Child welfare placement history |
Will use for baseline equivalence and to assess impacts for key subgroups (for example, looking at effects based on age at entry, time in care, and so on). This is also a key outcome domain for the study. |
Economic hardship |
Includes items such as missing meals or needing to borrow money from friends. These outcomes reflect a lack of self-sufficiency and may affect housing stability, a key outcome of the study. |
Government services received |
Will use to assess ability to access system resources, which is a key outcome domain for the study. |
Respondents may feel uncomfortable answering some of the questions on the surveys that address sensitive topics. To minimize this risk, youth are and will continue to be told they can skip any questions they do not want to answer. We have and will continue to train all staff who administer the surveys how to follow the protocols on dealing with sensitive questions and situations. Youth are able to end their participation in the study at any time without affecting the services they receive.
Prior to collecting baseline data, Mathematica seeks active consent from a parent, legal guardian, or legally authorized representative for those participants younger than age 18. Youth age 18 or older will be asked to consent for themselves. The consent form states that answers will be kept private to the extent permitted by law and not seen by anyone outside of the study team, that participation is voluntary, and that they may refuse to participate at any time without penalty (Appendix A). They are and will be informed that, to the extent permitted by law, individual identifying information will not be released or published; rather, data collection will be published only in summary form with no identifying information at the individual level. In addition, our protocol during the self-administration of the web instrument will provide reassurance that we take the issue of privacy seriously. It is and will continue to be made clear to respondents that identifying information will be kept separate from questionnaires.
The project team has IRB approval from the Health Media Lab IRB.
A12. Burden
Explanation of Burden and Cost Estimates
Table A.5 provides the remaining estimated annual burden and cost calculations for the data collection instruments included in this ICR.
We are expecting to enroll 500 youth and young adults in the study. As of May 2, 2024:
429 youth and young adults had enrolled,
380 youth and young adults had completed their baseline survey,
315 youth and young adults had received a 6-month follow-up survey, and
244 youth and young adults had received a 12-month follow-up survey
Youth Survey: Based on previous experience with similar instruments and the pre-test, the original version of the youth survey was estimated to take 30 minutes (0.5 hours) to complete. Taking into account the July 2023 OMB-approved deletions to the survey, the revised time estimated to complete the survey is 25 minutes (.42 hours).
The cost to respondents to the youth survey is estimated by assuming that 50 percent of youth will be age 18 or older at baseline,10 and then assigning a value to their time of $14.42 per hour, the 2024 Colorado minimum wage. The estimate of the proportion of youth who will be age 18 or older is based on the average age at intake for youth in Chafee services in comparison sites (18.6) and treatment sites (18.8).
Baseline. Over the next three years, we expect to survey about 120 youth at baseline. The total burden over three years is estimated to be 50.40 hours (120* .42). The annual burden for this data collection is estimated to be 16.8 hours (50.40/3).
Follow-Up 1 (6 Months). Over the next three years, we expect to survey approximately 185youth at the 6-month follow-up. The total burden over three years is estimated to be 77.70 hours (185 * 0.42) and the annual burden for this data collection is estimated to be 25.9 hours (77.70/3).
Follow-Up 2 (12 Months). Over the next three years, we expect to survey approximately 256 youth at the 12-month follow-up. The total burden over three years is estimated to be 107.52 hours (256* 0.42). The annual burden for this data collection is estimated to be 35.84hours (107.52/3).
Interview Guide for Pathways Sites (Treatment Sites) and Comparison Sites. Over the next three years, we expect to interview approximately60 additional program leaders and staff who deliver Pathways services and approximately 60 additional program leaders and staff who deliver comparison site services. The interview guide is estimated to take one and a half hours to complete. The total burden over three years is estimated to be 180 hours (120 x 1.5) total for treatment and comparison sites. The annual burden for this data collection is estimated to be60 hours (180/3) total for treatment and comparison sites.
The hourly wage estimate program leaders and staff is based on the mean hourly wage rate ($46.81) for social and community service managers (State Occupational Employment and Wage Estimates for Colorado, Bureau of Labor Statistics, Department of Labor, May 2023).
Program Director Check-ins for Pathways sites (Treatment Sites)11 and Comparison Sites. Check-in for Pathways and comparison sites are complete. Over the next three years, no additional check-in calls will be conducted.
Focus Group Discussion Guide for Pathways Youth (Treatment Youth) and Comparison Youth. Over the next three years, we expect to interview about 50 additional youth who receive services at Pathways sites and about 50 additional youth who receive services at comparison sites. If a youth requests a one-on-one interview instead of the focus group, we will accommodate the request by using the same discussion guide and adhering to the same amount of burden per individual. Based on previous experience with similar interviews, the focus group is estimated to take one and a half hours to complete. The total burden over three years is estimated to be 75 hours (50*1.5=for either treatment youth and comparison youth, or 150 hours total. The annual burden for this data collection is estimated to be 50 (150/3 = 75) hours total (or 75/3 = 25 hours each) for treatment and comparison youth.
Contact Information Update Requests. Over the next three years, we will implement two short contact information requests to provide an opportunity for youth to confirm their current contact information and continue to engage and familiarize youth with the SYSIL study (Instrument 5). The information requests will be sent 3- and 9-months after study enrollment. We anticipate the requests will take youth 5 minutes to complete. We anticipate 120 youth will need to complete both the 3- and 9-month contact information update requests during the 3-year period of this information collection request. An additional 185 youth completed the 3-month but will need to complete the 9-month contact information update during the 3-year period of this information collection request. The total burden over three years is estimated to be 19.2 hours (.08 hours x 120 youth x 2 requests; an annual burden of 6.4 hours) plus 14.8 hours (.08 hours x 185 youth x 1 request; an annual burden of 4.93 hours). The total contact information update requests are 425 respondents, for a total of 34 hours, with an estimated annual burden of 11.34 hours (34/3).
Table A.5 Estimated Annualized Cost to Respondents
Instrument |
No. of Respondents (total over request period) |
No. of Responses per Respondent (total over request period) |
Avg. Burden per Response (in hours) |
Total Burden (in hours) |
Annual Burden (in hours) |
Average Hourly Wage Rate |
Total Annual Respondent Cost |
SYSIL Youth Survey – Baseline |
120 |
1 |
0.42 |
50.4 |
16.8 |
$14.42 |
$242.26 |
SYSIL Youth Survey – Follow-Up Survey 1 (6 months) |
185 |
1 |
0.42 |
77.70 |
25.9 |
$14.42 |
$373.48 |
SYSIL Youth Survey – Follow-Up Survey 2 (12 months) |
256 |
1 |
0.42 |
107.52 |
35.84 |
$14.42 |
$516.81
|
Interview Guide for Pathways Sites (Treatment Sites) |
60 |
1 |
1.5 |
90 |
30 |
$46.81 |
$1,404.30 |
Program Director Check-ins for Pathways Sites (Treatment Sites)* |
0 |
0 |
.5 |
0 |
0 |
$46.81 |
$0 |
Interview Guide for Comparison Sites |
60 |
1 |
1.5 |
90 |
30 |
$46.81 |
$1,404.30 |
Program Director Check-ins for Comparison Sites* |
0 |
0 |
.5 |
0 |
0 |
$46.81 |
$0 |
Focus Group Discussion Guide for Pathways Youth (Treatment Youth) |
50 |
1 |
1.5 |
75 |
25 |
$14.42 |
$360.50 |
Focus Group Discussion Guide for Comparison Youth |
50 |
1 |
1.5 |
75 |
25 |
$14.42 |
$360.50 |
Contact Information Update Requests – 2 responses per respondent |
120 |
2 |
.08 |
19.2 |
6.4 |
$14.42 |
$92.29 |
Contact Information Update Requests – 1 response per respondent |
185 |
1 |
.08 |
14.8 |
4.93 |
$14.42 |
$71.14 |
Total |
|
|
|
599.62 |
199.87 |
|
* The program director check-ins do not require a unique instrument; they will be conducted using a subset of items from the interview protocol.
A13. Costs
There are no additional costs than what is outlined in A12 and A14.
A14. Estimated Annualized Costs to the Federal Government
The total cost to the federal government for the data collection activities under this ICR is estimated at $2,916,979.81. Annualized costs to the federal government are estimated at $979,326.60 for the proposed data collection. These estimates of costs are derived from Mathematica’s budgeted estimates and include labor rates and, direct and indirect costs and are displayed below in Table A.6.
Table A.6. Estimated Annualized Costs
Cost Category |
Estimated Costs |
Field Work |
$506,469.46 |
Analysis |
$1,070,700.07 |
Publications/Dissemination |
$1,339,810.28 |
Total costs over the request period |
$2,916,979.81 |
Annual costs |
A15. Reasons for changes in burden
The burden table shows the estimated burden remaining for data collection under this ICR. The annual burden for this information collection request is 199.87, which takes into account the burden already incurred and the request for additional burden associated with additional round of interviews and focus groups. The annual cost for this information collection request is $4,825.57, accounting for the burden costs incurred and increased for the proposed additional round of focus groups and interviews and the use of updated cost data.
A16. Timeline
Table A.7 presents the estimated timeline for data collection, analysis, and reporting for the impact and implementation study. The survey data will be archived, likely with the NDACAN (hhs.gov).
Table A.7. Plan and Time Schedule for Information Collection, Tabulation, and Publication
Project Activity |
Time Period |
Impact Study |
|
Data collection (baseline youth surveys) |
36 months, following OMB approval |
Data collection (follow-up youth surveys) |
48 months, beginning 6 months after baseline+ |
Data analysis |
36 months, 3–6 months after data collection begins |
Draft final report |
About 6 months after completion of data analysis |
Revised final report |
About 9 months after completion of data analysis |
Implementation Study |
|
Data collection (first round of focus groups and interviews) |
2–3 months, 4–6 months after enrollment begins |
Data collection (first round of Program Director check-ins) |
|
Data collection (second round of focus groups and interviews) |
2–3 months, 36 months after enrollment begins |
Data collection (second round of Program Director check-ins) |
1 month, about 24 months after enrollment begins |
Data collection (third round of focus groups and interviews) |
2–3 months, 45 months after enrollment begins (9 months after enrollment ends) |
Data analysis |
6–12 months after data collection begins |
Draft final report |
About 6 months after completion of data analysis |
Revised final report |
|
+ This time period represents the data collection period for all rounds of follow-up data collection: 6-month follow-up survey (to begin 6 months after baseline and continue for 24 months); 12-month follow-up survey (to begin 12 months after baseline and continue for 24 months).
A17. Exceptions
No exceptions are necessary for this information collection.
Attachments
Instrument 1: SYSIL Youth Survey (Baseline and Follow-Ups 1-2)
Instrument 2: Interview Guide – Updated June 2024
2a: Interview Guide (Treatment Sites)
2b: Interview Guide (Comparison Sites)
Instrument 3: Program Director Check-ins
3a: Program Director Check-ins (Treatment Sites)
3b: Program Director Check-ins (Comparison Sites)
Instrument 4: Focus Group Guide – Updated June 2024
4a: Focus Group Guide (Treatment Youth)
4b: Focus Group Guide (Comparison Youth)
Instrument 5: Contact Information Update Request
Appendix A: Consent and Assent Forms (revised July 2023)
Appendix B: List of Surveys Referenced
Appendix C: Emails and Text for Outreach to Youth
Appendix D: One-page Informational Documents for Implementation Study
References
Brown, M., L. Calderwood. “Can Encouraging Respondents to Contact Interviewers to Make Appointments Reduce Fieldwork Effort? Evidence from a Randomized Experiment in the UK.” Journal of Survey Statistics and Methodology 2, 484-497 (2014).
Couper, M.P. (2008). Designing effective web surveys. Cambridge, UK: Cambridge University Press.
Dion, R., A. Dworsky, J. Kauff, and R. Kleinman. “Housing for Youth Aging Out of Foster Care.” Washington, DC: Mathematica Policy Research, 2014.
Oh, A.Y., A. Caporaso, T. Davis, L. A. Dwyer, L. C. Nebling, B. Liu, and E. Hennessy. “Adolescents’ Participation in an Accelerometer Data Collection Component of a National Survey”. Field Methods (2021).
Pejtersen, J.H. “The effect of monetary incentive on survey response for vulnerable children and youths: A randomized controlled trial.” PLoS ONE 15(5), 2020.
U.S. Interagency Council on Homelessness (USICH). "Framework for Ending Youth Homelessness." Washington, DC: USICH, February 2013.
What Works Clearinghouse. “Assessing Attrition Bias.” Available at: https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_attrition_v2.1.pdf. 2013.
1 A “site” is a service unit that may include one or more counties. The 49 counties form 18 sites (10 intervention and 8 comparison) due to coordination of services across small adjacent counties.
2 The study team removed any questions that can be addressed through the use of administrative data
3 A “site” is a service unit that may include one or more counties. The 49 counties form 18 sites (10 intervention and 8 comparison) due to coordination of services across small adjacent counties.
4 This reflects changes approved by OMB in July 2023.
5 The Performance Management Information System (PMIS) is an online case management information system developed by the Center for Policy Research (CPR) for Pathways to Success.
6 When survey questions and data provided through administrative data are identical or similar enough to provide the necessary information, the project team removed the survey question(s) from the survey.
7 A dry bag is a flexible, waterproof bag with a roll-top closure. For SYSIL, the study contact information will be printed on the dry bag to help us familiarize youth with the study.
8 OMB approved the non-substantive change from a test of the $5 token of appreciation for the contact information (approved in November 2022) to a test of a two-part payment option for the 6-month follow-up survey in July 2023.
9 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.
10 For follow-up data collection, we estimate a 5 percent increase in the estimated number of youth age 18 and older for every six months. The estimated percentage of youth age 18 and older for follow-up data collection are: 55 percent for 6 months, and 60 percent for 12 months.
11 The program director check-ins do not require a unique instrument; they will be conducted using a subset of items from the interview protocol.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mathematica |
| File Modified | 0000-00-00 |
| File Created | 2024-07-22 |