Memo - NonSub Change - COVID-19

To: Josh Brammer,

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Caryn Blitz

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: July 24, 2020

Subject: Non-Substantive Change Request – Evaluation Studies of the Runaway and Homeless Youth Transitional Living Program (OMB #0970-0383)

This memo requests approval of non-substantive changes to the approved information collection, Evaluation Studies of the Runaway and Homeless Youth Transitional Living Program (TLP) (OMB #0970-0383)

The requested changes include (1) revising the study design from a pre-post outcomes study to a descriptive outcomes study with fewer in-person data collection time points (this will result in an overall reduction in burden), and (2) collecting information to help us understand the ways the novel Coronavirus/COVID-19 pandemic has affected TLP services and outcomes among runaway and homeless youth (RHY).

Background

The COVID-19 pandemic began just prior to the start of planned data collection with youth served in a sample of TLP programs that had been recruited from across the county. In light of the pandemic, ACF consulted with its TLP grant project officers to understand challenges grantees were experiencing in the field related to the pandemic, and what concerns study grantees were raising related to the potential added burden of study participation at this time. ACF is now proposing revisions to the approved information collection in order to reduce overall burden on participants and reduce reliance on in-person data collection. Specifically, we have determined that reliance on lengthy surveys to follow the outcomes of runaway and homeless youth over time is not the right option given health concerns about in-person survey administration, the local and state variation in stay-at-home orders, and other constraints on time and resources available for the evaluation. Because the emergence of COVID-19 has implications for the program services and outcomes central to the study—housing, education, and employment among an extremely vulnerable population—it is also important that the study account for TLP youths’ COVID-19-related experiences.

A descriptive study that relies primarily on existing administrative data, program data, and a brief survey of youth at baseline is better suited to answer the research questions, given the current public health context, available time, and resources.

Time Sensitivities

Due to time constraints on the funds available for the evaluation, the outcomes study (data collection and reporting) must be completed by September 30, 2021. To complete the study within the allotted timeframe, participant enrollment into the outcomes study will need to begin by early August 2020. A delay in launching study enrollment will result in a reduction of the number of quarters of post-enrollment administrative data that can be collected from NDNH within the study timeline and will have a severe negative impact on data analysis for the study because fewer quarters of post-enrollment employment data will be available for analysis.

Overview of Requested Changes

This change request proposes to modify the study design from the current pretest-posttest outcomes study design to a descriptive study design that will report on the means and frequencies of youth characteristics and outcomes before, during, and after TLP participation. The study will examine longitudinal trends in outcomes for a sample of up to 400 TLP youth, which reduces the number of respondents.

Under the revised study design, data to inform the study’s key outcomes will be obtained primarily from existing administrative data sources, with one brief survey of youth at baseline to collect background information. The administrative data sources will be the National Student Clearinghouse (NSC), National Directory of New Hires (NDNH), and TLP program data. This revision will reduce the burden on youth study participants from 1,104 hours to 659 hours.

Changes to the Study Design:

• Change the study design from a pretest-posttest outcomes study (examining change from baseline to 6 months later and 12 months later) to a descriptive study that will report on outcome means and frequencies and use latent growth curve analysis and survival analysis to examine average longitudinal trends in outcomes for the sample (before, during, and after program participation).

• Decrease the sample size to roughly 66 percent of the number of participants currently approved (i.e., from 600 youth to approximately 400 youth). This reflects the number of youth that can be reasonably recruited during the study’s timeframe and provide an adequate sample size for proposed analyses.

• Eliminate two waves of in-person data collection from youth (6- and 12-month follow up surveys). Direct data collection from youth will only occur once, upon study enrollment, rather than at three points in time. In lieu of youth follow-up surveys:

  • Add program data collection from TLP staff on youth outcomes (Youth Information Form). The data requested in this collection are data programs already collect about youth (see topic areas listed in the “Changes to Data Collection Instruments” section).

  • Decrease the average time for youth to complete the baseline collection from 0.62 hours to 0.28 hours by reducing the number of survey questions.

• Decrease the number of outcome domains examined from five to three. The currently approved outcomes study focuses on how youth who participate in the TLP fare in the areas of (1) stable housing, (2) positive social connections, (3) social and emotional well-being, and (4) education or (5) employment. We now seek to examine outcomes in the areas of (1) education, (2) employment, and (3) housing due to availability of administrative data sources to examine these outcomes.

• Eliminate data collection from participants on the following sensitive topics: substance use, sexual behavior, delinquent activity, involvement with law enforcement, health and mental health symptoms (e.g., depressive symptoms, PTSD symptoms), and participation in certain categories of social services (e.g., addiction/substance use treatment, mental health services). The study will collect participants’ SSNs and history of TANF receipt, which can be considered sensitive information. This information is necessary to match to NDNH and NSC administrative data.

• Reduce the amount of the incentive for respondents who complete the baseline survey (now called the Background Information Form) from $30 to $10.

• Eliminate tokens of appreciation for the 6-month and 12-month surveys, as they will no longer be administered.

Changes to Analytic Methods:

• Under the previous pretest-posttest design, the analysis was an examination of average individual change in outcomes from baseline to six months later and from baseline to 12 months later. Under the new descriptive study design, the analysis will calculate means and frequencies of outcomes for the sample before, during, and after program participation and use latent growth curve analysis and survival analysis to examine average longitudinal trends in outcomes for the sample before, during, and after program participation.

Changes to Study Materials:

• Update the youth and parent consent forms to add permission to collect administrative/program data.

• Eliminate the majority of questions contained in the Young Adult Baseline Survey (renamed the Background Information Form). The following questions will be retained:

  • CI_2. What is your birthdate?

  • H11a. Would you describe your gender as male, female, or something else?

  • I1. Are you of Hispanic, Latino, or Spanish Origin?

  • I2. What is your race?

• Add the following questions to the baseline survey:

  • What is your Social Security Number (SSN)?

    • [This question is being added because it is required to request matching data from the NDNH and NSC.]

  • At the time you enrolled in the TLP, had you or your family ever received welfare or cash assistance (also known as Temporary Assistance for Needy Families or TANF)?

    • [This question is being added because it provides essential baseline/background information about individuals (their families') self-sufficiency, an outcome the TLP seeks to improve.]

  • How did COVID-19 impact your housing? How did COVID-19 impact your education? How did COVID-19 impact your employment?

    • [These three questions are being added because they provide information to help contextualize findings in light of TLP youth's COVID-19-related experiences.]

• Eliminate the 6-month and 12-month Young Adult Follow-up Surveys.

• Add the Youth Information Form (YIF) to collect existing administrative data from program staff (i.e., data the TLP is already collecting from youth). This collection comprises 17 questions on the following topics:

  • At study enrollment: youth program entry date, TLP entry from Basic Center Programs (21-day shelter programs), housing status prior to entry, educational attainment and enrollment status, employment status, and SSN (verify the SSN provided on the youth Background Information Form or provide SSN if missing on that form).

  • At program exit: youth program exit date, exit destination, safe exit, educational attainment and enrollment status at exit, and employment status at exit.

  • At program exit: three questions about how COVID-19 affected programs' expectations for youth's education, employment, housing and exit planning.

    • [These three questions are being added because they provide information to help contextualize study findings in light of TLP youth's COVID-19-related experiences.]

  • At each aftercare contact: date of youth aftercare contact and current housing status (based on Homeless Management Information System living situation codes).

Changes to Estimated Burden:

The requested changes result in a net reduction in burden from 1,104 to 659 hours. The table below shows the changes between the current approval and the changes proposed in this request.

Instrument	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Annual Burden Hours
Background Information Form (Previously titled Young Adult Baseline Survey)	200 400	1	0.62 0.28	124 112
Young Adult 6-Month Follow Up Survey	200	1	0.61	122
Young Adult 12-Month Follow Up Survey	200	1	0.61	122
Youth Information Form (YIF)	30	48	0.38	547
	Total			659

Changes to Estimate of Cost to the Federal Government:

• The estimated annual cost has been updated to $660,866, which includes the cost of the research – survey design, information collection, analysis, and reporting – through the end of data collection.

Changes to Time Schedule for Information Collection, Tabulation and Publication:

• Pending OMB approval, participant enrollment into the study and collection of the baseline data will be reduced from occurring over a 24-month period to a 4-month period. Follow-up administrative data collection is projected to occur over an 8-month period for program data and up to 10-months for NDNH and NSC data.

5