SSA_TLPEval_Nonsubstantive Change_clean_07132020_for OMB

The Family and Youth Services Bureau (FYSB) and the Office of Planning, Research, Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seek approval for a non-substantive change to the currently approved information collection for the Evaluation of the Transitional Living Program (TLP) (OMB No. 0970-0383). The changes requested include (1) revisions to the study design from a pre-post outcomes study to a descriptive outcomes study with fewer data collection time points, and (2) collection of information to help us understand the ways the novel Coronavirus/COVID-19 pandemic has affected TLP services and outcomes among runaway and homeless youth (RHY).

The COVID-19 pandemic began just prior to the start of planned data collection with youth served in a sample of TLP programs that had been recruited from across the county. In light of the pandemic, ACF has reconsidered the study design and determined that reliance on lengthy surveys to follow the outcomes of runaway and homeless youth over time is not feasible given health concerns about in-person survey administration, local and state variation in stay-at-home orders, and other constraints on time and resources available for the study. Because the emergence of COVID-19 has implications for the program services and outcomes central to the study—housing, education, and employment among an extremely vulnerable population—it is also important that the study account for TLP youths’ COVID-19-related experiences. A descriptive study that relies primarily on existing administrative data, program data, and a brief survey of youth at baseline is better suited to answer the research questions, given the current public health context, available time, and resources.

The revised outcomes study design relies primarily on existing administrative data sources and eliminates both the 6-month and 12-month follow-up surveys with youth. The length of the revised baseline survey (Attachment C: Background Information Form) is significantly reduced to eliminate the majority of questions. These reductions reflect a narrowing of the scope of the research from focusing on outcomes in five areas (stable housing, positive social connections, social and emotional well-being, and education or employment) to three areas (housing status, education, and employment). We have also added a data collection form (Attachment D: Youth Information Form) which will be populated with existing TLP data by program staff. This non-substantive change request reduces the overall burden for the outcomes study.

Study Background

The purpose of the Evaluation of the Transitional Living Program Youth Outcomes Study is to provide ACF with information about how youth who participate in FYSB-funded TLPs fare over time in the areas of housing, education, and employment.

Currently approved data collection activities include three project components:

A Pilot Study to test the feasibility of using a random assignment study design with the TLP and runaway and homeless youth;
Interviews with TLP staff as part of a qualitative study to gain insight into program implementation (Implementation Interviews); and
A Youth Outcomes Study that uses a pretest/posttest study design and data collected through youth surveys to measure changes in youth outcomes associated with participating in the TLP.

The first two data collection components are complete. We propose the following changes to the Youth Outcomes Study:

Eliminate the 6-month and 12-month follow-up surveys;
Decrease the sample size from 600 participants to 400;
Reduce the youth baseline survey to a short Background Information Form (BIF) to be completed by participants upon study enrollment;
Supplement the BIF data with existing program data already being collected by TLP grantees, collected using a Youth Information Form (YIF); and
Use existing administrative data from the National Student Clearinghouse (NSC) and the National Directory of New Hires (NDNH)¹ to measure education and employment outcomes, respectively.

In reducing the baseline survey to a briefer BIF, we propose three other changes. First, we will request that participants provide their social security number (SSN) on the BIF so that participant SSNs can be used to collect administrative data from NDNH and NSC records, which require a SSN to match individuals to their records.² Second, we will request that youth complete a question on the BIF to indicate whether they or their family had ever received Temporary Assistance for Needy Families (TANF) benefits at the time they enrolled into TLP. This question is being added because it provides essential baseline/background information about individuals’ (their families’) self-sufficiency, an outcome the TLP seeks to improve. Third, we will ask that youth complete three questions on the BIF describing how the COVID-19 pandemic has influenced their housing, education and employment. These three questions are added to help contextualize study findings in light of the COVID-19 pandemic. Overall, the updates to the BIF reflect a significant reduction in the level of burden placed on study participants.

Legal or Administrative Requirements that Necessitate the Collection

The Runaway and Homeless Youth Act (RHYA), as amended by Public Law 106-71 (42 U.S.C. 5701 et seq.), provides for the Transitional Living Program (TLP), a residential program designed to prepare older homeless youth ages 16-21 for a healthy and self-sufficient adulthood. The following amendment was included in the 2003 “Runaway, Homeless Youth and Missing Children’s Assistance Act” (Public Law 108-96), which reauthorized the RHYA. This amendment directed the Secretary of Health and Human Services to evaluate RHYA part B programs to report on long-term housing outcomes for youth after exiting the program.

SEC. 119. STUDY OF HOUSING SERVCIES AND STRATEGIES

The Secretary of Health and Human Services shall conduct a study of programs funded under part B of the Runaway and Homeless Youth Act (42 W.S.C. 5714-1 et seq.) to report on long-term housing outcomes for youth after exiting the program. The study of any such program should provide information on housing services available to youth upon exiting the program, including assistance in locating and retaining permanent housing and referrals to other residential programs. In addition, the study should identify housing models and placement strategies that prevent future episodes of homelessness.

The RHYA was again reauthorized in 2008 under the Reconnecting Homeless Youth Act of 2008 (Public Law 110-378) and in 2018 under the Juvenile Justice and Delinquency Prevention Act of 2018 (Public Law 115-385).

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The Youth Outcomes Study is designed to describe youths’ average housing, education, and employment outcomes before, during, and after their TLP experience. In addition, the study will describe how TLP youths’ housing, education and employment service experiences were affected by the COVID-19 pandemic. To accomplish this, the research team will use descriptive statistics to report mean levels and frequencies of youth characteristics, COVID-19 experiences, and outcomes. The research team will use techniques such as survival analysis and latent growth curve analysis to describe trends in education and employment outcomes over time. To the extent the sample size permits, these findings will be associated with youth or program characteristics.

Research Questions

The study will address the following research questions:

What are the prevalence and distribution of housing stability, education, and employment outcomes among TLP youth before, during, and after program participation (as examined using means and frequencies)?
What are the trajectories of employment and earnings among TLP youth before, during, and after program participation (as examined using latent growth curve analysis)?
What is the extent to which youth sustain employment (overall and with the same employer) and enrollment in post-secondary education (as examined using survival analysis)?
What are the differences in primary study outcomes for key subgroups of youth based on demographic and program characteristics (as examined using cross-tabs and graphical display of trends)?
In what ways are TLP youth's housing, education, and employment experiences affected by the COVID-19 pandemic?

Study Design

The research team recruited 30 study sites from among the organizations awarded TLP grants in 2017 and 2018 to participate in the Youth Outcomes Study. Data collection will involve up to 400 runaway and homeless youth who enroll in the TLP at one of the 30 study sites and agree to take part in the study. All youth participating in these 30 TLP sites will be invited to enroll in the outcomes study and undergo informed consent (or assent with parental consent in the case of minors).³ (See Attachments A and B for youth and parent study consent forms.) Youth who agree to participate will be asked to complete the BIF (Attachment C) immediately after completing the informed consent form and agreeing to participate in the study.

The mode of data collection will be consistent across all sites. The BIF will be completed by youth (with the help of TLP staff if necessary). For privacy and data security reasons, the BIF will be collected via a secure web-based platform accessible from any Internet-enabled computer (desktop, laptop, tablet, or other such device).

Universe of Data Collection Efforts

The proposed collection for the Youth Outcomes Study involve youth ages 16-21 who are participating in the TLP, and who have experienced homelessness resulting in their participation in the TLP at one of 30 study sites across the U.S. Exhibit A.2.1 describes the data sources for the Youth Outcomes Study.

Exhibit A.2.1 Overview of Data Sources

	Component	Timing	Purpose	Estimated Burden
1	Background Information Form (previously Young Adult Baseline survey; Attachment C)	Collected from participants at the time of enrollment	Measure background characteristics, and information needed for administrative data matching	0.28 hours per response
2	Youth Information Form (Attachment D)	Collected from TLP staff members over a period of up to 8 months	Measure youths’ outcomes on housing, education, and employment via program-level data already collected by TLPs	0.38 hours per response
3	National Directory of New Hires	Requested quarterly over a period of up to 10 months	Measure youths’ connection to employment and collect wage records	n/a – existing administrative data
4	National Student Clearinghouse	Requested twice, over a period of up to 10 months	Measure youths’ connection to education including enrollment into a post-secondary institution and completion of certificates and degrees	n/a – existing administrative data

The first collection, the BIF, will be administered to RHY participating in the TLP and will occur upon youths’ enrollment into the study (prospective collection). The main purpose of the BIF is to collect a limited set of identifiers and background characteristics necessary for the study. The data elements are: date of birth, social security number, gender, race, ethnicity, history of TANF receipt at the time of TLP enrollment, and implications of the COVID-19 pandemic for youth's housing, education and employment. Participants’ names, dates of birth, and/or SSNs will be used to request data matching to existing administrative data sources to collect information on youths’ connections to housing, education, and employment before, during, and after their time in TLP. The BIF will also request how youth participants want to receive their gift card. Many of these characteristics will be used as statistical controls in the analysis, and for subgroup analysis as appropriate.

The second data collection component, the Youth Information Form (YIF), draws on existing program data already being collected by TLP grantees. Burden hours are requested to allow for TLP staff to transfer a limited set of data elements from this existing program data to the study team via a secure web-based study portal. In completing the YIF, TLP staff will provide the following data elements at the following time points (note these data may be provided either retrospectively or prospectively):

Youth status at enrollment: youth program entry date, TLP entry from a youth shelter (i.e., Basic Center Program or BCP) (yes/no), housing status prior to entry (based on the U.S. Department of Housing and Urban Development’s standardized Homeless Management Information System (HMIS) codes for previous living situation), educational attainment and enrollment status, employment status, date of birth and social security number (SSN; as verification of the date of birth and SSN provided on the BIF and to supplement any missing dates of birth or SSNs on the BIF). We refer to these elements collectively as “entry data.”
Youth status at program exit: youth program exit status, youth program exit date, exit destination (based on HMIS exit destination codes), safe exit (based on RHY-HMIS⁴ data element), educational attainment and enrollment status, employment status, service receipt during the COVID-19 pandemic, and changes to service goals’ expectations in housing, education, employment and/or exit related to the COVID-19 pandemic. We refer to these elements collectively as “exit data.”
Youth aftercare contact post exit from the program: date of youth aftercare contact and current housing status (based on HMIS living situation codes). We refer to these elements collectively as “aftercare data.”

The third and fourth data sources (NDNH data and NSC data) are existing administrative data sets that supply employment and education status, respectively. Their use greatly reduces the response burden on study participants by allowing us to decrease the length of the BIF and eliminate follow-up surveys of youth participants. NDNH and NSC data will be collected retrospectively.

The data collection components are presented in Exhibit A.2.1.

A3. Improved Information Technology to Reduce Burden

The data collection procedures reflect sensitivity to issues of efficiency, accuracy, and respondent burden. The BIF and YIF will be collected via a custom built, secure web-based study portal to reduce burden, improve accuracy of responses, and ensure data security.

A4. Efforts to Identify Duplication

This study will focus on a population that has not previously been surveyed nationally. Where feasible, information will be gathered from existing data sources; the information being requested through surveys is limited to that for which the youth (in the case of the BIF) or program staff (in the case of the YIF) are the best or only information sources. Instead of directly surveying youth participants, the study will rely on pre-existing TLP program data, NDNH data, and NSC data to obtain outcomes on youths’ housing status, education, and employment.

The study will collect the BIF data directly from youth participants as this is the most reliable source of information about their personal identifiers and background characteristics queried in the BIF.

A5. Involvement of Small Organizations

Most TLP grantees are small entities, operated by community-based organizations. The plan for collecting data from youth (BIF) and staff (YIF) is designed to minimize burden on such sites by utilizing a web-based survey platform and providing training and technical support from the research team to help both staff and youth navigate the study portal.

For the program data collection, the contractor, Abt Associates, will provide training, support, ongoing monitoring, and other assistance to help grantee staff securely transfer their program data to Abt Associates via the YIF in a secure web-based portal. In addition, the contractor will also provide support and technical assistance to TLP staff who have difficulty accessing the portal or completing the form on their own.

Findings from the study will in no way impact the funding or management of the grantees, although improvements in program design may ultimately occur once the effects of the program are determined.

A6. Consequences of Less Frequent Data Collection

The data being collected are essential to conducting a study of the TLP as the reauthorization language (Public Law 108-96) requires. The data are necessary for describing youth housing, education, and employment outcomes before, during, and after TLP participation, and for understanding the consequences of a global health crisis on these outcomes. Furthermore, without this information, ACF’s research questions cannot be answered and additional program-related decisions about TLPs would be based on insufficient information on program effectiveness.

A7. Special Circumstances

There are no special circumstances for the proposed data collection effort.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 16, 2018, Volume 83, Number 95, page 22688, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment E. During the notice and comment period, we did not receive any public comments.

OPRE and FYSB have worked to partner with other agencies in the review of the research design as well as the collection tools. Staff and leadership from OPRE have weighed in. Additionally, FYSB project officers and technical assistance staff have provided feedback regarding the project and have performed programmatic assessments of the potential grantee organizations to ensure that they have the capacity to support a study of this nature.

Consultation with Experts Outside of the Study

Researchers have and will continue to consult with experts in their respective disciplines from ACF and Abt Associates in developing the data collection instruments, study design, and protocols for data collection.

A9. Incentives for Respondents

As previously approved, each youth will receive an electronic gift card to Amazon.com upon submission of his/her/their survey. The token of appreciation is $10 for the BIF. The Youth Outcomes Study is collecting data from a very hard-to-reach population of minority and underrepresented youth that tends to be particularly distrustful of strangers and government agencies and that does not easily disclose information of a personal or sensitive nature. We believe that the gift card amount is necessary to encourage participation in the study and willingness to allow for their administrative data to be collected.

The gift cards are important for this study because the study participants are unaccompanied runaway and homeless youth who are overwhelmingly disconnected, highly mobile, and hard to reach, making them particularly difficult to obtain data for over extended time periods. Research has shown that tokens of appreciation are effective at increasing response rates for populations similar to participants in TLP (i.e., lower education levels, minority, and underrepresented populations).^⁵

Without the modest tokens of appreciation OMB has approved, the study risks having unacceptably low enrollment rates into the study, which would lead to insufficient statistical power (due to a small analytic sample size) and nonresponse bias (when survey participants differ in meaningful ways from non-participants).

A10. Privacy of Respondents

Protections for privacy are embedded in the study design. Data collection will only occur if informed consent (or assent with parental consent in the case of minors) is provided by youth themselves. Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.

Copies of the youth and parent consent forms for the outcomes study are provided as Attachments A and B. To ensure the privacy and safety of the runaway and homeless youth participating in the study, in cases where obtaining parental consent would place the youth at risk of harm, the contractor will obtain a waiver of parental permission from its Institutional Review Board (IRB). Federal regulations permit the IRB to approve research without parent permission “if the IRB determines that a research protocol is designed for conditions or for a subject population for which permission is not a reasonable requirement to protect the subjects, provided an appropriate mechanism for protecting the children who will participate as subjects in the research is substituted and provided further that the waiver is not inconsistent with federal, state or local law.”

All identifying information that will be collected with each youth’s informed consent (or assent with parental permission in the case of minors) and all records will be protected with security systems designed to safeguard privacy. All data will be securely protected under IRB-certified ethical research methods. Participants will be made aware of this and the consent form will also convey that answers will be kept private, that youths’ participation is voluntary, and that they may refuse to participate at any time.

Due to the sensitive nature of this research (see Section A11 for more information), the study will obtain a Certificate of Confidentiality from the National Institutes of Health (NIH). The study team currently has a Certificate for the TLP Evaluation’s Pilot Study (with a random assignment design) and will update it with NIH to extend the protections to cover individuals participating in the TLP Evaluation’s Youth Outcomes Study. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier. The BIF and YIF data transfer will occur via a secure, encrypted, passcode-protected digital platform, which will capture and store data in real time. Both the study portal and the server on which Abt will store participant data comply with FISMA moderate standards.

The Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security Plan that assesses all protections of respondents’ personally identifiable information.

The Contractor shall implement systems that comply with NIST SP 800-53, Security and Privacy Controls for Federal Information Systems and Organizations applicable baseline requirements; and other applicable NIST guidance as well as HHS and ACF policies and other guidance to protect all instances of sensitive information during storage and transmission. The Contractor shall: ensure that these controls are incorporated into the Contractor’s property management/ control system and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information.

The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

A11. Sensitive Questions

Some questions in the BIF ask for information of a sensitive nature—for example, youths’ SSN, history of TANF receipt, housing circumstances, attachment to education and employment, and the implications of the COVID-19 pandemic for TLP youth’s housing, education, and employment. SSNs are necessary for matching to NDNH and NSC administrative data. The other topics are personal, but also essential to understanding a youth’s overall situation, and the consequences of the global pandemic in their lives. Data on these topics will be collected because the program being examined (TLP) is designed specifically to address and improve well-being related to these very factors.

Through the informed consent process, youth and their parents will be informed about the uses of youth’s sensitive information. Youth will be assured of privacy and told they do not have to answer a particular question if they do not wish to. Copies of the youth and parent consent forms for the Youth Outcomes Study are provided as Attachments A and B.

A12. Estimation of Information Collection Burden

The current approval for data collection under OMB #0970-0383 includes 140 burden hours for the Implementation Interviews component of the TLP Evaluation and 1,104 burden hours for the Youth Outcomes Study. The Implementation Interviews component is complete. To date, no data collection has started for the Youth Outcomes Study component. Exhibit A.12.1 provides an updated burden table, specific to the remaining Youth Outcomes Study component and incorporating changes resulting from updates to the study design. The Youth Outcomes Study will now involve a total of 400 respondents and data collection will take place over a period of up to 10 months. The total burden annualized over one year is expected to be 659 hours. These burden estimates are based on experience with similar data collection efforts for other studies.

There are no annual costs to youth respondents. The BIF will be completed by unpaid youth. The total annual cost for program staff time to complete the YIF is estimated to be $12,963.17.⁶

Exhibit A.12.1 Revised Annual Burden Estimates and Costs for the Youth Outcomes Study

Instrument	Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours*	Median Hourly Wage
Background Information Form (BIF)	400	1	0.28	112	--
Youth Information Form (YIF)	30	48	0.38	547.2	$23.69
Estimated Total Annual Burden Hours: 659
Estimated Total Burden Costs on Study Participants: $12,963.17

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost of the research activities covered under this request will be $660,866. This includes costs for recruiting and training sites; ongoing data collection, support, and monitoring; data analysis; and reporting.

Annual costs of Federal employee time is estimated to be: $12,700 Executive Management Oversight (10% FTE), and $39,962 Project COR and Task Lead (40% FTE).

A15. Change in Burden

The requested changes to study design result in a net reduction in burden of 445 hours.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

When data collection is completed, the Contractor will analyze the data and provide ACF with a written report detailing the findings. Detailed data tabulations will be part of the report and available for study. No personal information will be included in the report.

Pending OMB approval, participant enrollment into the study and collection of the BIF data will occur over a 4-month period (anticipated August through November 2020). Follow-up administrative data collection is projected to occur over a period of up to 8 months for the YIF (anticipated to occur August 2020 through March 2021) and up to 10 months for NDNH and NSC data (anticipated to occur September 2020 through at most June 2021, depending on whether the timing for the final report allows). Analysis and reporting would occur after BIF and administrative data collection is completed.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the OMB number and expiration date.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 The primary purpose of the NDNH, operated by the Office of Child Support Enforcement (OCSE), Administration for Children and Families, U.S. Department of Health and Human Services is to assist state child support agencies in locating parents and enforcing child support orders; however, Congress has authorized specific state and Federal agencies to receive information from the NDNH for authorized purposes. http://www.acf.hhs.gov/programs/css/resource/a-guide-to-the-national-directory-of-new-hires

2 NSC data can be matched based on name and date of birth or SSN. For youth who do not have a valid SSN, we plan to match NSC data based on study participants’ name and date of birth.

3 We will invite to participate in the study all youth who are either enrolled in a participating TLP during the study enrollment period or exited a participating TLP program within the past 12 months prior to the study enrollment period and had stayed in the TLP for at least 30 days.

4 The Runaway and Homeless Youth-Homeless Management Information System (RHY-HMIS) is a data repository used by RHY grantees to submit required program performance measures and other program data to FYSB. FYSB uses these data to monitor grantees’ performance and ongoing continuous quality improvement of the RHY Program.

5 Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto. 1992. An experiment in monetary incentives. In JSM proceedings, 393–98. Alexandria, VA: American Statistical Association.

6 An hourly wage of $23.69 for program staff is based on mean wages for SOC code 21-0000 – Community and Social Service Occupations as reported by BLS: https://www.bls.gov/oes/current/oes_nat.htm#21-0000.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	OPRE OMB Clearance Manual
Author	DHHS
File Modified	0000-00-00
File Created	2021-01-13

SSA_TLPEval_Nonsubstantive Change_clean_07132020_for OMB