Supporting Statement A - Assessing Options LTO July 2018

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity

OMB Information Collection Request

0970 - 0356

Supporting Statement

Part A

January 2018

(Updated July 2018)

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Brett Brown

A1. Necessity for the Data Collection

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for contacting organizations for the purpose of gathering additional information about program evaluations for the Assessing Options to Evaluate Long-Term Outcomes (LTO) Using Administrative Data: Identifying Targets of Opportunity study. Permission to contact organizations for this purpose is requested under ACF’s generic clearance for formative data collections. This information collection falls under two goals of the formative generic: (1) inform the development of ACF research, and (2) maintain a research agenda that is rigorous and relevant.

This supporting statement is the an update of the previous OMB submission for the LTO study. In February 2018, OMB approved 40 burden hours for data collection related to the evaluation template. ACF is now requesting an additional 10 burden hours increase the number of respondents to the evaluation template from 20 to 25 respondents. This request includes burden remaining from the original request (4 respondents) in addition to burden for an additional 5 respondents.

Study Background

The Administration for Children and Families (ACF), Office of Planning, Research, and Evaluation (OPRE) has launched the Assessing Options to Evaluate Long-Term Outcomes (LTO) Using Administrative Data study. Linking administrative data sets to program evaluation records is a promising and potentially low-cost means of tracking long-term impacts of social interventions: long-term defined here as greater than five years.

The study will take a four-phase approach to identify promising evaluations and administrative data sets for possible linkage (shown in Exhibit 1 below). Phase 3 is the focus of this information collection request. Phases 1 and 2 will rely on publicly available information. Phase 4 is an analysis phase based on the information collected in previous phases.

In Phase 1: Scan, we will capitalize on information holdings at existing clearinghouses and our own knowledge to compile a set of evaluations that meet the basic criteria for inclusion.¹ In Phase 2: Curate, we will whittle this list down to 16 to 25 “major evaluations” based on the rigor and content of the studies. We will also begin collecting information on administrative records sources for potential matching. In Phase 3: Collect, we will collect more in-depth information on major evaluations and the potential for matching to administrative records. This includes information on data ownership, existence of personally identifying information, IRB and consent form allowance and restrictions, and past findings. In Phase 4: Analyze, we will analyze the full range of feasibility considerations for those studies and potential approaches for overcoming any challenges, and draft an internal report. MDRC is the contractor for this work.

Exhibit 1: Long-Term Outcomes Study Four-Phase Work Plan

Legal or Administrative Requirements that Necessitate the Collection

There are no legal or administrative requirements that necessitate the collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

The purpose of the current information collection request is to seek approval to gather information about the 16 to 25 more major evaluations from:

• Project directors, principal investigators, and other members of the original evaluation research teams of the major evaluations selected.

• Data archiving staff and evaluation contractors familiar with the major evaluations selected.

Data will be collected to assess the practical and legal feasibility for accomplishing the linkages (e.g. existence of personally identifiable information, IRB and consent form allowances or restrictions), assess potential costs where possible, determine data ownership, identify prior history of linkage, past findings, and current availability of relevant data and metadata. Information will be gathered from publicly available sources first, and then through outreach to individuals familiar with the evaluations.

The information gathered will help the study team assess the feasibility for matching the major evaluations’ records to administrative records. Through this analysis, we will not only be able to propose a promising configuration of evaluations and administrative data sets for future long-term follow-up efforts, but will also be able to share best practices for future researchers doing this type of work. The resulting internal report to ACF will summarize findings and identify the most promising targets of opportunity to link existing evaluations to administrative data providing long-term outcomes.

Research Questions

The study will explore the following research questions:

• Which evaluations have the most potential for long-term matching, and from which administrative data source(s) might key outcomes be measured?

• What are the key considerations when thinking about matching an evaluation dataset to administrative records?

• What activities are required to link evaluation datasets to administrative data sources containing long-term outcomes?

• Among current evaluations, what can be done to improve the capacity to link the evaluation data to administrative data in the future?

Study Design

As mentioned, our study follows a four-phase approach and is designed to reduce the scope of data collection by ruling out some studies based on publicly available information. This will help to minimize burden.

This data collection effort is part of Phase 3: Collect and is a systematic review of past evaluations using standard questions. These questions are listed in an evaluation template that will be filled in by the research team and then sent to individuals familiar with the evaluations to review and fill in any missing information (see Appendix A).

Universe of Data Collection Efforts

Evaluation Template (Appendix A). The only external data collection effort involved with the current request for approval consists of gathering more in-depth information on selected evaluations and the potential for matching to administrative records by contacting evaluation research teams, data archiving staff, and evaluation contractors.

A3. Improved Information Technology to Reduce Burden

The burden on the evaluation research teams and other individuals that will be contacted is minimal and the study team plans to use improved information technology wherever possible. When available, information from the internet will supplement requests for information in order to minimize burden. Communication with the evaluation research teams will be mainly through email and an electronic version of the evaluation template will be shared so individuals can easily review and insert information.

A4. Efforts to Identify Duplication

Before reaching out to the original evaluation teams, MDRC will review all publicly accessible information regarding the evaluations. Based on our initial review, some types of information about evaluations are not readily available (e.g. whether the evaluation dataset contains personally identifying information, and the language used in the Informed Consent forms) from public sources. To our knowledge, this information has not been centralized in one location and has not been collected previously. Questions posed to the original evaluation teams and other relevant individuals will only be aimed at confirming information previously collected and addressing gaps in the publicly accessible information.

A5. Involvement of Small Organizations

To minimize burden, we have made the data collection process as flexible as possible. Individuals completing the template will be able to do so at times convenient for them.

A6. Consequences of Less Frequent Data Collection

Not collecting this information would severely limit the ability to learn which evaluations are good candidates for long-term follow-up. By restricting the questions posed to the evaluation research teams and other relevant individuals to only those questions that cannot be answered using publicly accessible sources, we will avoid undue burden on the respondents.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the generic clearance for information collection. This notice was published on September 15, 2014, Volume 79, Number 178, page 54985, and provided a sixty-day period for public comment. The second notice was published on January 9, 2015, Volume 80, Number 6, page 1420, and provided a thirty-day period for public comment. ACF did not receive any comments.

Consultation with Experts Outside of the Study

We do not currently anticipate engaging any outside experts.

A9. Incentives for Respondents

No incentives for respondents are proposed for this information collection.

A10. Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract, are trained on data privacy issues.

A11. Sensitive Questions

There are no sensitive questions in this data collection.

A12. Estimation of Information Collection Burden

Previously Approved Information Collections

The last generic clearance request was approved for 40 burden hours.

Newly Requested Information Collections

As of June 2018, we used 32 of the 40 approved burden hours. In order to do a second round of data collection where we will reach out to 9 additional respondents, we will need 18 burden hours.We have 8 burden hours remaining from the previous generic clearance approval and are now requesting an additional 10 burden hours. The estimated burden hours for this renewal are based on our previous experience with the data collection tool.

Total Burden Requested Under this Information Collection

We anticipate contacting one key informant for each of the 16 to 25 major evaluations identified in order to request information. We anticipate it will take respondents two hours, on average, to complete the data collection.

Instrument	Total Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Evaluation template	9	1	2	18	$33.90	$610.20
Estimated Annual Burden Total				18		$610.20

Total Annual Cost

To compute the total estimated annual cost, the total burden hours were multiplied by $33.90, the average hourly wage for management, professional, and related workers in the civilian workforce as reported by the BLS NCS (2010).² The estimated total cost is $610.20.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The total cost for the data collection activities under this current request will be $131,475. The estimate includes the costs of project staff time on collecting the information for the evaluation template and reaching out to evaluation teams. The annual costs to the Federal government are the same since the proposed data collection will take place within one year.

A15. Change in Burden

We are requesting an additional 10 burden hours to cover the second round of data collection. We have 8 burden hours remaining from the previous generic clearance approval.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

Initial outreach to evaluation research teams and other relevant individuals for the purpose of information gathering started in Quarter 2, 2018 and will be completed by Quarter 1, 2019 (pending OMB approval). The analysis of the information collected will be summarized in an internal report to ACF. If there are sufficient resources and methodological interest, we may also produce and publish a summary brief for the public.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1We are investigating employment-related evaluations as well as evaluations that focus on some aspect of child or youth development, where there is an opportunity to use administrative data to look at employment and other outcomes in early adulthood.

2 U. S. Department of Labor, Bureau of Labor Statistics “National Compensation Survey: Table 1: Summary Mean hourly earnings and weekly hours for selected workers and establishment characteristics.” 2010. http://www.bls.gov/ncs/ocs/sp/nctb1344.pdf.

8