Supporting Statement A - LTO Generic

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part A

April 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer:

Brett Brown

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the Formative Data Collections for ACF Research (0970-0356).

• Progress to Date: A previous generic OMB request was submitted for the Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity project in January 2018. An updated request was submitted for the above-mentioned request in July 2018. This is a new generic request that builds from the previous two.

• Description of Request: For this effort we will be conducting a series of semi-structured interviews to collect information from evaluation teams to assess the practical and legal feasibility for linking evaluation data to administrative data. The data collected in the study will not be generalized to a broader population. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Administration for Children and Families’ (ACF) Office of Planning, Research, and Evaluation (OPRE) at the U.S. Department of Health and Human Services (HHS) seeks approval to contact organizations to gather additional information about program evaluations for the Assessing Options to Evaluate Long-Term Outcomes (LTO) Using Administrative Data: Identifying Targets of Opportunity study.

ACF/OPRE launched the LTO study in 2017. Linking administrative data sets to program evaluation records is a promising and potentially low-cost means of tracking long-term impacts of social interventions; long-term defined here as greater than five years. The LTO project sets out to help ACF/OPRE understand the current parameters for and feasibility of linking employment- and child/youth development-related evaluations to administrative datasets.

To date, the LTO project has selected and reviewed evaluations and administrative data sources with the goal of assessing the practical, legal, and ethical feasibilities for accomplishing the linkages. Thirteen evaluations were identified as “potentially feasible” candidates for long-term follow-up based primarily on the existence of personally identifying information (PII) needed for matching key outcome data from an administrative source. Eight of the 13 evaluations have been selected for further examination.

This current information collection request will allow the team to provide guidance to ACF on the steps and resources needed to pursue long-term follow-up (using admin data) for each of the 8 studies in our sample. Based on the findings from this data collection effort, ACF may decide to fund long-term follow-up for one or more of these studies.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of the current information collection request is to gather information about the 8 major evaluations from:  

• Project directors, principal investigators, and other members of the original evaluation research teams of the major evaluations selected. 

• Evaluation legal representatives, Institutional Review Board (IRB) representatives, original (primary) study funders, and administrators of data sources that would be collected during long-term follow-up.

Data will be collected to assess the practical and legal feasibility for accomplishing the linkages (e.g. existence of personally identifiable information, IRB and consent form allowances or restrictions), assess potential costs where possible, determine data ownership, identify prior history of linkage, past findings, and current availability of relevant data and metadata. Information will be gathered from publicly available sources first, and then through outreach to individuals familiar with the evaluations.

The information gathered will help the study team assess the feasibility for matching the major evaluations’ records to administrative records. Through this analysis, we will not only be able to propose a promising configuration of evaluations and administrative data sets for future long-term follow-up efforts but may also be able to share best practices for future researchers doing this type of work. These data will be used to prepare an in-depth long-term outcome feasibility report that will feature a detailed profile for each of the 8 studies selected for the feasibility assessment

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for research and evaluation (0970-0356):

• inform the development of ACF research

• maintain a research agenda that is rigorous and relevant

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions

The project will answer the following research questions:

• Which evaluations have the most potential for long-term matching and what are the views of the various stakeholders as it relates to an extended follow-up study linking administrative data to evaluation data?

• What are the key research questions that might justify the risk to participants of conducting long-term follow-up research and what are the most feasible administrative data sources to pursue?

• What are key practical, legal, and ethical considerations when thinking about matching an evaluation dataset to administrative records?

• What are the key steps, estimated costs, and timeline for obtaining access to administrate data, assessing its suitability, and analysis?

Study Design

Our approach is centered around conducting semi-structured interviews with study stakeholders; this work will be staged across two waves (the interview protocols for both waves are in Instrument 1: Informed Consent and Interview Protocols).

In Wave 1, the project team will seek to interview one lead evaluation informant per study. These interviews are expected to yield critical information about the rationale and feasibility of conducting long-term follow-up research, identifying research questions that would motivate extended follow-up and outlining data sources, requirements, constraints and conditions that speak to feasibility. Wave 1 interviews will also include respondent-driven identification of Wave 2 respondents who could provide specialized insights to inform each study’s feasibility assessment.

• In Wave 2, the team will conduct additional interviews with respondents to drill-down on specific, applicable feasibility constructs. Categories of respondents are expected to include, evaluation data mangers, evaluation legal representatives, IRB representatives, original (primary) study funders, and possibly a practitioner representative.

All respondents will be provided an advance memo outlining the purpose of the interview and previewing the topical substance of each conversation (A: Appendix A - LTO Outreach Cover Memo).

This method of data collection allows us to gather critical information needed to determine if any of the 8 evaluations are viable candidates for long-term follow-up. It also allows us to identify additional key informants who can provide us with the information needed to make the feasibility assessment as it is unrealistic to expect one person to be knowledgeable about all of the information we are requesting.

This method of data collection is not without its limits. One limitation is that the questions are tailored to informants that have particular roles on study projects. There is a possibility that an informant may be able to address questions that were designed to be answered by respondents who had different roles. Another possibility is that the informant may not know the answer to a question because the informant’s role was more limited than expected. Some of the information may also be hard to access (particularly for older studies, where information is more likely to be stored in an archived format).

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Data Collection Wave 1	Wave 1 Interview Protocol	Respondents: Evaluation project director and/or Principal Investigator (PI) Content: Long-term follow-up research questions, program theory substantiating long-term follow-up, policy relevance preliminary perspective on select Wave 2 content, ethical parameters of long-term follow-up, identifying Wave 2 respondents, data availability Purpose: Yield critical information about the rationale and feasibility of conducting long-term follow-up research and identifying Wave 2 respondents	Mode: Email and tele-conference interviews Duration: 2.5 hours
Data Collection Wave 2	Wave 2 Interview Protocols	Respondents: Data manager, legal representative, data source administrators, original (primary) funder, and practitioner representatives Content: Data availability, quality of personally identifiable information, copies of study documents (e. g. informed consent, data collection instruments, codebooks, etc.), legal parameters of consent, data sharing agreements, study contracts, privacy and security standards, disposition of the data after the analysis, steps (and associated costs) for gaining access to specific data sets, ethical parameters of long-term follow-up, policy relevance, IRB oversight, study data ownership, accessibility of existing research findings for study participants Purpose: Yield critical information about the rationale and feasibility of conducting long-term follow-up research	Mode: Email and tele-conference interviews Duration: 2.5 hours

Other Data Sources and Uses of Information

The information collected will be combined with the information collected in the earlier phases of the LTO project (OMB number 0970-0356). All of the compiled information will be used to inform the feasibility assessment for linking administrative data to evaluation data to examine long-term outcomes and impacts.

A3. Use of Information Technology to Reduce Burden

Communication with the evaluation informants will occur through teleconferences and emails for outreach and follow-up. The interviews will be recorded to reduce the need for respondents to repeat information.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

During the original performance period, the project team reviewed publicly accessible information regarding the evaluations. Based on this review, some types of information about the evaluations are not readily available from public sources (e.g. whether the evaluation dataset contains personally identifying information, and the language used in the Informed Consent forms). Some of this information was collected in the earlier phases of the LTO project.

This current information collection request will go a step further; we will be asking the evaluation teams more in-depth questions about the rationale and feasibility of conducting long-term follow-up research. This is information that is not publicly available.

A5. Impact on Small Businesses

We do not expect this to impact small businesses.

A6. Consequences of Less Frequent Collection

Not collecting this information would severely limit the ability to learn which evaluations are good candidates for long-term follow-up and the ability to develop an in-depth long-term outcome feasibility report. Relying only on publicly available information and the information collected in earlier phases of the LTO project would provide an incomplete picture of the feasibility of doing long-term follow-up.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of the overarching generic clearance for formative information collection. This notice was published on October 11, 2017, Volume 82, Number 195, page 47212, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

We do not currently anticipate engaging any outside experts. 

A9. Tokens of Appreciation

No incentives for respondents are proposed for this information collection. 

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

The project team will collect contact information for potential Wave 2 respondents to contact and schedule interviews with Wave 2 respondents. Responses will be synthesized at the study-level; no contact information will be connected to responses.

Information will not be maintained in a paper or electronic system from which they are actually or directly retrieved by these respondents’ person identifiers.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Data Security and Monitoring

This project does not involve the collection of quantitative data and no individual level information about human subjects is being collected. The information collected will be stored on a restricted, password protected project folder for which only those who work on the project have access.

A11. Sensitive Information¹

There are no sensitive questions in this data collection.

A12. Burden

Explanation of Burden Estimates

For Wave 1 of data collection, we anticipate contacting one key informant for each of the 8 major evaluations identified in order to request information. For Wave 2 of data collection, we anticipate contacting 3-5 key informants for each of the 8 major evaluations.

We expect the average burden hours to be 2.5 per informant. 1 hour has been allocated for interview time and 1.5 hours have been allocated for gathering and review of the relevant documentation including project study contracts, study findings, data sharing agreements, informed consent form, IRB application, data dictionaries and codebooks. Across both waves of data collection, we anticipate 108 total burden hours.

Estimated Annualized Cost to Respondents

To compute the total estimated annual cost, the total burden hours were multiplied by $33.90, the average hourly wage for management, professional, and related workers in the civilian workforce as reported by the BLS NCS (2010).² The estimated total cost is $3,661.20.

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Wave 1 interviews	8	1	2.5	20	$33.90	$678.00
Wave 2 interviews	35	1	2.5	88	$33.90	$2,983.20
Total				108		$3,661.20

A13. Costs

There are no additional costs to respondents.

A14. Estimated Annualized Costs to the Federal Government

The total cost for the data collection activities under this current request will be $409,740. The estimate includes the costs of project staff time on reaching out to evaluation teams and conducting the semi-structured interviews. The cost also includes time to analyze the data collected and the creation of the long-term feasibility report. The annual costs to the Federal government are the same since the proposed data collection will take place within one year.

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$45,768
Field Work	$323,211
Publications/Dissemination	$40,761
Total/Annual costs over the request period	$409,740

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for ACF research (0970-0356).

A16. Timeline

Development of the data collection protocols for this request took place between November, 2019 and February, 2020. Initial outreach to evaluation informants and other relevant individuals for the purpose of information gathering will begin upon OMB approval of this information collection and is expected to take place for about 3 months. This timeline may be extended, as needed, given current uncertainty related to the COVID-19 pandemic. The analysis of the information collected will be summarized in an extensive internal report to ACF, expected in late 2020.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Appendix A - LTO Outreach Informant Interviews Cover Memo

Appendix B - LTO Outreach Informant Interviews Phone Script

Appendix C - LTO Outreach Informant Interviews Email

Instrument 1 - Informed Consent and Interview Protocols

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

2 U. S. Department of Labor, Bureau of Labor Statistics “National Compensation Survey: Table 1: Summary Mean hourly earnings and weekly hours for selected workers and establishment characteristics.” 2010. http://www.bls.gov/ncs/ocs/sp/nctb1344.pdf.

11