Appendix C - LTO Example Template

Appendix C

Evaluation Template – WorkAdvance example

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data

Evaluation Template: WorkAdvance (Example)

This data collection effort is part of the Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data study, which is funded by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services.

This study is a new effort by OPRE to determine the feasibility of linking administrative data sets to program evaluation data. Data are being collected to assess the practical and legal feasibility for accomplishing the linkages, to assess potential costs, to determine data ownership, and to identify prior history of linkage, past findings, and current availability of relevant data and metadata. The information collected on each evaluation will be submitted to OPRE, along with a memo that summarizes the findings and identifies the most promising targets of opportunity.

Can you please review the information in the template, confirm or correct as needed, and fill in any missing information? We expect it will take you 1-2 hours to review and fill in this template, depending on the availability of information.

Please note that your participation in this data collection is voluntary and your information will be kept private to the extent permitted by law.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 3/31/2018.

Evaluation
Lead evaluator	MDRC
Funder(s)	Social Innovation Fund grant to Mayor’s Office for Economic Opportunity (NYC)
Federal project officer	N/A
Status Ongoing or completed? Start and end date	Original evaluation is completed (2011-2016) Extended follow-up analysis is ongoing (2016-2019)

Evaluation dataset
Does the evaluation dataset still exist?	Yes
Data destruction Timing Data sources	- Extended follow-up analysis is ongoing - Consent and agreements say PII must be destroyed at completion of the study - Consent form/waiver signed by participants in Oklahoma for state UI data says "all individual data will be destroyed at the end of the project
Location/ownership Who has the dataset? Who owns the dataset?	MDRC (original evaluator)
Was a public access or restricted use file created?	No
What personally identifying information is available?	SSN First, middle, last name Date of birth Contact information (phone number, email, address)
Administrative datasets linked to in original study	State unemployment insurance wage and benefits data
Measures/data sources available in evaluation dataset	Individual-level data from: Baseline survey Year 2 survey Unemployment insurance wage and benefits MIS
Codebook/instruments available	Available from MDRC: - Baseline and survey instruments - Record layouts from administrative data - Quality check memos
Data sharing agreements for sample roster/link file	None

Feasibility
Have any extended follow-up analyses been done? Length of follow-up Data sources used	Yes - 3 years (completed) and 5 years (in process) - State unemployment insurance wage records - For 5 year, may also match to CARRA archives
Did the study collect informed consent?	Yes - Separate ICF used for each site (versions are very similar) - New York and Oklahoma sites had additional informed consent forms that were required by state agencies to release UI wage and benefits records
Informed consent description Are specific data sources mentioned? Is a length of data collection mentioned? Are there restrictions around who can access the data? Or when the data needs to be destroyed?	Data items/sources mentioned in consent form: - Identifying information - PII and contact information - Interviews - baseline and follow-up surveys, as well as interviews of participants' employers - Program tracking data on use of services - "(1) your employer, wages, hours of work, and unemployment insurance benefits from state and federal agencies, (2) your enrollment in, completion of, and funding for education and training and employment related activities from public, private, and nonprofit agencies such as, for example, WIA funded One-Stops and the National Student Clearinghouse"
	Length of data collection mentioned in consent form: - 3 years pre-enrollment - 10 years post-enrollment
	Language on data access/destruction in consent form: - "All personally identifying data about you will be destroyed upon the conclusion of the study"
Linking to administrative data sources Is it possible to link the evaluation dataset to other administrative data sources? If not, why not?	Yes, it is possible to match to at least some data sources through 10 years of follow-up. The language for employment and earnings data is fairly broad, and only mentions it may be collected from “state and federal agencies.”

Study and findings
Study design	Randomized controlled trial
Study enrollment period	June 2011-June 2013
Number of sites	4
Site location(s)	New York (2 sites) Tulsa, OK Cleveland, OH
Provider name(s)	Per Scholas St. Nicks Alliance Towards Employment Madison Strategies Group
Sample size Overall By research group	2,564 total sample members 1,293 program group members/1,271 control group members
Sample characteristics	Adults (average age in mid-30s), mostly male, mostly unemployed, nearly 100% had high school diploma and over half some college
Intervention type and key services tested	Sectoral training and career pathways program model Model components: - Intensive screening - Career readiness services - Occupational skills training leading to a credential - Job development and placement - Retention and advancement services
Key outcomes in original study (e.g. primary outcomes)	Employment, earnings Primary outcomes were earnings in Quarter 10 (from state UI data) and currently employed (from Year 2 survey)
Strength of findings Size of impact Consistency Timing Outcomes	- Large impacts on employment and earnings throughout most of follow up period for one site - Little to no effect on any economic outcomes in any time period for one site - Some impacts, mainly late in the follow up period and for the late cohort at two sites
Treatment contrast Were there participation impacts? On what services?	Yes, large participation impacts in all model components, including sectoral training, as measured by Year 2 survey
Control group embargo Length of embargo Ineligible services/programs	3 years Controls not eligible for services through providers Passive enrollment of Cs after embargo period
Follow-up in original study Length of follow-up Data sources used	- 2 Year survey (80 percent response rate; average respondent was interviewed 22 months after study enrollment) - 10 quarters of state unemployment insurance wage records