Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity

Appendix A

Evaluation Template

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data

Evaluation Template

This data collection effort is part of the Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data study, which is funded by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services.

This study is a new effort by OPRE to determine the feasibility of linking administrative data sets to program evaluation data. Data are being collected to assess the practical and legal feasibility for accomplishing the linkages, to assess potential costs, to determine data ownership, and to identify prior history of linkage, past findings, and current availability of relevant data and metadata. The information collected on each evaluation will be submitted to OPRE, along with a memo that summarizes the findings and identifies the most promising targets of opportunity.

Can you please review the information in the template, confirm or correct as needed, and fill in any missing information? We expect it will take you 1-2 hours to review and fill in this template, depending on the availability of information.

Please note that your participation in this data collection is voluntary and your information will be kept private to the extent permitted by law.

An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 3/31/2018.

Evaluation
Lead evaluator	Please verify the information that we prepopulated.
Funder(s)	Please verify the information that we prepopulated.
Federal project officer	Please indicate the project officer for the original evaluation.
Status Ongoing or completed? Start and end date	Please indicate the dates of the contract for the original evaluation.

Evaluation Dataset
Does the evaluation dataset still exist?	Yes/no
Data destruction Timing Data sources	Please indicate whether any data destruction has been conducted, what data were destroyed, and any other restrictions around who can access the data.
Location/ownership Who has the dataset? Who owns the dataset?	Please indicate the location and ownership of the dataset.
Was a public access or restricted use file created?	Yes/No. If yes, please indicate which type of file was created and where the file(s) are stored.
What personally identifying information is available for a potential match to an administrative data source?	For example, SSN, name, date of birth, contact information.
Administrative datasets linked to in original study	Please indicate which datasets were linked and their origin. For example, if you received UI wage data, was it state UI data or data from the National Directory of New Hires?
Measures and data sources) available in the evaluation dataset	Please indicate the broad domains of measures and/or data sources. For example, MIS data, SNAP benefits data, etc.
Data dictionaries/Codebook /instruments available	Please indicate whether record layouts, codebooks, data dictionaries and/or survey instruments are available and whether they can be shared externally. Please provide links as appropriate.
Data sharing agreements for sample roster/link file	Are there agreements in place that specifically pertain to the sample membership file? The sample membership file contains the PII needed for matching which is typically (but not always) collected at the time of enrollment.

Feasibility
Have any extended follow-up analyses been done? Length of follow-up Data sources used	Was any additional follow-up done after the original evaluation contract? If so, please indicate the length of common follow-up and which data sources were included.
Did the study collect informed consent? If not, please explain why and how data collected for the evaluation was obtained.	Yes/no. And (if possible) please indicate if consent was obtained on paper or electronically. If no, please explain why consent was not collected and how data were obtained for the evaluation.
Informed consent description Are specific data sources mentioned? Is a length of data collection mentioned? Are there restrictions around who can access the data? Or when the data needs to be destroyed? Is there any other information we should know about regarding permissions to use the data?	Please explain the language used in the consent. How precise was the language regarding which data sources will be collected, the length of data collection, restrictions around who can access the data, and when the data needs to be destroyed? Please provide a copy of the consent form(s) if convenient.
Linking to administrative data sources Is it possible to link the evaluation dataset to other administrative data sources? If not, why not?	Please indicate whether you think it’s possible to link the evaluation dataset to other administrative data. If not, please indicate the reason you do not think it is possible (e.g. consent form restrictions, previous IRB rulings, lack of personally identifying information, etc.).

Study and findings
Study design	Please indicate the research design used. For example, randomized control trial, matching design, etc.
Study enrollment period	Please indicate the range of dates when participants entered the study.
Number of sites	Please verify the information that we prepopulated.
Site location(s)	Please verify the information that we prepopulated.
Provider name(s)	Please verify the information that we prepopulated.
Sample size Overall By research group	Please verify the information that we prepopulated.
Sample characteristics	Please characterize the general characteristics of the sample. For example, their age and other demographics that are relevant to the intervention.
Intervention type and key services tested	For example, sectoral training or job search assistance.
Key outcomes in original study (e.g. primary outcomes)	Please specify the key/primary outcomes. And how these were measured. For example: employment status as defined as employment status 2 quarters after program exit.
Strength of findings Size of impact Consistency Timing Outcomes	Please characterize the overall pattern of impacts on key/primary outcomes only. Please specify the direction, size, and statistical significance of the key outcomes. The size of the impact should be indicated in percentage or dollar terms, or as an effect size.
Treatment contrast Were there participation impacts? On what services?	Please indicate whether there were participation impacts and if so, what services had impacts. Would you characterize these impacts as “large” or “small?” Please focus on the key services that were tested in this study.
Control group embargo Length of embargo Ineligible services/programs	Please indicate the length of the embargo and what services were covered by the embargo.
Follow-up in original study Length of follow-up Data sources used	Please indicate the common follow-up period – in other words, the amount of follow-up that the latest entrants have. Please also indicate the data sources that were used in the impact analysis.