Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data: Identifying Targets of Opportunity

Appendix A

Evaluation Template

Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data

Evaluation Template

This data collection effort is part of the Assessing Options to Evaluate Long-Term Outcomes Using Administrative Data study, which is funded by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF), U.S. Department of Health and Human Services.

This study is a new effort by OPRE to determine the feasibility of linking administrative data sets to program evaluation data. Data are being collected to assess the practical and legal feasibility for accomplishing the linkages, to assess potential costs, and to identify prior history of linkage, past findings, and current availability of relevant data and metadata. The information collected on each evaluation will be submitted to OPRE, along with a memo that summarizes the findings and identifies the most promising targets of opportunity.

Can you please review the information in the template, confirm or correct as needed, and fill in any missing information? We expect it will take you 1-2 hours to review and fill in this template, depending on the availability of information.

Please note that your participation in this data collection is voluntary and your information will be kept private to the extent permitted by law.

PRA Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 06/30/2021.

Evaluation: [Link to Evaluation here]

1. Lead evaluator

Please verify the information that we prepopulated.

2. Funder(s)

Please verify the information that we prepopulated.

3. Federal project officer

Please indicate the project officer for the original evaluation.

4. Contract status

Please indicate if the evaluation is ongoing or completed and the dates of the contract for the original evaluation.

Available datasets and documentation

5. Does data from the evaluation still exist?

Please indicate yes/no. If yes, please indicate if either of the following types of datasets are available:

• Evaluation dataset: includes measures used in the analysis, including research group, random assignment date, covariates, and/or outcome measures.

• Sample membership file: contains the personally identifying information (PII) for sample members; this information is typically (but not always) collected at the time of study enrollment.

6. Location

Please indicate the location (e.g. agency, organization, and/or the physical location) of the available dataset(s).

7. Was a public access or restricted use file created?

Please indicate yes/no. If yes, please indicate which type of file was created and where the file(s) are stored.

8. Is any personally identifying information (PII) available in the evaluation dataset, a restricted use/public access file, a sample membership file, and/or another type of file for a potential match to an administrative data source? If so, what types of PII are available? And in which file(s)?

For example, are SSN, name, date of birth, and/or contact information for sample members available? If any PII are available, please indicate which dataset(s) the PII is available in.

9. Administrative datasets linked to in original study

Please indicate which administrative datasets were linked to and their origin. For example, if you received UI wage data, was it state UI data or data from the National Directory of New Hires?

10. Measures and data sources available in the evaluation dataset, restricted use/public access file, and/or sample membership file

Please indicate the broad domains of measures and/or data sources that are currently available. For example, MIS data, SNAP benefits data, etc. Please also indicate which dataset(s) the measures are available in.

11. Data dictionaries/codebooks/instruments available for the evaluation dataset, the restricted use file, and/or a sample membership file

Please indicate whether record layouts, codebooks, data dictionaries, and/or survey instruments are available and whether they can be shared externally. Please provide a copy or the documents or links as appropriate.

12. Data sharing agreements for the evaluation dataset or sample membership file

Are there agreements in place that specifically pertain to the evaluation dataset or sample membership file? Please quote or summarize any data use/destruction requirements in the data sharing agreements.

Feasibility

13. Have any extended follow-up data collection or analyses been done by your or another organization?

Was any additional follow-up or data collection done after the original evaluation contract? If so, please indicate the length of common follow-up and which data sources were included.

14. Did the study collect informed consent?

Please indicate yes/no. And (if possible) please indicate if consent was obtained on paper or electronically and attach a copy of the consent document(s). If no, please explain why consent was not collected and how data were obtained for the evaluation.

15. Is there any information we should know regarding permissions to use the data?

Please share any other relevant information about permissions to use the data not captured in other fields (e.g. data provider must be contacted if the list of authorized users has changed).

16. Linking to administrative data sources

Assuming that legal and IRB issues are handled, please indicate whether you think it’s possible to link the PII for sample members in the evaluation to administrative data sources in the future. If not, please indicate the reason you do not think it is possible (e.g. consent form restrictions, previous IRB rulings, lack of personally identifying information, etc.).

Study and findings

17. Study design

Please indicate the research design used. For example, randomized controlled trial, matching design, etc.

18. Study enrollment period

Please indicate the range of dates when participants entered the study.

19. Number of sites

Please verify the information that we prepopulated.

20. Site location(s)

Please verify the information that we prepopulated.

21. Provider name(s)

Please verify the information that we prepopulated.

22. Sample size, overall and by research group

Please verify the information that we prepopulated.

23. Target population

Please characterize the target population for the evaluation. For example, age, education level, and other criteria.

24. Intervention type and key services tested

For example, sectoral training or job search assistance.

25. Key outcome in original study (e.g. primary outcomes)

Please specify the key/primary outcomes and how these were measured. For example: employment status as defined as employment status 2 quarters after program exit.

26. Strength of findings

Please characterize the overall pattern of impacts on key/primary outcomes only. Please specify the direction, timing, and statistical significance of the key outcomes.

27. Treatment contrast

Please indicate whether there were participation impacts and if so, what services had impacts. Would you characterize these impacts as “large” or “small?” Please focus on the key services that were tested in this study.

28. Control group embargo

Please indicate the length of the embargo and what services were covered by the embargo.

29. Follow-up in original study

Please indicate the common follow-up period – in other words, the amount of follow-up that the latest study entrants have. Please also indicate the data sources that were used in the impact analysis.