Instrument 1
Semi-Structured Staff Interview Informed Consent Script
NOTE: A copy of the consent script will be sent to the respondent in advance of the interview. The interviewer will confirm that they have received the consent script when the call begins.
MDRC, a non-profit social policy research organization—working with the Office of Planning, Research & Evaluation (OPRE) and the Administration for Children and Families at the Department of Health and Human Services—is conducting research for the Assessing Options to Evaluate Long-Term Outcomes (LTO) using Administrative Data project. The purpose of the research is to: (1) identify promising studies1 that could benefit from longer-term follow-up and (2) investigate what the current parameters are for, and the feasibility is of, linking evaluation and administrative datasets. We are conducting interviews with researchers that worked on some of the studies we identified to be good candidates2 for long-term follow-up, based on information you or a member of the [insert evaluation] research team have shared with the LTO project team about the study. Our goals are to learn about:
the research benefits that would be gained from additional follow-up;
the availability (and quality) of the data to be linked;
the parameters of the study’s data-related agreements;
the steps that may be required for accomplishing these linkages (e.g. obtaining any relevant permissions to link study and administrative data); and
the potential costs for pursuing and conducting long-term follow-up (where possible).
Your participation in this interview will help us understand the procedures and associated costs for pursuing longer term follow-up on [insert study].
Privacy
Please note that your participation in this interview is completely voluntary and will require a time commitment of approximately one hour. Your information will be kept private. Your name and position will not be connected to responses in any written materials without your express permission. We will ask to audio record our conversations for our data to accurately reflect what is said. Audio-recording is voluntary. Audio files will not be transcribed, and we will not share these recordings with anyone else outside of the LTO project team at MDRC. These recordings will be stored on a secure server and then deleted at the end of this project. If, at any time, you would like us to turn off the recorder, we can do that. If you would like to say something “off the record,” we will turn off the recorder and not take notes until you say we can resume.
Do you agree to participate?
Do you have any questions before we begin?
Thank you for agreeing to participate in this interview.
More information
If you would like more information about this project, you may contact Alexandra Pennington, LTO Project Director, at (212) 340-8847 or Alexandra.Pennington@mdrc.org and/or LTO@mdrc.org.
According to the Paperwork Reduction Act, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 6/30/2021.
***
Wave 1 - Semi-Structured Evaluation Lead Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Evaluation Lead
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
3. How long have you been in this position?
A.(b) Familiarity with [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Availability of Evaluation Personally Identifiable Information (PII) and Long-term Follow-up Plans
Now we’d like follow-up on some of the information you/your project team provided to us previously, in [insert date]. Specifically, we’d like to confirm some of the details regarding the availability of study participant PII as well as any long-term follow-up plans your [insert organization] currently has for [insert study].
B.(a) Following up on initial data collection request responses?
6. Does the PII for the study participants still exist?
Was it destroyed?
Was it returned? To whom?
7. What types of PII are available?
Which identifiers (e.g. SSN) are still available? Are they available for all study participants or only for some?
8. Which organization(s) maintain files with study participants PII?
Where and/or how are these files maintained (e.g. crosswalk)?
What types of data are captured on the file that includes the PII?
Is the file in an easily accessible format?
Does another organization have to approve any future use of the data?
If so, what organizations?
9. Are you currently planning or pursuing any additional follow-up?
(If additional follow-up planned) Would this follow-up entail using administrative data sources or other data sources?
What funding sources have you looked into? Have you secured funding?
Have you already identified any administrative data sources?
What is the timeline for the follow-up project?
If no – is there a reason you haven’t pursued additional follow-up? [Probe for reasons, like not allowed in consent form, etc.]
C. Policy Relevance and Need for Evidence
We’d like to discuss the evaluation’s findings and policy relevance.
C.(a) Prior Research and Policy Relevance
10. What, in your view, were the most important findings from the evaluation?
Why? [probe for rationale]
11. How, if at all, did the evaluation’s findings inform policy or program practice?
Can you give us examples?
To what extent is the study informing policy practice today?
C.(b) Conceiving New Long-term Follow-up Questions
12. What research questions from the original [insert evaluation], if any, would long-term follow-up help to address?
What would the value be of additional evidence or long-term follow-up? Would it look something like [offer study-contextualized examples, i.e., do earnings increase?]
13. Are there additional outcomes, measures, or research questions beyond those that were included in the original evaluation that might be interesting to include in new research or long-term follow-up?
What are they?
Why were they excluded from the original research?
If long-term follow-up were to be done, what do you expect might be found? Was the program/model designed to affect outcomes in the long-term?
Study-specific probes
14. If conducting long-term follow-up is beneficial, over or for what period of time should follow-up be done?
Why?
15. If conducting long-term follow-up is beneficial, what data sources might be useful or feasible to access?
Why?
D. Identifying Wave 2 Respondents
In the cover memo we sent you, we highlighted that we will be doing a second wave of data collection focusing on other aspects of determining feasibility for long-term follow-up. We’d like to ask you some questions about who we could possibly talk to gather more specific information.
D.(a) Data Availability
16. We'd like to talk to someone about the availability of the study data, specifically the quality and completeness of the PII. This could be someone who was responsible for collecting, processing, or maintaining the data, possibly the study’s data manager. Who would you recommend?
Can they also provide us with study data documents, such as codebooks, data dictionaries, etc.?
If not, are these materials accessible for you or your organization’s future reference? We may have follow-up questions about these materials.
D.(b) Study Legal Framework
17. We’d like to better understand what data sharing agreements, memorandums of understanding (MOUs) and other agreements were or are in place to support the original evaluation research activities and/or any potential future activities. Is there someone we could talk to who could describe the legal landscape and confirm information from these agreements to help inform data availability and feasibility for conducting long-term follow-up?
Which organizations are party to these agreements?
Could these be shared with us?
D.(c) Original IRB Review
18. Did your organization’s IRB review the original evaluation’s research activities?
Were there other IRBs providing secondary review or did other IRBs code to the original IRB?
How was it determined that multiple IRBs would need to be involved and which IRB would lead the review?
We’d like to talk to someone about the ethical considerations of conducting long-term follow-up for this study. This could be one of your IRB representatives or someone from the evaluation team who was closely involved in the process for obtaining IRB approval for the study. Is there anyone you recommend?
Is there any documentation available from the original IRB application and feedback? Could this documentation be shared or is there someone who can address questions about its contents?
D.(d) Identifying Potential Long-term Follow-up Data Sources
19. We'd like to talk to someone about potential administrative data sources that could be leveraged for long-term follow-up for this study. This could be administrators for data sources linked in the original evaluation or other data sources that might be feasible to access given the legal and ethical parameters of the original study. Is there anyone you can recommend?
Could this person speak to the steps and associated costs for gaining access to specific data sets, as well as the privacy and security standards in place for accessing these data?
D.(e) Study Oversight
20. We’d like to talk to someone who oversaw the original study to better understand their role and interests in potential long-term up study efforts. This could be the original primary study Funder and/or project officer. Is there anyone you can recommend?
If not, would your organization need to connect with this organization or agency in the event that ACF/HHS were to provide funding to pursue long-term follow-up for this study? Why or why not?
Could you describe what steps might be involved to engage this stakeholder?
D.(f) Other Stakeholders for Future Follow-up
21. Are there any other individuals, organizations, or agencies we should speak to (or that you would like to speak to) about this study potential for long-term follow-up? This might include: other researchers, practitioners (whose work is directly impacted or connected by this research), study participants, and/or organizations and agencies with oversight over human subjects research. Is there anyone you can recommend?
What role did these stakeholders have in the original study?
What role might these stakeholders have in potential follow-up efforts?
E. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study(e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
22. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
23. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. We will send a follow-up email to you and may ask for the contact information of the people you referenced earlier in the interview. Do you have any questions for me?
Wave 2 - Semi-Structured Evaluation Data Manager Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Evaluation Data Manager
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
3. How long have you been in this position?
A.(b) Relationship and Familiarity to [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Availability of Evaluation Personally Identifiable Information (PII), Baseline Data, and Outcome Data
Now we’d like to confirm the availability of PII, baseline data, and outcome data for study participants.
B.(a) Information on Participant PII Data
6. Previously, we’ve asked the PI for [insert evaluation] about the existence of PII for the study participants. S/he let us know that this information is still available. To reconfirm with you, is this still correct? Or has something changed?
Was it destroyed? If so, when?
Was it returned? To whom? If so, when?
7. Can you help us reconfirm the types of PII that are available?
Which identifiers (e.g. SSN) are still available?
8. Which organization(s) maintain files with study participants PII?
Where/how are these files maintained (e.g. cross-walk)?
What types of data are captured on the file that includes the PII?
Where is the file located? Is the file in an easily accessible format?
Does another organization have to approve any future use of the data?
If so, which organization(s)?
9. Are PII available for all study participants?
Does PII availability/quality vary by:
type (e.g. contact information may be less complete)?
type of study participant (e.g. more SSNs for experimental group)?
some other dimension (e.g. people randomly assigned (RA) after X date had better PII)?
If PII availability/quality varies, please explain why?
Was any documentation that summarizes the quality of the PII maintained?
If so, can this be shared?
Are PII available for the children or other family members of study participants? If so, for whom and what types of PII are available?
B.(b) Information on Participant Baseline Data
10. Did you collect data on participants at study entry?
Do these data still exist?
Was it destroyed/returned? If so, when?
11. Please describe the types of data collected at study entry (e.g. demographics, employment status, income and sources, etc.)
Is there an instrument or codebook available?
12. Which organization(s) maintain files with study participant baseline data?
Where and how are these files maintained (e.g. on same file as PII, on PUF/RUF, analysis file, etc.)?
Is the file in an easily accessible format?
Does another organization have to approve any future use of the data?
If so, which organization(s)?
13. Are the baseline data available for all study participants?
Does availability/quality vary by:
type (e.g. some questions may be less complete)?
type of study participants (e.g. data was collected differently for experimental and control groups)?
some other dimension (e.g. people randomly assigned after X date)?
If PII availability/quality varies, please explain why? Was any documentation that summarizes the quality of the baseline maintained?
If so, can this be shared?
B.(c) Information on Participant Outcome Data
14. Does the outcome data still exist?
If not, why? Was it destroyed/returned?
15. Please describe/confirm the types and sources of study participant outcome data (e.g. follow-up survey, administrative data, etc.)
Are instruments and/or codebooks available?
16. Which organization(s) maintain files with study participant outcome data?
Where/how are these files maintained (e.g. on same file as PII, on PUF/RUF, analysis file, etc.)?
Is the file in an easily accessible format?
Does another organization have to approve any future use of the data?
If so, which organization(s)?
17. Are the outcome data available for all study participants?
Does availability/quality vary by type?
by types of study participants?
by some other dimension (e.g. people RA'd after X date)?
If so, why?
Was any documentation that summarizes the quality of the outcome data maintained?
If so, can this be shared?
C. Ethical Parameters of Long-Term Follow-Up
We’d now like to learn more about the informed consent process and the consent language for [insert evaluation].
C.(a) Informed Consent of the Original Study
18. Did the study collect informed consent?
If yes, do you have a copy of the informed consent form(s)?
Can you share it with us?
If not collected, [confirm exemption due to common rule or waiver of consent as reason].
Did study participants know they were a part of a research study?
Did study participants have the opportunity to withdraw from the research? If so, how?
19. Did the informed consent form indicate how long study participant's information would be collected or accessed or maintained?
If so, for how long?
Was a specific date provided for the end of the study, or was there any reference(s) to the end of the study?
How much additional follow-up (FUP) - beyond the FUP that's already been done - is covered by the informed consent?
Any reference(s) to data destruction plans at the end of the study?
Any reference(s) to using the data for secondary or long-term follow-up research purposes?
20. What data sources or types of data to be collected were mentioned in the informed consent form? And from which sources would these data be obtained?
Does the form specify specific agencies to provide the data?
Does the form specify person-level data collection for study participants only?
21. Which organization(s) or agencies (if any) are specified in the consent form as members of the research team collecting/accessing/using the study participant data?
C.(b) Additional Consent(s) for the Original Study
22. Did any of the administrative data sources used in the original study require additional informed consent forms?
Can you share those forms with us? Or the specific language around specific topics?
Did any of the language specify how these data would be used, maintained, and/or for how long?
Was a specific date provided for the end of the study, or was there any reference(s) to the end of the study?
23. Did the original study administer any surveys? If so, did these include any language around consent, privacy, or confidentiality?
Can you share those forms/surveys with us? Or the specific language around these topics?
Did any of the language specify how these data would be used, maintained, and/or for how long?
Was a specific date provided for the end of the study, or was there any reference(s) to the end of the study?
C.(c) Informed Consent for Long-Term Follow-Up
24. Do you think it would be possible and/or feasible to re-consent participants in the event that long-term follow-up for the study were pursued? Why or why not?
25. Are there any current plans to re-consent participants? For what proportion of the original study participants will re-consent be attempted?
What length of time does new consent cover?
Which data sources/types and for whom?
What organizations/agencies are specified as members of the research team?
26. Does the new consent form mention the data sources or types recommended by the PI to answer long-term follow-up (LTFU)questions?
D. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study (e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
27. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
28. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. Do you have any questions for me?
Wave 2 - Semi-Structured Evaluation Legal Representative Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Evaluation Legal Representative
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
Do you have any regular responsibilities related to [insert evaluation]? If so, can you please describe them?
3. How long have you been in this position?
A.(b) Relationship and Familiarity to [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Prime Study Contract
6. Now we’d like to ask you some questions about the prime study contract that was put in place for the evaluation.
B.(a) Permission to use PII or Baseline Data for Long-Term Follow-Up
7. Is there a contract that describes all the data collection requirements to be fulfilled for the study?
Does this contract cover the data that is collected from participants at study entry (e.g. PII and baseline data)?
8. Which organizations or agencies are listed as parties to the agreements?
9. Can you share any language that speaks to the permissions to use or share the data covered in this agreement for the purposes of long-term follow-up [provide data-sharing examples, as needed]?
If no such language exists, are there any reasons you are aware of that may preclude the use or sharing of this information for the purposes of long-term follow-up research?
10. What time period is covered in this agreement?
When is the end date of the agreement?
11. Can you share any language regarding what happens to the data covered in this agreement at the end of the contract or agreement (e.g. destruction, retention, return, public-use/restricted-use file creation, etc.)?
What is the length of the retention period?
By when must the data be destroyed or returned?
What types of data were included in the restricted-use file or public-use file?
12. If a contract renewal were needed to facilitate long-term follow-up, can you describe any apparent factors that may help or hinder this process?
Has there been recent engagements with the other party for this or another study? Have these engagements been successful?
Have there been any recent changes in data use and sharing procedures and/or policies?
C. Data Sharing Agreements for Study Outcome Data
We’d like to ask some questions about the data sharing agreements that were established to obtain study participants’ outcome data
C.(a) Permission to use Outcome Data for Long-Term Follow-Up
13. Are there contracts in place that cover the acquisition and use of study participants’ outcome data?
What type(s) of study data are covered in this agreement?
If there are multiple study sites, are there data sharing agreements for each site?
14. Which organizations or agencies are listed as parties to the agreement?
15. Can you share any language that speaks to permissions to use or share the data covered in this agreement for the purpose of long-term follow-up [provide data-sharing examples, as needed]?
If no such language exists, are there any reasons why this information could not be used or shared for the purposes of long-term follow-up?
16. What time period is covered in this agreement?
When is the end date of the agreement?
17. Can you share any language regarding what happens to the data covered in this agreement at the end of the contract or agreement (destruction, retention, return, public-use files, restricted-use files, etc.)?
What is the length of the retention period?
By when must data be destroyed or returned?
What data must be included in the restricted-use file or the public-use file?
18. If a contract renewal were needed to facilitate long-term follow-up, can you describe any apparent factors that may help or hinder this process?
Has there been recent engagements with the other party for this or another study? Have they been successful?
D. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study (e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
19. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
20. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. Do you have any questions for me?
Wave 2 - Semi-Structured IRB Representative Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Evaluation Institutional Review Board (IRB) Representative
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
Do you have any regular responsibilities related to [insert evaluation]? If so, can you please describe them?
3. How long have you been in this position?
A.(b) Relationship and Familiarity to [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Ethical Parameters of Long-Term Follow-Up
Now we’d like to talk about the ethical parameters to be considered when conducting long-term follow-up for this evaluation.
B.(a) Long-Term Follow-Up Exemption
Some of the studies under consideration for the long-term follow-up did not collect consent from participants prior to study entry because the study was exempt from the Common Rule as a “research and demonstration project” conducted “by or subject to the approval of department or agency heads, and which are designed to study, evaluate, or otherwise examine: (i) Public benefit or service programs; (ii) procedures for obtaining benefits or services under those programs; (iii) possible changes in or alternatives to those programs or procedures; or (iv) possible changes in methods or levels of payment for benefits or services under those programs”.
6. Would the long-term follow-up for these studies be exempt? If not, please describe why this may not necessarily be the case.
Are you aware of any recent scenario(s) where a long-term follow-up study was approved when informed consent was not collected for this exemption reason?
Please describe how, if at all, the original exemption factored into the IRB’s deliberations.
Was a waiver of consent requested?
B.(b) Contract Limitations
Some of the studies under consideration for long-term follow-up do not have contracts that include language about how long study participants PII and/or baseline data can be retained.
7. Is this a factor that would preclude IRB approval of a long-term follow-up study proposal? If not, please describe why this may not necessarily be the case.
Are you aware of any recent scenario(s) where a long-term follow-up study was approved when the contract did not have explicit language about the retention period of study participant PII and/or baseline data?
Please describe how, if at all, this factored into the IRB’s deliberations.
B.(c) Contract Form Limitations
Some of the studies under consideration for long-term follow-up have consent forms that do not mention the intended long-term follow-up data source and/or the follow-up period and/or the follow-up period is not mentioned.
8. Are these factors that would preclude IRB approval of a long-term follow-up study proposal? If not, please describe why this may not necessarily be the case.
Are you aware of any recent scenario(s) where a long-term follow-up study was approved when the consent form from the original study does not mention these elements?
Please describe how, if at all, this factored into the IRB's deliberations.
Was a waiver of consent requested?
B.(d) Permissions to Share PII with Administrative Data Provider
Some of the studies under consideration for long-term follow-up do not have contracts that include language about permissions to share study participant data with an administrative data provider for the purposes of long-term follow-up.
9. Is this a factor that would preclude IRB approval of a long-term follow-up study proposal? If not, please describe why this may not necessarily be the case.
Are you aware of any recent scenario(s) where a long-term follow-up study was approved when the contract(s) did not include these data-sharing permissions?
Please describe how, if at all, this factored into the IRB's deliberations.
C. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study(e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
10. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
11. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. Do you have any questions for me?
Wave 2 - Semi-Structured Administrative Data Source Provider Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Administrative Data Source Provider for [Insert Data Source] and [Evaluation]
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
Do you have any regular responsibilities related to [insert evaluation]? If so, can you please describe them?
3. How long have you been in this position?
A.(b) Relationship and Familiarity to [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Long-Term Administrative Data Source
Now we’d like to talk about the [insert administrative dataset name] available through your agency/organization—including confirming any detail we have found publicly—and its potential as a long-term follow-up data source for [insert evaluation(s)].
B.(a) Process and Timeline for Gaining Access to the Data Source
6. Please describe the process for researchers to request access to study participant administrative data.
Do researchers need to submit a formal research request for approval?
Who must provide approval for access to specific datasets?
Are copies of study participant consent and/or evidence of IRB approval required?
What is the process for establishing a data sharing agreement, and how long does this process typically take?
Are there restrictions on who can access the data?
Where and how do researchers access the data once approved?
B.(b) Data Coverage and Quality of Data Source
7. Please describe what types of data are available and what population(s) are covered by this source.
What personal identifiers are available?
Is there any geographic information available?
What time period is covered by these data?
How often are these data collected and/or aggregated?
How often are the data updated?
How long are these data retained?
B.(c) Matching Procedures of Data Source
8. Please describe the process used to retrieve administrative data records for a research study sample (i.e. matching procedures).
What personal identifiers are needed to retrieve these records?
How do researchers typically submit these identifiers?
How much time does this process take and who is involved?
Are there any restrictions on what information is returned on the records (e.g. are person identifiers returned)?
Are there any other restrictions?
B.(d) Costs of Accessing and Matching to Data Source
9. Please describe the costs and/or cost structure for accessing these data.
Do these costs include those associated with matching the data, using a facility to analyze the data, and/or disclosing analysis results? Do these costs include any other work, and if so, please describe?
B.(e) Documentation for Data Source
10. Is there any publicly available documentation on the data that is available for research use, the process for gaining access to the data, the matching procedures, and/or costs?
C. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study(e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
10. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
11. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. Do you have any questions for me?
Wave 2 - Semi-Structured Funder Interview Protocol
Informants: First Name, Last Name, Affiliation
Informant Type: Evaluation Funder or Study Oversight Representative
Interview Length: Up to 60 minutes
[Questions and probes will be framed around respondent’s answers, and some questions and/or probes may not be asked]
A. Respondent Background
We’d like to start by asking you a few questions about yourself and your work at [organization].
A.(a) Current Position
1. What is your current employer and position?
2. What are the primary responsibilities of your role?
3. How long have you been in this position?
A.(b) Relationship and Familiarity to [insert evaluation]
4. How did you first become involved in the [insert evaluation], and how long were you involved in the research activities?
When was this?
5. What parts of [insert evaluation] were you substantively involved in?
Probe for work on evaluation that was more or less intensive based on research activity/phase (e.g. analysis design/synthesis, data acquisition, implementation research, etc.)
If evaluation is ongoing - Do you currently have any regular responsibilities related to [insert evaluation]? If so, please describe them.
B. Prior Research and Policy Relevance
We’d like to discuss the evaluation’s findings and policy relevance.
6. What, in your view, were the most important findings from the evaluation?
Why? [probe for rationale]
7. How, if at all, did the evaluation’s findings inform policy or program practice?
Can you give us examples?
To what extent is the study informing policy practice today?
C. Long-term Follow-up Plans
Now we’d like to learn about any long-term follow-up plans your [insert organization] currently has for [insert study].
8. Are you currently planning or pursuing any additional follow-up?
If yes, funding sources, data sources, timeline?
If no, any concerns about HHS/ACF/OPRE potentially funding this long-term follow-up effort?
What role would [funder] want/need to have in this effort?
Are there any necessary clearance(s) [funder] would have to provide to clear the path for this work?
If so, please describe.
D. Conceiving New Long-term Follow-up Questions
We’d like to receive your input on any new insights that could be gleaned from [insert evaluation].
9. What research questions from the original [insert evaluation], if any, would long-term follow-up help to address?
What would the value be of additional evidence or long-term follow-up? Would it look something like [offer study-contextualized examples, i.e., do earnings increase?]
10. Are there additional outcomes, measures, or research questions beyond those that were included in the original evaluation that might be interesting to include in new research or long-term follow-up?
What are they?
Why were they excluded from the original research?
If long-term follow-up were to be done, what do you expect might be found? Was the program/model designed to affect outcomes in the long-term?
Study specific probes (e.g. connections between promising sites in ERA and H2E study)
11. If conducting long-term follow-up is beneficial, over or for what period of time should follow-up be done?
Why?
12. If conducting long-term follow-up is beneficial, what data sources might be useful or feasible to access?
Why?
E. Closing Questions / Removing Barriers
Over the course of the project, we are learning that there is some uncertainty about the steps required to conduct long-term follow-up studies, since there is considerable variation across several dimensions per study(e.g. contract language, consent form language, interventions, target populations, etc.). Sometimes, additional procedures (e.g. negotiating new agreements) do not emerge until long-term follow-up pursuits begin. We’d like to ask you a couple of closing questions about the conditions that may help/hinder the feasibility of these sorts of efforts in the future.
13. Are there any potential barriers or questions you see arising in this work as it relates to your evaluation about the topics covered in this interview?
How do you see COVID-19 impacting future long-term follow-up research efforts among the topics discussed today?
14. What would help make conducting long-term studies follow-up easier for you and/or your organization? What kind of processes should ACF be thinking about to foster this kind of work or make it easier to do?
OK. Those are all the questions I have for you. Thank you. Do you have any questions for me?
1 This project is focused on identifying employment and child/youth development-related program evaluations.
2 The project team identified promising studies based on set of criteria agreed upon by ACF/OPRE (e.g. intervention type, study completion year, funder, strength of the evidence, study design). We then developed and fielded a data collection template to study teams to gather more in-depth information to better understand the feasibility for conducting long-term follow-up. Through these efforts, the “good candidates” for long-term follow-up were selected based primarily on evidence that (1) the PII needed for matching and (2) the key outcome data from an administrative data source are available.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Deondre' Jones |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |