Feedback on ECE-RISE Pilot Project

Interview Instrument 3. Interview for Applicants Selected Who Did Not Participate

Thank you for agreeing to speak with us today. My name is [NAME], and I’m joined by my

colleague, [NAME]. We’re from the Urban Institute, a nonprofit social and economic policy

research organization in Washington, DC.

1. (Who is leading the research and funding it) In partnership with Mathematica, we are leading the Child Care Evaluation and Capacity Building Center under a federal contract with the Administration for Children and Families in the U.S. Department of Health and Human Services.

2. As part of the larger project, we are conducting a process evaluation to objectively document the benefits and challenges of the ECE-RISE capacity building pilot. The results of the process evaluation will provide ACF information that can improve future capacity building efforts, enabling CCDF Lead Agencies to contribute more to ACF evidence-building efforts, and improve their ability to seek evidence as they make child care policy and operational decisions.

3. (Requirements) Over the next 45 minutes, we will be asking you a series of questions designed to gather in-depth information regarding the application process, implementation/participation, and outcomes of ECE-RISE. You may not know the answer to every question, and that is fine. If there are any questions that you don’t feel knowledgeable about or don’t feel comfortable answering, just let us know and we will move on. You can also let us know names and affiliations of other people who may be more informed on those topics, and we can follow up with them.

4. (Voluntary) I’ll remind you that this interview is voluntary. There will be no consequences if you decline or stop the interview. If you need to take a break at any time, please let us know.

5. (Consent to record) We’ll take notes during our discussion, but if it’s okay with you, we would also like to record this interview to help fill in our notes. If you would like me to stop recording at any time during the interview, please let me know. We will delete the recording once our analysis is complete.

6. (What we will do with the data) The information we gather during your interview will be paired with what we hear in other interviews and with other information from a document review. Ultimately, we will create a memo that will be shared only with our federal project officers at the Office of Planning, Research, and Evaluation, which is the research arm of the Administration for Children and Families.

7. (Privacy) We ask that you participate in a private setting away from earshot and viewing by unauthorized persons to include family members and we want you to understand that given the technical limitations of zoom and similar internet platforms, we cannot guarantee the privacy of what might be said. [If group interview: While we will maintain the privacy of what is said, we cannot control what other participants may say outside of the interview]. We will not identify you by name in our products. If we quote you in our study products or describe something you shared, we will never use your name will make attempts attribute the quote in any way that someone could not identify you. However, because of your role on the ECE-RISE project and regular interactions with some ACF staff, they may figure out who you are from the remarks that we report. Therefore, we cannot guarantee complete privacy of your participation or the views you express.

8. (Risks and benefits) If you share something about a challenge you experienced that is sensitive, there could be a risk of reputational harm because people might be able to identify you due to your role on the ECE-RISE project. Sharing challenges helps other people learn, but you have to decide how comfortable you are with what you share. You can ask us not to include some parts of the information in the memo that will be delivered to OPRE. Otherwise, there are minimal risks to participation in this interview. While there are no direct benefits to you, you may benefit from knowing your experiences with ECE-RISE could be used to inform and improve future ACF capacity building efforts.

Do you have any questions for me about the study?

Do we have your consent to proceed with our interview?

Do we have your permission to record?

[If the interviewee says yes] Thank you.  If you are ready, I will start recording now.  

[If the interviewee say no, research assistant will be prepared to take close-to-verbatim notes.] 

Background

I’d like to begin by getting to know a little about your work at MDHS.

1. Can you please describe your role at MDHS and how long you’ve been in your position?

Application Process

Next, let’s talk about the ECE-RISE application process, including how your agency decided to apply, what you had to do to apply, and your perspective of the process.

2. How did you hear about ECE-RISE?

3. Why was your agency interested in participating?

4. How did your agency decide to apply?

   1. Who was involved in the decision?

   2. What information was used to help make the decision?

   3. What was going on at the state or agency level that factored into your decision to apply?

5. Who from your agency was involved in the application process?

   1. [Probe if not mentioned] What was your role as part of the application process?

[Note to interviewer: skip remaining items in this section if respondent was not part of the application process]

6. We know it was a while ago, but please walk us through what you remember about the application process. As a reminder, you completed a form and then you were selected for a follow up interview.

   1. At the time, did it seem like an easy or hard process?

   2. Did someone outside of the team need to approve your submission? If so, what role were they in?

7. What was most challenging about the application process? For example, understanding the application requirements, finding the required information, the time provided to submit, and the like.

   1. What made that challenging?

   2. What assistance, if any, did the ECE-RISE program provide?

8. What suggestions do you have about improving the information available about how to apply or the application process?

Expectations of Implementation of and Participation in the Project

Our understanding is that your agency was accepted to participate in ECE-RISE; however, your agency withdrew from the program prior to holding a kick-off meeting with the ECE-RISE program team. We’d like to discuss your agency’s decision to withdraw and the circumstances that led to it.

9. Why did MDHS choose to withdraw from ECE-RISE?

   1. Who was involved in that decision?

   2. What changed between when you applied for ECE-RISE and the offer of ECE-RISE participation?

   3. [Probe if not mentioned] Did the level of support available from the program play a role in this decision? How so?

   4. [Probe if not mentioned] Did the expected level of effort required of MDHS to participate in ECE-RISE play a role in this decision? How so?

   5. [Probe on information found during document review related to MDHS’s withdrawal reasons]

10. What was going on at the state or agency level that factored into the decision to withdraw?

    1. [Probe on staff turnover, funding changes, organizational changes, or other findings from document review]

11. What needed to be different in order for you to continue with the program?

    1. What, if anything, could the program have done to allow MDHS to continue with the program?

       1. [Probe if not mentioned: deferring start up or participation to a later date, if money had been associated with the support, or other factors found during document review]

12. Even though your agency did not participate in ECE-RISE, were there any benefits to going through the application process? If so, what were they?

    1. [Probe if not mentioned] Did it identify agency strengths and areas for growth? Have you implemented any changes internally based on what you learned? Have you implemented the research project described in your application?

13. If you were offered the opportunity to apply again, would you be likely to do so? Why or why not?

Closing

Those were all the questions we had for our discussion today.

14. Is there anything we didn’t ask that you’d like to tell us about your experience with the ECE-RISE program?

Thank you for taking the time to share your thoughts and experiences with us.

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: The purpose of this information collection is to help the government understand the benefits and challenges of a research and evaluation capacity building pilot and will be used to improve future capacity-building projects. Public reporting burden for this collection of information is estimated to average 45 minutes per respondent, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0401 and the expiration date is 06/30/2024. If you have any comments on this collection of information, please contact Teresa Derrick-Mills at tderrick-mills@urban.org.

Interview Instrument 3. Interview for Applicants Selected Who Did Not Participate