Feedback on ECE-RISE Pilot Project

Interview Instrument 2. Interview for Participant Team

Thank you for agreeing to speak with us today. My name is [NAME], and I’m joined by my

colleague, [NAME]. We’re from the Urban Institute, a nonprofit social and economic policy

research organization in Washington, DC.

1. (Who is leading the research and funding it) In partnership with Mathematica, we are leading the Child Care Evaluation and Capacity Building Center under a federal contract with the Administration for Children and Families in the U.S. Department of Health and Human Services.

2. As part of the larger project, we are conducting a process evaluation to objectively document the benefits and challenges of the ECE-RISE capacity building pilot. The results of the process evaluation will provide ACF information that can improve future capacity building efforts, enabling CCDF Lead Agencies to contribute more to ACF evidence-building efforts, and improve their ability to seek evidence as they make child care policy and operational decisions.

3. (Requirements) Over the next hour and a half, we will be asking you a series of questions designed to gather in-depth information regarding the application process, implementation/participation, and outcomes of ECE-RISE. You may not know the answer to every question, and that is fine. If there are any questions that you don’t feel knowledgeable about or don’t feel comfortable answering, just let us know and we will move on. You can also let us know names and affiliations of other people who may be more informed on those topics, and we can follow up with them.

4. (Voluntary) I’ll remind you that this interview is voluntary. There will be no consequences if you decline or stop the interview. If you need to take a break at any time, please let us know.

5. (Consent to record) We’ll take notes during our discussion, but if it’s okay with you, we would also like to record this interview to help fill in our notes. If you would like me to stop recording at any time during the interview, please let me know. We will delete the recording once our analysis is complete.

6. (What we will do with the data) The information we gather during your interview will be paired with what we hear in other interviews and with other information from a document review. Ultimately, we will create a memo that will be shared only with our federal project officers at the Office of Planning, Research, and Evaluation, which is the research arm of the Administration for Children and Families.

7. (Privacy) We ask that you participate in a private setting away from earshot and viewing by unauthorized persons to include family members and we want you to understand that given the technical limitations of zoom and similar internet platforms, we cannot guarantee the privacy of what might be said. [If group interview: While we will maintain the privacy of what is said, we cannot control what other participants may say outside of the interview]. We will not identify you by name in our products. If we quote you in our study products or describe something you shared, we will never use your name will make attempts attribute the quote in any way that someone could not identify you. However, because of your role on the ECE-RISE project and regular interactions with some ACF staff, they may figure out who you are from the remarks that we report. Therefore, we cannot guarantee complete privacy of your participation or the views you express.

8. (Risks and benefits) If you share something about a challenge you experienced that is sensitive, there could be a risk of reputational harm because people might be able to identify you due to your role on the ECE-RISE project. Sharing challenges helps other people learn, but you have to decide how comfortable you are with what you share. You can ask us not to include some parts of the information in the memo that will be delivered to OPRE. Otherwise, there are minimal risks to participation in this interview. While there are no direct benefits to you, you may benefit from knowing your experiences with ECE-RISE could be used to inform and improve future ACF capacity building efforts.

Do you have any questions for me about the study?

Do we have your consent to proceed with our interview?

Do we have your permission to record?

[If the interviewee says yes] Thank you.  If you are ready, I will start recording now.  

[If the interviewee say no, research assistant will be prepared to take close-to-verbatim notes.] 

Background

I’d like to begin by getting to know a little about your work at TWC and with the ECE-RISE program.

1. Can you please describe your role at TWC and how long you’ve in in your position?

2. What was your primary role on the ECE-RISE project?

   1. [Probe if not mentioned] When did you join the project team? What were your primary tasks?

Application Process

Next, let’s talk about the application process, including how your agency decided to apply, what you had to do to apply, and your perspective of the process.

3. How did you hear about ECE-RISE?

4. How did your agency decide to apply?

   1. Who was involved in the decision?

   2. What information was used to help make the decision?

   3. What was going on at the state or agency level that factored into your decision to apply?

5. Who from your agency was involved in the application process?

   1. [Probe if not mentioned] What was your role as part of the application process?

[Note to interviewer: skip remaining items in this section if respondent was not part of the application process]

6. We know it was a while ago, but please walk us through what you remember about the application process. As a reminder, you completed a form and then you were selected for a follow up interview.

   1. At the time, did it seem like an easy or hard process?

   2. Did someone outside of the team need to approve your submission? If so, what role were they in?

7. What was most challenging about the application process? For example, understanding the application requirements, finding the required information, the time provided to submit, and the like.

   1. What made that challenging?

   2. What assistance, if any, did the Mathematica team provide?

8. Reflecting back on what you know now about what it takes to participate in ECE-RISE, what suggestions do you have about improving the information available about how to apply or the application process?

Process Outcomes

Next, I’d like to talk about TWC’s goals for participating in ECE-RISE and your perspective on the extent to which the agency achieved its goals.

9. What were TWC’s most important goals for participating in ECE-RISE?

   1. If you had other goals, what were they?

   2. [If not mentioned, probe on goals mentioned in TWC’s application]

10. From your perspective, in what ways did your agency team meet your most important goals for the project?

    1. [Probe based on answer to previous question]

11. What challenges or barriers made it difficult for your TWC team to accomplish your project goals?

    1. [Probe on any challenges found during document review]

    2. Did ECE-RISE help you strategize on this difficulty even if it could not be overcome?

       1. If yes, how did it help you and what was the limitation?

       2. If no, why didn’t or couldn’t ECE-RISE help?

12. What are the most important strengths TWC had internally that helped your team be successful in the project?

    1. From your perspective, did TWC have these strengths prior to participation in ECE-RISE?

    2. [Probe on any strengths found during document review]

13. In what ways has ECE-RISE increased TWC's research and evaluation capacity?

    1. What is your team able to do now that you could not do or felt less certain about doing when you applied?

       1. [If not mentioned, probe on data collection, data analysis, using data in decision-making, effectively sharing data, areas noted in application that they wanted to develop capacity in]

14. Sometimes when people participate in capacity-building projects, they identify more things that they don’t know how to do and want to learn about. Did anything like that come up for your team?

    1. If yes, what is it and to what extent was ECE-RISE able to help you address it?

15. Do you think TWC is now able to implement a similar research project on its own? What about guiding a contract with a firm to do the research for them?

    1. [if not] What barriers exist? Are the barriers the same for both situations (i.e., doing research itself versus contracting out)?

16. What additional skills or supports does TWC need in order to continue to grow its research and evaluation capacity?

17. Has TWC been able to use what you learned in your ECE-RISE project to answer an important policy or operational question or do you expect to be able to do so?

    1. If yes, please tell us more about that.

    2. If no, please let us know why not.

18. What do you think is the most important thing that ECE-RISE contributed to your team?

    1. To what extent do you see that contribution making a difference into the future?

    2. We know that many things can make it difficult to sustain the gains made during capacity building efforts, such as staff turnover, leadership changes, or policy changes. How sustainable do you think the gains your agency made during ECE-RISE are? Why do you say that?

Implementation of and Participation in the Project

Now, let’s talk about what participating in ECE-RISE was like. We are thinking of the project and your participation as having three primary stages for the purpose of this interview. First, the kick-off period which goes from formalizing the participation agreement up through the kick-off meeting. Second, the primary participation period. And third, the wind-down/wrap up period, or the point at which ECE-RISE project team members may have started to say that the project would be ending soon and begun planning for pilot activities to conclude. We understand that the primary participation period (the second stage) is likely to have some subparts that you want to tell us about a little bit more – we welcome you sharing any of your experiences. We’ll talk about each of the three stages in order, starting with the kick-off period.

19. Thinking back to the kick-off period which went from around [state time frame from document review], describe what you remember about the activities ECE-RISE did to get your team ready for the program.

    1. [Probes on learning about the LI² framework, developing a project workplan, or other key startup activities found during document review, if not mentioned]

    2. What do you remember as being most valuable about those activities?

    3. Is there anything you remember about those activities that felt like they took a lot of time but seemed less valuable?

20. About how often did you connect with the ECE-RISE project team members during the kick-off period and in what ways (email, telephone, video conference, in person)?

21. [If not mentioned] What do you remember about the kick-off meeting itself, which was held [state date from document review]? What occurred during the meeting and what was the result?

    1. Where was the meeting held and who attended?

       1. [If meeting was held virtually] Looking back, would you have preferred it to be held in person? Why do you say that?

    2. Let’s move forward and talk about the primary participation stage. This is the stage when your agency worked with the ECE-RISE team to build TWC’s capacity and implement an evaluation project, so from [state time frame from document review].

22. Please describe how the ECE-RISE team carried out the program with you.

    1. How often did you meet and with whom?

    2. What ECE-RISE activities did your team engage in and how often?

       1. [If not mentioned, probe on monthly coaching sessions, Road Map for Change, reflection tool, intensive support tracker, Learn site visit, data boot camps or hackathons, road tests, or other activities found during document review)

       2. What about these activities were helpful? What was unhelpful?

       3. Did your agency request changes to any activities that Mathematica proposed? If yes, what changes were requested, why, and what was the response?

23. What resources did ECE-RISE provide to support building your team’s research and evaluation capacity? We’re thinking of resources in terms of knowledge, tools, access to experts, trainings or similar non-financial supports.

    1. [If not mentioned, probe on expert consultations, how-to webinars, briefs or other publications, or other resources found during the document review]

    2. What about these resources was helpful? What was unhelpful?

    3. Did your agency request any specific resources?

       1. If yes, what were they? Were these resources provided? If any weren’t provided, what was the reason?

24. It is our understanding ECE-RISE used the Learn, Innovate, Improve (or “LI-squared”) framework. The LI² framework is a cycle of identifying the problem or challenge, developing specific strategies, and iteratively testing and refining those strategies. Are you familiar with the LI² framework? [If yes, as the prompts below, else go to next item]

    1. In what ways was this framework helpful? What was unhelpful about it?

    2. How prepared is your team to use this framework in the future once the ECE-RISE program ends?

25. Do you think the time and resources ECE-RISE provided your team during primary participation stage was too much, too little, or just right? Why do you say that?

26. How would you characterize the level of effort required by your team – high, medium, or low? Please describe what that means to you and about how many hours of staff time you think your team contributed to the project.

    1. Do you think the level of effort required of your agency was more or less than expected when you applied? Why do you say that?

27. As you may remember, in the original project design there was going to be a second agency that was supported and some of the activities were intended to conducted with the two agencies together.

    1. Is there anything about your experience that you think may have been enhanced if there had been another participating agency that you could interact with? If yes, please provide examples.

    2. Based on your experience, do you think having another agency to interact with could have detracted from your experience in any way? If yes, what concerns would you have?

Let’s turn to the final stage of ECE-RISE – the wind down or wrap up period. This is when ECE-RISE project team members started to signal that it was time to begin planning for pilot activities to conclude as the project would soon be ending. The wind down period may feel like it overlaps some with the primary participation state, but for our interview, we’ll say it occurred roughly from [state timeframe from document review].

[Note to interviewer: depending on timing, TWC may be in the wind down process during the interview. If so, change questions to present tense as applicable]

28. What activities did ECE-RISE do to prepare you for the end of the program?

    1. [Probe on wind down activities found during document review. Examples may be creating sustainability plans, look-back exercises, project close convening or celebration]

    2. In what ways were these activities helpful? What was unhelpful?

29. Were any tasks in your workplan incomplete when ECE-RISE ended?

    1. If so, what were they?

    2. What resources did ECE-RISE provide during the wind down stage to support your team in completing those tasks post-program?

Closing

Thank you for taking the time to share your thoughts and experiences with us. I have just a couple more questions before we end our time together today.

30. Overall, what was the best part of your team participating in ECE-RISE?

    1. What was the most challenging part?

31. Is there anything we didn’t that you’d like to tell us about your experience participating in the ECE-RISE program?

Thank you again for your time today.

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN: The purpose of this information collection is to help the government understand the benefits and challenges of a research and evaluation capacity building pilot and will be used to improve future capacity-building projects. Public reporting burden for this collection of information is estimated to average 90 minutes per respondent, including the time for reviewing instructions, gathering and maintaining the data needed, and reviewing the collection of information. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0401 and the expiration date is 06/30/2024. If you have any comments on this collection of information, please contact Teresa Derrick-Mills at tderrick-mills@urban.org.

Interview Instrument 2. Interview for Participant Team