Family Self-Sufficiency Demonstration Development Grants Evaluation Support: Data Collection for Final Report

SEEDS Discussion Guide to Assess Grant Recipients’ Final Research and Evaluation Capacity

Discussion Guide to Assess Grant Recipients’ Final Research and Evaluation Capacity

Introduction and consent

Thank you so much for meeting with us today. My name is [introduce self] and I am a member of the Supporting Evaluation Efforts for Demonstrations in Self-sufficiency (SEEDS) team. As you know, my colleagues [NAMES] provided research and evaluation support to you as one of the Family Self-Sufficiency Development Demonstration (FSSDD) grant recipients, with funding from the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services. The intention of the SEEDS project was to build grant recipients’ research and evaluation capacity and help them generate evidence related to their programs, which aim to improve the lives of children and their families.

The purpose of the discussion today is to gather feedback from you about your experience with the FSSDD grant and SEEDS support. We also want to understand how your organization’s research and evaluation capacity may have changed as a result of the FSSDD grant and SEEDS support. We will use information gathered in this discussion to inform a report for OPRE, which will not be made public. SEEDS is seeking this information to improve future technical assistance and support to grant recipients. This is not a compliance exercise. Nothing you discuss will have any effect on your ability to receive funding from ACF, now or in the future.

We will keep your responses private. In the report we produce, we will blend your responses with responses of other grant recipients and not use any individual names, so no single finding or statement will be attributed to an individual person or grant recipient. Your responses will in no way affect your current or future grant opportunities. We anticipate this interview will take approximately 60 minutes.

I would like to record our conversation today so I don’t miss anything. Is it okay with you if I record the conversation? If you want me to turn the recorder off for any reason or at any time, just say so. The recording will assist with notetaking and only be accessed by evaluation support team members while it is stored on a secure drive at Mathematica. We’ll destroy the recording at the end of this project. [INTERVIEWER: TURN THE RECORDER ON]

Okay, I have now turned on the recorder. Now that I have the recorder on, I need to ask you again, is it okay if I record this conversation? [Interviewer: Get verbal consent to record after beginning to record.]

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN:

The purpose of this information collection is to provide evaluation support to innovative interventions serving individuals, children, and families facing challenges to economic independence to expand the evidence base. Public reporting burden for this collection of information is estimated to average 60 minutes per response. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0356 and the expiration date is 02/29/2024. If you have any comments on this collection of information, please contact Julia Lyskawa at JLyskawa@mathematica-mpr.com.

Warm-Up

Before we begin, for each person participating in this discussion, please tell us your name, title, and give us a sense of your role within your organization. What are your day-to-day responsibilities? How long have you been with your organization?

[For each person participating] What was your role or involvement with the FSSDD grant specifically?

Support SEEDS team provided to Grant Recipients 

First, I’d like to hear more about the specific activities and resources you accessed through SEEDS support during the grant period. For this discussion, “project” or “grant project” refers to the research and evaluation activities you proposed doing in your FSSDD applications and worked on with SEEDS support.

1. What SEEDS activities did your team participate in throughout this project? What SEEDS resources did your team utilize? What were your impressions of those activities and resources?

Interviewers: Potentially probe on individualized coaching, learning communities, training, site visits, Road Test Challenge, GRAM, templates/resources, individualized consults, etc.

2. [As applicable, if the GR mentions any personalized consults] What were your team’s experiences with any personalized consultations you received?

Interviewers: Probe as needed to ask for input on the structure (frequency, content) and usefulness (content, expertise of expert consultant) of the consultations.

3. [As applicable, if the GR mentions Road Test Challenge] What did your team learn from the Road Test Challenge? Has your team conducted other road tests since the Road Test Challenge? How confident does your team feel about conducting your own road tests in the future?

4. [As applicable, if the GR identifies a particular learning community] What made you/your team decide to opt into that learning community? What did you/your team take away from the experience?

5. To what extent has your team applied learnings from the SEEDS activities/supports to other projects or areas of your organization? Can you share an example?

6. What elements of SEEDS support were the most useful? Which elements were the least useful? Why?

Grant recipient goals, operations, and achievements

Next, we’d like to talk about whether and how your project goals evolved during the grant period and what you achieved through your project.

1. What was you/your team’s initial understanding of the FSSDD grant? What did you/your team think the purpose of the grant was?

2. How do you understand the purpose of the grant now, towards the end of the project? Is it different than you initially thought?

3. Based on discussions at the very beginning of the grant period, your project sought to [fill in goals from prior calls]. Did your team’s goals or focus for the FSSDD project change during the grant period? If so, how and why? [if applicable: Can you tell me more about your decision to change those goals and the factors that influenced your decision?]

4. What did your team do internally to prepare for the grant? Who was involved in those preparations?

5. What work did your team do internally (i.e., outside of SEEDS coaching calls) to make progress on your FSSDD project goals over time, during the grant period?

Interviewers: Probe on whether and how teams met internally between SEEDS coaching meetings, whether and how teams communicated internally about their FSSDD projects, etc.

6. What does your team see as your primary achievements from this grant?

Research and evaluation capacity

In the next set of questions, we want to discuss your organization’s capacity for research and evaluation, or “evidence capacity.” Evidence capacity means the knowledge, skills, behaviors, and resources that, together, make it possible for organizations to create and use evidence to make decisions. We’ll discuss if and how your evidence capacity has changed along five different dimensions over the course of the grant period. We’ll also touch on what factors helped you make progress or hindered your progress.

1. Leadership:

   1. To what extent do you/your organization’s leaders support research and evaluation activities? How do they show that support? Please speak to leaders’ level of support both before and after the grant.

   2. In what ways do you/your organization’s leaders use data or research to inform their decision making or continuous improvement? How has this changed over the course of the grant, if at all?

   3. What factors support you/your leaders in their efforts to use evidence in the course of their work? What factors hinder you/your leaders in efforts to use evidence?

2. Evidence-driven organizational culture:

   1. To what extent do you/your team see your organization as committed to the use of evidence? To what extent do staff at different levels see the value of evidence and understand how it is relevant to their work?

   2. How does the use of evidence show up in your organization’s culture? How has that changed, if at all, during this grant?

   3. What factors allow your organization to pursue an evidence-driven culture? What barriers keep your organization’s culture from being more evidence-driven?

3. Evidence infrastructure (tools, resources, routines, processes):

   1. What tools, resources, and data systems are in place to help your team collect, store, and analyze data? How has that changed, if at all, during this grant?

   2. To what extent does your organization use tools, resources, and data systems on an ongoing basis for continuous improvement? How has that changed, if at all, during this grant?

   3. What factors have allowed your organization to use those tools, resources, and data systems effectively? What may be preventing your organization from using those tools, resources, and data systems effectively?

4. Staff capabilities (knowledge, skills, abilities) to create and use evidence:

   1. To what extent do staff have the knowledge and skills to conduct or manage research and evaluation activities or to use data in their day-to-day practice? How has that changed, if at all, during this grant?

   2. How confident do staff feel about being able to carry out those activities on their own after the grant ends?

   3. [If working with an external evaluator] What role did your external evaluator play over the course of the grant?

   4. What supports are in place to help staff strengthen their research, data, and evaluation capabilities? What barriers prevent staff from further strengthening their research, data, and evaluation capabilities?

5. Engagement and communication in creating and using evidence:

   1. How does your organization collaborate with staff and external partners to seek their input into research and evaluation activities? How has that changed, if at all, during this grant?

   2. How does your organization disseminate data and evidence to staff and external partners? How has that changed, if at all, during this grant?

   3. What factors supported your/your team’s ability to collaborate and communicate with staff and external partners around research and evaluation? What hinders your/your team’s ability to collaborate with staff and external partners around research and evaluation?

6. What impact has this grant had on…

   1. Your organization?

   2. Your organization’s use of data, research, and/or evaluation?

   3. The way your organization provides services to children and families?

Perceptions, feedback, and lessons learned

The next set of questions focuses on your feedback and lessons learned from this grant experience.

1. How did your/your team’s expectations for this grant compare to what actually happened?

2. Think back to the beginning of the grant period. What were your/your team’s expectations for the individualized coaching support you were going to receive through SEEDS? How does this compare to the coaching support you actually received?

3. Is there anything you/your team hoped to learn or achieve through this grant but didn’t? If so, please explain.

4. Would your organization apply for another FSSDD grant, if there was an opportunity to do so? Why or why not?

Moving forward

To wrap up our conversation, we’d like to discuss your team’s plans for after the FSSDD grant ends.

1. Looking forward, what are your team’s future goals around research and evaluation? To what extent does your team feel equipped to achieve these goals?

2. What specific plans does your organization have to continue research and evaluation activities?

3. If you’ve been working with an external evaluator, will your organization continue to do so in the future? If not, how will your organization staff research and evaluation activities moving forward?

4. How have you/will you use what you’ve learned through this grant outside of this project?

DRAFT 12/15/23 4