OFVPS Outcomes and Measurements Listening Session Draft Interview Guide_08092024

O MB #: 0970-0531

Expiration Date: 9/30/2025

MITRE-OFVPS Outcome Measurement Listening Session Guide

Agenda:

Topic	Time (~90 minutes total)
Welcome and Background	10 minutes
Questions: Section 1 – Frameworks, Tools, and Training/Technical Assistance (T/TA) Section 2 - Defining and Tracking Success Section 3 - Resource Center Recommendations	25 minutes 25 minutes 20 minutes
Wrap-Up/Next Steps	5 minutes

Overview of Listening Session Question Sections
Section 1 – Frameworks, Tools, and T/TA Provision	This section seeks to gather examples of existing outcomes measurement frameworks and tools T/TA providers use to support FVPSA DV/SA grantee programs.
Section 2 – Defining and Tracking Success	This section seeks to understand the training and technical assistance provided to grantees on how FVPSA DV/SA shelters and programs track their success and measure program outcomes.
Section 3 – Resource Center Recommendations	This section seeks to gather recommendations for what T/TA providers think grantees (local domestic violence programs, shelters, tribes, and culturally specific organizations) may need to be more effective in supporting outcome measurement for the services that they provide to survivors and their children.

Welcome and Background

MITRE:

Hi everyone! We will wait a couple more minutes for people to join.

Good morning/afternoon, and welcome everyone!

Thank you for joining us today for this listening session.

My name is _[name]____________, and I am a _[researcher or title]__ with the MITRE Corporation.

My team and I are facilitating this listening session on behalf of the ACF Office of Family Violence Prevention and Services (referred to in this conversation as OFVPS) team. MITRE operates the CMS Alliance to Modernize Healthcare—which is the Health Federally Funded Research and Development Center (Health FFRDC). The team members with me today are [names], who will be listening and taking notes to make sure that I don’t miss anything important.

During this session, I will ask questions about: 

• Existing outcomes measurement frameworks and tools you use as T/TA providers supporting grantee programs;

• How shelters, local programs, tribes, and culturally specific organizations track their outcomes and measure program outcomes;

• Recommendations you have as T/TA providers on how to improve tools for outcome measurement.

• And finally, grant recipient recommendations regarding resources they need to be more effective in their work

We will synthesize the insights you provide into a recommendations report for OFVPS. We truly appreciate your time as we look forward to gaining a deeper insight into outcomes measurement insights, needs and approaches for the grantee programs you serve.

Housekeeping

We’ll start our meeting with some housekeeping in a moment and then jump right into the discussion. Here you can see our full agenda of topics for the session [refer to slide].

Before we get into the discussion, we’d like to review some “rules of engagement” to ensure we allow for a dynamic discussion while also ensuring everyone is able to participate and provide meaningful input.

• To optimize your meeting experience, please log in to Zoom through the app rather than dialing in.

• We encourage you to have your video on.

• Please put your name, role, and organization in the chat.

• Please introduce yourself when speaking and answering questions.

• Feel free to use the Zoom “raise hand” and chat functions to participate. We will do our best to ensure that everyone who wants to contribute to the conversation has the opportunity to share.

• Please keep your microphones muted while you are not speaking to prevent background noise.

• To allow everyone to participate in the time we have together, we ask that you be mindful of making space for others to share their ideas and offer their feedback.

Disclaimer  

[MITRE Facilitator reads]:

The Office of Management and Budget (OMB) approved the questions for today’s listening session. The OMB # is 0970-0531 and the expiration date is 9/30/2025. Please note that participation in these listening sessions is completely voluntary and you may leave the call at any point. The estimated time for this session will be 90 minutes. Your participation in these listening sessions (or decision to not participate) will not affect your grant funding in any way. Personally identifiable information collected will be only “business card information”, i.e., respondents’ first and last names, email addresses, and institutional affiliations. MITRE plans to record today’s session for internal notetaking purposes only. Once we have verified our notes, we will destroy the recording. We will not attribute anything you share during this session to you or to your organization in the recommendations report we are preparing for OFVPS. Does anyone have any objections to MITRE recording this conversation? [If there are no objections, notetaker hits the record button. If there are any objections, the MITRE team will aim to capture more verbatim notes]. Any questions before we get started?

Discussion Section 1 – Frameworks, Tools, and T/TA Provision

• How does your organization provide outcome measurement T/TA to FVPSA grantees (states, territories, tribes, coalitions, domestic violence shelters, culturally specific programs)?

  • Probe: What is the balance between in-person and virtual support? What is the frequency of engagement with grantees?

  • Probe: How many grant recipients do you support? How well has it worked for you to have this number of grantee programs?

• In your role as T/TA providers, how do you use the logic models grantees are required to create for their funded programming?

  • Probe: How much do the grantees you support use the logic models in their reporting and monitoring of program results?

  • Probe: What gaps do you see in the current logic models?

  • Probe: What gaps do you see between the logic models grantees develop and their performance and use of these logic models?

  • Probe: What additional tools do your grantee programs use to track and measure program results?

• In your experience, how well does the current approach to training and TA allow you to provide the support that grantees need?

  • Probe: What is working well for you as a T/TA provider? (Prompt for utility of tools, updated outcome measurement guidance from OFVPS, and T/TA provider staff time, if respondents do not mention these.)

  • Probe: Which training methods or materials have you found most effective in supporting your grantee partners in measuring outcomes and the impact of their services?

  • Probe: What are your challenges as a T/TA provider? (Prompt for time constraints, capacity challenges, and gaps in tools if respondents do not mention these.)

    • Probe: How do these challenges differ by the type of grantee you support (states, territories, tribes, coalitions, domestic violence shelters, culturally specific programs and sexual assault programs)?

    • Probe: What types of (new) partnerships or models might help minimize these challenges?

Discussion Section 2 – Defining and Tracking Success

• What definitions of success do your grantee partners use for their funded programs?

  • Probe: How are these definitions established? By OFVPS? By FVPSA statute? By grantees? By you as a T/TA provider?

  • Probe: How well do these definitions tell the stories of what programs are doing and who they are reaching?

  • Probe: To what extent do these definitions focus on program outputs? (If respondents do not offer answers, prompt with number of survivors/dependents referred for or receiving services, and number of trainings provided.)

  • Probe: To what extent do these definitions focus on program outcomes? (If respondents do not offer answers, prompt with changes in knowledge, self-efficacy, or risk status.)

  • Probe: How much do outcome measures vary across funded programs?

• How are the needs and experiences of survivors captured in defining success and program outcomes?

  • Probe: How is survivor satisfaction with programs assessed? (Prompt for surveys, focus groups, assessments, if respondents don’t mention these

  • Probe: How are program outcomes tailored to address the specific needs of the populations grantees serve?

• How are outcomes measured and reported across grantee programs?

  • Probe: How is outcome measurement information tracked by grantee programs and who is responsible for tracking/reviewing?

  • Probe: What challenges do you (and your grantee partners) encounter with measuring outcomes?

  • Probe: To what extent have you been able to suggest or implement improvements to outcome reporting based on the challenges you noted?

Discussion Section 3 – Resource Center Recommendations

• What additional training, tools, or resources do states, territories, tribes, coalitions, culturally specific organizations, domestic violence shelters, and sexual assault programs need to better support outcome measurement?

  • Probe: What efforts are already underway to support improved outcome measurement?

  • Probe: What are the barriers to offering the resources you recommend?

• What recommendations, if any, would you like to share for measuring outcomes for FVPSA -funded SA/DV programs?

  • Probe: Which elements of the current reporting tools would you like to keep? Which would you like to discard?

  • Probe: Which of these would you like to keep but with major revisions?

  • Probe: What are your favorite outcomes or models that are currently being used or that you wish were being used within the context of SA/DV grant-funded programs?

  • Probe: If you had a magic wand, what changes would you want to see in how these programs measure and report their outcomes?

• If you had a magic wand, what changes would you want to see in how these programs measure and report their outcomes?

Wrap Up and Next Steps

• Before we wrap up, is there anything else I should have asked, or you’d like to share?

[Thank participants for their time. Explain the timeline for next steps and remind them that their feedback will be synthesized and shared back with OFVPS to make recommendations to outcome reporting tools and guidance. Stop recording if the session was recorded.]

Post Meeting and Internal Team Roles

Primary Facilitator: Leads the discussion during the listening sessions. Primary facilitator will secure access to a premium Zoom account (enables longer meetings, more participants, etc.).

Secondary Facilitator: Supports the primary facilitator by monitoring the chat. The secondary facilitator also serves as backup in the event primary facilitator is unavailable or experiences technical difficulties. Secondary facilitator will secure access to a premium Zoom account (enables longer meetings, more participants, etc.).

Primary Notetaker: Captures relevant information and content during the listening session. Primary notetaker will also share their screen, record the session, and save chat history and transcript before closing out of Zoom. After the listening session, upload the documents to MITRE SharePoint site and clean up the notes ahead of high-level analysis. Upload meeting notes to MITRE SharePoint site 1-2 days after the listening session with the naming convention ‘Listening Session X Notes_YYMMDD’.

Secondary Notetaker: Support the primary notetaker by capturing relevant information and content during the listening sessions. Secondary notetaker also serves as backup in the event primary notetaker is unavailable or experiences technical difficulties.