ACF Evidence Capacity Support

EvCap ORR ASFR Program administrator protocol

EvCap Engagement with ORR and ASFR on Performance Measurement: Discussion guide for parties involved in ORR program administration and delivery

Thank you for your willingness to speak with me today about how the Office of Refugee Resettlement (ORR) measures program performance. Your participation is very important to this study. My name is [name] and I work for Mathematica, an independent research firm.

As mentioned in the email we sent you, we are conducting a study for the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services to learn how ORR might increase the relevance and quality of the data ORR collects on program performance. In particular, we are focused on the data points ORR collects from programs that aim to measure program outcomes. [For state-level and Resettlement Agency respondents: You may be familiar with some measures through the AOGP or other program-specific reporting forms.]

This interview is one of up to 35 interviews we will conduct with staff at organizations that are funded by ORR programs. We will also be interviewing program participants. This interview will last approximately 60 minutes.

Participation in this discussion is voluntary, and you can choose to not answer or skip any question if you wish. Our internal report will describe the experiences and viewpoints expressed by those we speak with, but comments will not be attributed to specific individuals or programs. No individuals will be quoted by name. We will share transcripts from all discussions with staff at ACF and transcripts may be used by ACF or other researchers in reports about performance measurement. Before we share transcripts, we will remove all names and any information that could identify you.

I’m going to read a statement for your awareness: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: 0970-0531, Expiration: 9/30/2025.

During the discussion, my colleague [name] will be taking notes. [If relevant: Also, on today’s call are [name(s)] from ACF, who are interested in learning more about this work.] With your permission, we would like to audio-record our conversation to help us keep track of your responses to the questions. This information is meant simply to serve as a record of what you and I discussed, and we will not share the recording outside of the study team. The recording will be destroyed at the end of the project.

Is it okay to record? [Note to interviewer: Start recording if participant says yes.]

Is it okay to share transcripts of this conversation with ACF?

Do you have any questions before we begin?

[Note to interviewer: Bold text indicates questions to prioritize if you are short on time. Tailor parenthetical text to role of respondent.]

Background information

To get started, we’d like to talk about your background and role with [agency/organization].

1. What is your job title or position?

2. Can you briefly describe the roles and responsibilities in your job?

3. [If applicable] What ORR-funded programs and/or services does your [agency/organization] provide?

Current outcome and performance measures

Next, we’d like to discuss required ORR performance measures. In our email to you before this interview, we included a list of the performance measures that we would like to focus on.

[Note to interviewers: spend no more than 15 minutes on this section.]

4. How [do you/does your program(s)/do the programs you oversee] collect and report these performance measures? [Interviewers screen share performance measures.]

5. Can you say a bit about your organization’s efforts to collect and report the data for the performance measures? We are interested in the time or resources it takes to collect this information.

1. Are some items easier or harder to collect and report?

2. How might collection and reporting be streamlined?

3. How burdensome is collecting and reporting this information? Can you share an example of what this burden looks like?

6. To what extent do you think the data used in the current performance measures are accurate? In what ways are they potentially inaccurate?

1. Are there any questions that are hard for you or for the people you serve to answer?

2. Are there any items where the instructions are unclear?

3. Are there any items that are hard for your organization to calculate?

7. How do [you/your program(s)/the programs you oversee] use the data you collect for ORR beyond reporting to the federal government?

1. [If not already mentioned] How do you use performance measure data to inform:

1. decision making?

2. program design?

3. program funding?

4. program operations?

5. increasing program effectiveness?

6. areas of improvement?

2. Who else in your organization uses the data? Do you know how they use it?

8. Do [you/your program/the program(s) you oversee] collect any additional data that you use for program monitoring or improvement—or in any other way? If so, can you describe this data and how it is used?

Theory of Change

Next, we want to move to discussing the draft ORR Refugee Programs National Theory of Change. Over the past several years, ORR has taken steps to improve the relevance and quality of information programs collect and produce to fulfill federal reporting requirements. In 2023, ORR convened workgroups to develop a national theory of change as one part of this process. A theory of change explains how a set of early and intermediate accomplishments sets the stage for producing long-range results. ORR is now collecting feedback on the Theory of Change, including through these interviews.

[Note to interviewer: Show bulleted list on screen for each of the elements in the Theory of Change. Interviewers should focus their follow up questions below on any sections of the Theory of Change identified in response to the first question in this series. Interviewers may also select follow up questions based on expertise of the respondent and time remaining in the interview.]

9. We would like your reactions to this Theory of Change based on your experience serving refugees. The Theory of Change describes three key assumptions about how ORR programs can lead to anticipated outcomes [read assumptions]. Is there anything you would add or subtract from this list?

10. It identifies four “problems and challenges” faced by refugees that ORR programs aim to address [read challenges]. Can you think of anything you would add or subtract?

11. It identifies six “activities, services, and strategies” ORR takes to address these problems and challenges and get to outcomes [read activities, services, and strategies]. Is there anything you would add or subtract?

12. How do you think about success for participants in your programs? What 2-3 outcomes do you hope they achieve:

1. In the first 90 days?

2. In the first year?

3. In the first five years after arrival?

13. ORR’s draft Theory of Change describes four potential high-level outcomes for the refugee program as a whole. What is missing or could be changed? [Note to interviewer: Follow up to focus discussion on high-level outcomes for the whole program, rather than just one part of refugee services.]

1. Does this align with your organization's goals for program participants?

2. Based on your experience with federal reporting, would your organization need to do anything different to report information like this? Do you have a sense of how difficult this would be?

Theory of Change relative to existing program data points

We are curious what information the state (or local affiliates) may already collect in these domains, even if it is not reported to the federal government.

14. What, if any, data do you collect on the outcomes and impacts listed in the [Physical and Behavioral Health] domain beyond what you already report to ORR? [Note to interviewer: Display domain list.]

1. [Repeat question for “Education and English Language Proficiency.”]

2. [Repeat question for “Employment and Economic Stability.”]

3. [Repeat question for “Social Adjustment and Integration.”]

15. Whether or not you view the Theory of Change as aligned with your current data collection, how might data related to the outcomes in the Theory of Change be helpful [to your program/the program(s) you oversee]?

16. Thinking about the other data you collect to support internal management of your program (program monitoring), would any of that data help inform the Theory of Change? If so, please explain.

17. What other feedback or thoughts do you have about the Theory of Change?

Closing

18. Are there any questions you thought we might ask but did not?

1. Anything else about data collection and reporting you would like to share with us?

Thanks again for your time and your valuable insights. If you think of anything to add later, you can email me at [email].

DRAFT 10/07/24 Mathematica^® Inc. 3