SSA Prog Supp GenIC - ACF Evidence Capacity Support

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

ACF Evidence Capacity Support

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

January 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Nicole Deterding

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the generic title of Formative Data Collections for Program Support (0970-0531).

• Description of Request: The Office of Planning, Research, and Evaluation in the Administration for Children and Families proposes to conduct up to 35 one-hour interviews with Office of Refugee Resettlement (ORR) grant recipient programs and up to 12 group interviews with refugees who receive services or funds from ORR programs (up to 40 individuals across the groups). The purpose of the interviews is to gather input about (1) a theory of change for a national refugee program, (2) current performance reporting practices, and (3) potential revisions to performance reporting. OPRE will oversee the data collection, and Mathematica will collect the data. OPRE will share findings with ORR to inform potential changes to performance reporting.

We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: This request is time sensitive. Much of the funding for this work expires on September 30, 2024. As such, we hope to have OMB approval before June 2024 to allow time for interviews, analysis, and reporting before the end of September 2024.

A1. Necessity for Collection

The Office of Refugee Resettlement (ORR) is working to improve performance reporting across programs and grant recipients. As part of the Administration for Children and Families (ACF) Evidence Capacity Support project, the Office of Planning, Research, and Evaluation (OPRE) and its contractors at Mathematica seek to support ORR’s efforts to improve performance reporting through:

• Gathering input from ORR grant recipients on current performance reporting practices, ORR’s theory of change, and potential revisions to reporting.

• Finding ways to address data challenges including uniformity in performance measures across programs, duplicated administrative data across programs, a lack of timely administrative data, and a lack of client-level data.

• Developing a set of performance measures that focus on outcomes related to well-being for refugees served by the grantees. The measures support performance management, performance improvement, and meaningful accountability.

This proposed information collection is necessary to inform these efforts. The information we will collect is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision maker, and is not expected to meet the threshold of influential or highly influential scientific information.

A2. Purpose

Purpose and Use

The purpose of this proposed information collection is to obtain input from ORR grant recipients and individuals from refugee populations to inform ORR’s theory of change and revisions to program performance reporting, including measures and processes. ACF will use this information to improve performance reporting requirements for grant recipients. Specifically, ORR aims to use the feedback to inform the following efforts:

• Updates to ORR’s theory of change

• Improvements to current performance reporting practices, including finding ways to address data challenges such as uniformity in performance measures across programs, duplicated administrative data across programs, a lack of timely administrative data, and a lack of client-level data

• Development of a set of performance measures that focus on outcomes related to well-being for refugees served by the grantees

• Ensuring that measures support performance management, performance improvement, and meaningful accountability

This proposed information collection meets the following goals of ACF’s generic clearance for formative data collections for program support (0970-0531):

• Obtaining input on the development of program performance measures (PM) from grant recipients or experts in a relevant field

Guiding Questions

• Current performance measurement

• How complete and high in quality are current performance measure data? Are the current measures reliable?

• How great is the burden on grant recipients to supply data? Does the burden extend to individuals who receive grant-funded services?

• Who uses the data, for what purpose, and with what level of satisfaction? Do the data inform meaningful decisions about program funding, design, operations, or effectiveness?

• Potential changes to performance measurement

• How well do ORR grant recipients think the theory of change captures the goals of refugee resettlement programs? Are there key concepts that should be added or subtracted?

• How does the theory of change align with (1) the data that resettlement organizations currently report to the federal government and (2) information organizations currently collect and do not currently report?

• What would have to be done to collect uniform information that aligns with the theory of change, streamlines federal reporting, and reduces administrative burden on grant recipients and program participants?

• What kinds of information may be better collected through other means, such as program monitoring site visits, topical case studies, and formal program evaluation?

Study Design and Universe of Data Collection Efforts

Data collection activity	Instruments	Respondent, content, purpose of collection	Mode and duration
Interviews with grant recipients	Instrument 1: Interview discussion guide	Respondents: Up to 35 grant recipients, including people involved in program administration and delivery from state grant recipients, national resettlement agencies, and local affiliates. Content: Questions related to grant recipients’ current performance reporting; feedback on a theory of change for refugee resettlement; alignment of current reporting with that theory of change; and suggested revisions to performance reporting. Purpose: To gather information that ORR can use to collect meaningful and better-quality data.	Mode: Videoconference call Duration: 1 hour/ respondent
Interviews with refugees	Instrument 2: Participant discussion guide Appendix: Program participant screening questions	Respondents: Representatives of each of six refugee populations. Populations and representatives will be identified in consultation with ORR. This equates to 12 group interviews with up to 40 individuals. Content: Questions related to components of the theory of change on refugee resettlement, including challenges and outcomes identified in the theory of change. Purpose: To gather information that ORR can use to collect meaningful and better quality data.	Mode: Videoconference call Duration: 1 hour/ group

Other Data Sources and Uses of Information

Interviews will be combined with other sources of information. Before conducting interviews, we will review: (1) written materials related to current or planned ORR performance reporting and (2) portals and online systems that capture ORR grant recipients’ performance data.

A3. Use of Information Technology to Reduce Burden

Participating in this information collection will not require any in-person activities or printing of any materials. The study team will use information technology as appropriate to reduce the burden on respondents. Mathematica will conduct the interviews via a videoconferencing platform that can be accessed on a computer, phone, or tablet (e.g., Webex). When we schedule the interview, we will email respondents an overview of the interview topics.

Interviews and focus groups will be transcribed. With participant consent, information in the transcripts may be revisited for secondary analysis at a future date, with a goal of reducing burden on the group of respondents.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The proposed interviews do not duplicate other data collection efforts. The information the contractor will collect through the interviews is not available through other existing data sources.

A5. Impact on Small Businesses

Resettlement agencies and local affiliates involved in information collection may be considered small businesses. The study team will make every effort to reduce burden on all participating organizations. The study team will ensure interviews are scheduled at a time convenient for the respondent.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published two notices in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on January 28, 2022, (87 FR 4603), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.

Consultation with Experts

No consultations have taken place with experts outside of the project team. 

A9. Tokens of Appreciation

It is extremely important to provide those with lived experience with equitable compensation or tokens of appreciation for participation. As noted in a 2022 report by the HHS Office of the Assistant Secretary for Planning and Evaluation, this “helps ensure a diverse population with varied views can participate” (3).¹ As such, we plan to provide honoraria to refugee participants, as described in section A13.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

When potential respondents self-nominate for participation, the research team will collect refugee participants’ names, email addresses, phone numbers, refugee or other status on arrival to the U.S., gender identity, arrival date, country of origin, religion and group membership (e.g. ethnic group or tribe), and city and state of current residence. The purpose of collecting this PII is to:

1. Determine eligibility and selection into culturally appropriate interview groups;

2. Contact potential participants to schedule interviews;

3. Monitor and seek to achieve balance across the interview groups in key areas related to the data collection, including length of time in the U.S., primary type of services received, urbanicity, refugee status or stream, and gender;

4. Send honoraria to refugees who participate in interviews.

All notes taken during the interviews will be de-identified (see below for additional information). The study team will not maintain information in a paper or electronic system that retrieves data by using an individual’s personally identifiable information in the way that could trigger the Privacy Act of 1974, as amended (5 U.S.C. 552a).

Assurances of Privacy

Any information collected will be kept private. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private. We may share collected transcripts from interviews/discussions with ACF staff or other researchers to support them in reports about refugee experiences or performance measurement. However, in all circumstances, we will de-identify or remove all PII such as names and email addresses before any usage or sharing of any data including interview notes, records, and transcripts. As specified in the contract, the contractor will comply with all federal and departmental regulations for private information.

Data Security and Monitoring

People who will have access to the artifacts of this data collection, including interview notes, recordings, and transcripts, have received training about data privacy issues. Mathematica will store data on its secure drive, which only authorized users have access to. The secure drive uses Windows Active Directory security groups for access control, and uses Encrypting File System (EFS) on demand. Mathematica will use a secure file transfer protocol to transmit data to ACF.

A11. Sensitive Information ²

The proposed information collection does not request any sensitive information.

A12. Burden

Explanation of Burden Estimates

For the interviews with up to 35 grant recipients, the number of respondents was chosen to allow for geographic and variation within different categories of parties involved in ORR program administration and delivery. This includes state-level grant recipients (state government agencies and replacement designees), national resettlement agencies, and local affiliates. To ensure a diversity of perspectives, respondents will vary in terms of geography (participants from the four major US Census regions (Northeast, South, Central/Midwest, and West), service area (urban/rural), and the types of ORR-funded programs and services delivered. Respondents will be asked to participate in one interview that is estimated to take approximately 60 minutes to complete.

For the small group interviews with up to 40 refugees, the number of respondents was chosen to allow for variation within the refugee population. Refugees who self-nominate will be asked to complete a set of screening questions to determine eligibility and selection into culturally appropriate interview groups. (Questions include name, email address, refugee or other status on arrival to the U.S., gender identity, arrival date, country of origin, religion and group, and city and state of current residence.) Representatives of each of six refugee populations will be selected to ensure a diversity of perspectives. Respondents will be asked to participate in one small group interview that is estimated to take approximately 60 minutes to complete.

Estimated Annualized Cost to Respondents

The estimated annual cost for respondents is shown in Exhibit 2. The source for the mean hourly wage information for respondents is Bureau of Labor Statistics, Occupational Employment and Wages, May 2022. Burden estimates were calculated assuming one program director from each program would contribute to their program’s interview response. Hourly wage for program directors is estimated at $38.13 based on the BLS average earnings for Social and Community Service Managers. Burden estimates for refugees is estimated at $29.76 and reflects the average wage for all occupations.

Instrument	No. of respondents (total over request period)	No. of responses per respondent (total over request period)	Avg. burden per response (in hours)	Total/ annual burden (in hours)	Average hourly wage rate	Total annual respondent cost
Instrument 1: Discussion guide for the grant recipient interview	35	1	1	35	$38.13	$1,334.55
Instrument 2: Discussion guide for the refugee interview	40	1	1	40	$29.76	$1,190.40
Instrument 2 Appendix: Program participant screening questions	80	1	0.067	5.36	$29.76	$159.51
Totals:	115			80.36		$2,684.46

A13. Costs

Directly engaging the communities ACF serves and including these individuals in ACF research is in line with the following priorities of the current Administration and HHS:

• Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (EO 13985)

• Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government

• ASPE’s Methods and Emerging Strategies to Engage People with Lived Experience (2021)

• ASPE’s Recruiting Individuals with Lived Experience (2022)

Consistent with the guidance documents referenced, we propose to offer refugee participants an honorarium for their time spent providing their expertise and experience. Specifically, we propose to offer $30 honoraria for participation in interviews to ensure a diverse representation of participants across the country.

Equitable compensation is in line with leading practices for ethical engagement of those with lived expertise and advancing equity for populations who have been historically underserved. Providing equitable compensation recognizes the value of the time provided by participants, helps to remove barriers to participation, and affirms that the contributions from those with lived experience are as valuable as those from other experts.

As noted in the 2022 report by ASPE this “helps ensure a diverse population with varied views can participate” (3).³ Additionally, a 2021 brief by ASPE says that “Providing [those with lived experience] with compensation commensurate with the rates that other experts—i.e., experts engaged based on their expertise as practitioners or researchers, rather than lived experience—receive helped recognize the valuable and unique expertise that people with lived experience lend, which promoted meaningful engagement.” The report goes on to specify that not doing so could result in “unintended consequences…when lived experience engagements have scarce resources and experts are undercompensated, which can undermine, disregard, and/or marginalize people with lived experience” (11).⁴

A14. Estimated Annualized Cost to the Federal Government

Activity	Detail	Estimated Cost
Interview administration	Full-time equivalent time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses that would not have been incurred without this collection of information Refugee participant honoraria	$71,734
Analysis and initial dissemination	Full-time equivalent time Operational expenses (such as equipment, overhead, printing, and staff support) Other expenses that would not have been incurred without this collection of information	$61,680
Total costs over the request period		$133,414

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

November 2023

December 2023

Jan 2024

Feb 2024

March 2024

April 2024

May 2024

June 2024

July 2024

Aug 2024

Sep 2024

Oct 2024

Nov 2024

Protocol development

X

OMB submission and clearance

X

X

Conduct discussions and analysis (Program Admin)

X

X

X

X

Conduct discussions and analysis (Program Participants)

X

X

Draft and finalize results brief

X

X

X

X

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

• Instrument 1: Program administrators: interview discussion guide

• Instrument 2: Program participants: interview discussion guide

• Instrument 2 Appendix: Program Participant screening questions

1 U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. “Recruiting Individuals with Lived Experience,” by Jasmine Forde, Sonia Alves, Lauren Amos, Ryan Ruggiero, Annalisa Mastri, Kate Bradley, Nkemdiri Wheatley, Tonyka McKinney, Dana Jean-Baptiste, Jeremiah Donier, Madison Sandoval-Lunn, Wilnisha Sutton, Roger De Leon, Kataney Prior, Laura Erickson, Amanda Benton, and the HHS staff peer learning community on equitably engaging people with lived experience. Washington, District of Columbia: 2022. https://aspe.hhs.gov/reports/recruiting-individuals-lived-experience

2 Examples of sensitive topics include (but are not limited to): Social Security number; sexual behavior and attitudes; illegal, antisocial, self-incriminating, and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, such as family, pupil-teacher, or employee-supervisor; mental and psychological problems potentially sensitive for respondents; religion and indicators of religion; community activities that indicate political affiliation and attitudes; legally recognized privileged and analogous relationships such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (for example, unemployment or WIC or SNAP); and immigration/citizenship status.

3U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. “Recruiting Individuals with Lived Experience,” by Jasmine Forde, Sonia Alves, Lauren Amos, Ryan Ruggiero, Annalisa Mastri, Kate Bradley, Nkemdiri Wheatley, Tonyka McKinney, Dana Jean-Baptiste, Jeremiah Donier, Madison Sandoval-Lunn, Wilnisha Sutton, Roger De Leon, Kataney Prior, Laura Erickson, Amanda Benton, and the HHS staff peer learning community on equitably engaging people with lived experience. Washington, District of Columbia: 2022. https://aspe.hhs.gov/reports/recruiting-individuals-lived-experience.

4
U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation. “Methods and Emerging Strategies to Engage People with Lived Experience. Office of the Assistant Secretary for Planning and Evaluation,” by Syreeta Skelton-Wilson, Madison Sandoval-Lunn, Xiaodong Zhang, Francesca Stern, and Jessica Kendall. Washington, District of Columbia: 2021. https://aspe.hhs.gov/reports/lived-experience-brief.

10