SSA Prog Supp GenIC - Pathways to Work Evidence Clearinghouse Feedback Sessions

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Pathways to Work Evidence Clearinghouse Feedback Sessions

Formative Data Collections for Program Support

0970 – 0531

Supporting Statement

Part A

November 2024

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Amelia Popham and Marie Lawrence

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a generic information collection under the umbrella generic, Formative Data Collections for Program Support (0970-0531).

• Description of Request:

The Pathways to Work project team is seeking clearance to conduct six focus group sessions with 6-8 participants each, representing different stakeholder groups, including workforce program providers, TANF staff, policymakers, researchers, and individuals with lived experience. These sessions aim to gather feedback on evidence needs, usage, website content, and product preferences related to the Pathways to Work Evidence Clearinghouse, a federal website sharing findings on employment and training interventions for low-income job seekers. The information collected will be used to improve product offerings and the Pathways to Work website for workforce professionals. It is not intended to be generalized to a broader population or to inform influential policy decisions. We do not intend for this information to be used as the principal basis for public policy decisions.

• Time Sensitivity: The Pathways to Work website is scheduled to be overhauled and re-released next spring. In order for this data collection to inform the structure and offerings on the website, data collection will need to be completed by January 2025. In order to meet that timeline, we will need to begin data collection by early December.

A1. Necessity for Collection

The Pathways to Work Evidence Clearinghouse aims to improve evidence-based decision making among employment and training programs serving low-income job seekers. To ensure Pathways to Work effectively meets user needs, direct feedback from key stakeholder groups is essential. This information collection will provide critical insights to enhance the Clearinghouse's utility and downstream influence on decision making.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The Pathways to Work Evidence Clearinghouse (pathwaystowork.acf.hhs.gov) is a federal website sponsored by the Office of Planning, Research, and Evaluation (OPRE) within the Administration for Children and Families (ACF). The website shares findings from systematic evidence reviews examining the effectiveness of interventions designed to improve outcomes for job seekers with low incomes. One of the website's goals is to help program providers and policymakers, including state and local TANF administrators, make evidence-informed decisions as they design programs to improve the odds that the job seekers they serve succeed.

The Pathways to Work project team seeks clearance to hold sessions with small groups of individuals representing six different customer groups in the winter of 2024. These sessions aim to help us learn more about the evidence needs of a diverse range of users. This will include feedback on the following themes: current evidence use, evidence needs, website content, and product preferences. Facilitators will use the proposed facilitation guides for each session. Questions will be tailored to each group's relevant context within those themes. There will be six sessions in total, with one session per user group of 6-8 participants each:

1. Workforce program providers

2. Frontline staff of TANF or workforce programs

3. State TANF administrators and state-level TANF staff

4. State and local policymakers

5. Researchers, including evaluators and university-based academics

6. Individuals with lived experience (i.e., individuals who have participated in employment and training programs or have received TANF benefits in the last five years)

Information from these feedback sessions will be summarized in an internal report and will be used to find ways to make the Pathways to Work Evidence Clearinghouse more useful to workforce professionals. This information will also be used to inform a major revision of the website.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests/Guiding Questions

1. What information do program providers, policymakers, and individuals with lived experience hope to obtain when they visit the Pathways to Work Evidence Clearinghouse?

2. How can the Pathways to Work Evidence Clearinghouse provide more engaging and useful content?

Study Design/Information Collection Procedures and Processes

The Pathways to Work Clearinghouse will identify individuals for the feedback sessions from a list of individuals who were nominated, identified, or volunteered to participate in the feedback sessions from a number of sources. These sources include:

• Nominations from Evidence Champions on the Pathways to Work project. These are field-based practitioners who advise the project.

• Nominations from ACF's Office of Family Assistance (OFA).

• Volunteers solicited through our project's quarterly newsletter.

• Nominations from the Pathways to Work team's professional networks and internal teams (Abt Global and American Institutes for Research). We will recruit participants who represent a range of backgrounds, geographic diversity, and diversity of professional experience. Nominations will be vetted by cross-checking credentials with publicly available information.

All feedback sessions will be recorded, transcribed and analyzed using either qualitative data analysis software or excel spreadsheets. We will categorize data based on themes using inductive methods. Through this method we will derive themes and identify relationships and patterns across the data. Findings will be reported in via short brief.

Data Collection Activity	Instruments	Respondent, Content, Purpose of Collection	Mode and Duration
Feedback Session A	Focus Group with Workforce Development Professionals who Oversee Workforce and Job Training Programs	Respondents: Workforce Development Professionals who oversee workforce and job training programs Content: Work responsibilities, services offered, personal research use, decision-making processes, use of research in decision making, accessing research (trusted sources, challenges, and potential solutions), research needs (products and content), and dissemination preferences Purpose: Identify how Pathways to Work can better serve the needs of workforce development professionals	Mode: Focus group (virtual) Duration: 90 minutes
Feedback Session B	Focus Group with Frontline Staff Directly Responsible for Delivering TANF or Workforce Development Services	Respondents: Frontline staff directly responsible for delivering TANF or workforce development services Content: Work responsibilities, services offered, personal research use, decision-making processes, use of research in decision making, accessing research (trusted sources, challenges, and potential solutions), research needs (products and content), and dissemination preferences Purpose: Identify how Pathways to Work can better serve the needs of frontline staff	Mode: Focus group (virtual) Duration: 90 minutes
Feedback Session C	Focus Group with TANF Administrators or State-Level TANF Staff	Respondents: TANF Administrators or state-level TANF staff Content: Work responsibilities, services offered, personal research use, decision-making processes, use of research in decision making, accessing research (trusted sources, challenges, and potential solutions), research needs (products and content), and dissemination preferences Purpose: Identify how Pathways to Work can better serve the needs of TANF administrators and state-level TANF staff	Mode: Focus group (virtual) Duration: 90 minutes
Feedback Session D	Focus Group with State and Local Policymakers	Respondents: State and local policymakers Content: Work responsibilities, services offered, personal research use, policy-making process, use of research in decision making, accessing research (trusted sources, challenges, and potential solutions), research needs (products and content), adapting Pathways to Work for policymakers, and dissemination preferences Purpose: Identify how Pathways to Work can better serve the needs of state and local policymakers	Mode: Focus group (virtual) Duration: 90 minutes
Feedback Session E	Focus Group with Researchers	Respondents: Researchers Content: Use of the Pathways to Work Evidence Clearinghouse, understanding evidence, identifying research gaps, researcher engagement strategies Purpose: Identify how Pathways to Work can better serve the needs of researchers	Mode: Focus group (virtual) Duration: 90 minutes
Feedback Session F	Focus Group with Individuals with Lived Experience	Respondents: Individuals with lived experience Content: Experience participating in employment or training programs, meaning of “research”, selecting a job training program, dissemination preferences Purpose: Identify how Pathways to Work can better serve the needs of individuals with lived experience, and how their experiences can inform the presentation of data on the website	Mode: Focus group (virtual) Duration: 90 minutes

Other Data Sources and Uses of Information

No other sources of information will be used in conjunction with the information collected through the focus groups.

A3. Use of Information Technology to Reduce Burden

The collection of information will primarily involve the use of web-based technology. The Pathways to Work project team has chosen to conduct the feedback sessions using online platforms, which allows for electronic submission of responses. This decision to adopt web-based collection methods was made to reduce burden on respondents by eliminating the need for travel and allowing participation from diverse geographic locations.

The use of online platforms for these focus group sessions also enables more efficient scheduling, participation, and potentially recording and transcription of the sessions (if we get participants’ consent to do so).

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

This information does not exist elsewhere so it is necessary for us to engage in this data collection. We will minimize the burden on potential respondents by using technology to facilitate scheduling. We will maximize utility to the government by sharing both the findings in a memo whose insights can be broadly applicable for other evidence to action efforts.

A5. Impact on Small Businesses

No small businesses will be involved with this information collection.

A6. Consequences of Less Frequent Collection

This is a one-time data collection.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection request to extend approval of the umbrella generic with minor changes. The notice was published on January 28, 2022, (87 FR 4603), and provided a sixty-day period for public comment. ACF did not receive any comments on the first notice. A second notice was published, allowing a thirty-day period for public comment, in conjunction with submission of the request to OMB. ACF did not receive any comments on the second notice.

Consultation with Experts

For our actual data collection, we'll be partnering with experts beyond our immediate project circle. This includes frontline staff from program providers, local TANF agency representatives, and community college personnel. Our Evidence Champions, while providing valuable high-level strategic perspectives on Pathways to Work, are nationally-recognized experts who may not represent our typical users. This current data collection effort aims to supplement their feedback by gathering insights from individuals with direct, hands-on experience. By engaging these local experts, we'll gain a more accurate picture of our typical users' needs and experiences, offering a practical counterpoint to the strategic view provided by our Evidence Champions.

A9. Tokens of Appreciation

We are not proposing tokens of appreciation but will provide honoraria, as described in A13.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

We are collecting Personally Identifiable Information. To conduct recruitment, schedule meetings, and process honoraria, we will need to collect the respondent’s name, title, organization, and email address.

In preparing data for analysis, we will strip all personal identifiers from the data so the individual responses cannot be linked to a respondent. Although our recordings of these sessions will generate a transcript, the transcript will be deleted immediately after session notes are finalized so individuals cannot be reidentified in the notes or in any of our analytic files.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

This information collection was submitted to Abt Global’s Institutional Review Board for review (IRB# 00001281; FWA# 00000664 [expires 6/7/2029]). The IRB determined that this project does not need formal review by the Abt IRB because none of the SOW activities meet the US federal definition of human subjects research. The IRB also conducted a separate “ethics/privacy” check which included the development of a data security plan.

Data Security and Monitoring

Notes, transcripts, and recordings of feedback sessions will be kept in a SharePoint folder that will only be accessible to the staff working directly on the feedback sessions. Access to this SharePoint folder is easily monitored using the project’s Microsoft Teams channel. The final dataset of respondents’ answers will be stripped of identifying information—including not only names but also details that could reidentify a respondent—before it is transmitted to OPRE as part of the project transition at the end of the contract.

A11. Sensitive Information ¹

The information collected does not request sensitive information.

A12. Burden

Explanation of Burden Estimates

There will be 48 participants in the Feedback Sessions. This includes 32 individuals with lived experiences and researchers, and 16 participants who are state, local, and/or tribal government representatives. The hour burden on respondents is expected to vary due to differences in activity, size, and complexity of the tasks involved for each group, but the average estimated time for the feedback session and prep time is 2 hours. Each respondent only attends one feedback session.

Estimated Annualized Cost to Respondents

The wage data used in this cost estimation is derived from the Occupational Information Network (O*NET). The hourly wage for each respondent group was determined by matching the group to the most appropriate O*NET occupation code. The total cost for each group was then calculated by multiplying the number of respondents by the total burden hours and the hourly wage. For the "Individuals with lived experience" group, we used the medical assistant occupation as a proxy for hourly wages. This choice was made because it is reported to be one of the common occupations for individuals receiving federal job training services (https://www.pw.hks.harvard.edu/post/publicjobtraining).

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total/ AnnualBurden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
Feedback Session A: Workforce Program Providers	8	1	2	16	$35.69	$571.04
Feedback Session B: Frontline Staff	8	1	2	16	$29.53	$472.48
Feedback Session C: State TANF Administrators and Staff	8	1	2	16	$35.69	$571.04
Feedback Session D: State and Local Policymakers	8	1	2	16	$23.12	$369.92
Feedback Session E: Researchers	8	1	2	16	$42.70	$683.20
Feedback Session F: Individuals with Lived Experience	8	1	2	16	$18.40	$294.40
Total	48	1	2	96	$30.85	$2,962

A13. Costs

Directly engaging the communities ACF serves and including these individuals in ACF research is in line with the following priorities of the current Administration and HHS:

Advancing Racial Equity and Support for Underserved Communities Through the Federal Government (EO 13985) 

Further Advancing Racial Equity and Support for Underserved Communities Through the Federal Government

Presidential Memorandum on Restoring Trust in Government through Scientific Integrity and Evidence-Based Policy Making

ACF Evaluation Policy 

HHS Strategic Plan FY 2022-2026,

ASPE’s Methods and Emerging Strategies to Engage People with Lived Experience (2021) 

ASPE’s Recruiting Individuals with Lived Experience (2022)

Equitable compensation is in line with leading practices for ethical engagement of those with lived expertise and advancing equity for populations who have been historically underserved (as noted in section A1, advancing equity is a priority, as highlighted in the referenced EOs in that section). Providing equitable compensation recognizes the value of the time provided by participants, helps to remove barriers to participation, and affirms that the contributions from those with lived experience are as valuable as those from other experts.

As noted in the 2022 report by ASPE this “helps ensure a diverse population with varied views can participate.” Additionally, in an earlier report it was noted that “Providing [those with lived experience] with compensation commensurate with the rates that other experts—i.e., experts engaged based on their expertise as practitioners or researchers, rather than lived experience—receive helped recognize the valuable and unique expertise that people with lived experience lend, which promoted meaningful engagement.”

Consistent with the guidance documents referenced, and to ensure involvement with a variety of people with diverse experiences and perspectives in relevant fields, we propose to offer participants an honorarium for their time spent providing their expertise and experience. Specifically, we propose to offer each respondent an honorarium of $100. The study team anticipates it will take respondents about 30 minutes to prepare for focus group sessions and anticipates that each session will take approximately 90 minutes.

A14. Estimated Annualized Costs to the Federal Government

The estimated annualized cost to the federal government is $36,193.95.

Cost Type	Total Hours for All Staff	Number of Staff on Each Task	Average Hourly Wage Rate (loaded)	Total Cost to Federal Government
Meeting logistics (scheduling calls, delivering honoraria)	12	1	$132.60	$1,591.20
Feedback sessions	18	2	$181.04	$3,258.63
Data analysis	64	2	$156.82	$10,036.32
Reporting	32	3	$202.82	$6,490.32
Internal reviews and revisions	36	4	$241.16	$8,681.68
Honoraria				$4,800
Federal staff time	20	2	$66.79	1,335.80
Total	182			$36,193.95

Federal staff average hourly wage rate was determined by using the GS-14 Step 1 hourly rate for an employee located in Washington, D.C., according to the U.S. Office of Personnel Management’s General Schedule.

A15. Reasons for changes in burden

This is for an individual information collection under the umbrella formative generic clearance for program support (0970-0531).

A16. Timeline

Estimated Timeline for Pathways to Work Clearinghouse Feedback sessions

Activity	Proposed Timeline
Upon OMB Approval, Recruitment Begins	November 2024
Host Feedback Sessions	December 2024
Analyze Data and Draft Findings	January 2025
Submit Draft Brief to OPRE	February 2025
Revise and Submit Final Brief	March 2025

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

1. Attachment A: Pathways to Work Evidence Clearinghouse Feedback Session Instruments

2. Attachment B: IRB Determination

1 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

12