SSB - FSSDD SEEDS GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Family Self-Sufficiency Demonstration Development Grants Evaluation Support: Data Collection for Final Report

Formative Data Collections for ACF Research

0970-0356

Supporting Statement

Part B

October 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Amelia Popham, Emily Ross, Sarita Barton, and Siri Warkentien

Part B

B1. Objectives

Study Objectives

The objectives of the information collection are to 1) describe the type and frequency of coaching and support activities grant recipients participated in; 2) understand grant recipients’ perceptions of the support they received; and 3) assess the extent to which grant recipients developed their capacity to conduct research and evaluation activities during the grant period. We will conduct semi-structured discussions and administer surveys with staff from each of the 20 Administration for Children and Families’ (ACF) Family Self-Sufficient Development Demonstration (FSSDD) grant recipients. Each FSSDD grant recipient received technical assistance support to conduct research and evaluation activities on an innovative intervention intended to improve family and economic well-being among populations with low incomes.

Responses will be used to assess the grant recipients’ needs, capabilities, and readiness for further rigorous research and evaluation activities. Responses will also be used to document the work of the FSSDD Evaluation Support team and share lessons learned about strategies for building grant recipients’ evaluation capacity through the provision of technical assistance.

Generalizability of Results

This study is intended to present internally-valid descriptions of the coaching and support FSSDD grant recipients received and their perceptions of the quality and impact of the support, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The data collection methods are appropriate for the objectives of the information collection. Qualitative, semi-structured discussions will support detailed descriptions of grant recipients’ experiences with support and coaching during the grant period, including the challenges they encountered and what they learned. These discussions will also allow Contractor staff to gather information about grant recipients’ plans for research and evaluation activities after the grant period and develop recommendations for how ACF and its Contractors can best support and build grant recipients’ evaluation capacity in the future. Brief web-based surveys will provide information on the utility of specific services and supports provided by the FSSDD Evaluation Support team; gather information about specific research methods used; and offer an additional avenue for candid feedback.

Data collected under this information collection are not intended to be representative of broader populations. Results are not designed to be representative of or generalizable to a given subpopulation—the intent is to gather information specific to each grant recipient’s experience to inform lessons learned for building the research and evaluation capacity of human services organizations through the provision of technical assistance. Data will not be used to assess participant outcomes. Such limitations will be included in written products associated with this data collection.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

The unit of analysis is the FSSDD grant recipient, of which there are twenty. The respondents are individuals who are employed in leadership, service delivery, and research positions by the organization that received the FSSDD grant and external evaluators that were hired by the grant recipient organizations under the terms of the grant.

Sampling

Respondents are staff working in organizations that are ACF grant recipients. They will be asked to participate in these discussions and complete surveys as part of their grant activities. The Contractor will use non-probability, purposive sampling to identify respondents who can provide information on key questions. Because participants will be purposively selected, they will not be representative of the population of human services organizations or ACF grant recipients.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

We developed three instruments to support this information collection:

• Instrument 1_SEEDS Discussion Guide GR Final Research and Eval Capacity

• Instrument 2_SEEDS Discussion Guide GR Resource Allocation

• Instrument 3_SEEDS Survey to Collect GR Feedback

The questions in the instruments are informed by the research questions guiding the analysis (See Supporting Statement A, section A2) and will provide the information needed for reporting purposes.

Instrument 1 assesses the grant recipients’ capacity to build and use evidence (evidence capacity) at the conclusion of their project period. It includes additional questions about grant recipients’ perceptions of the support they received, challenges they faced during the grant, and plans for their future research and evaluation efforts. Before conducting data collection activities with the FSSDD grant recipients, Contractor staff will review grant recipients’ grant applications, data from the initial capacity assessment, and internal notes from two years of individualized coaching meetings. Contractor staff will use this information to indicate in the discussion guide which information is already known and not request this information from the participant. This will reduce the burden associated with this information collection and ensure no information collected is duplicative of what was already shared with ACF and the Contractor. Respondents will include the key members of the grant recipient’s FSSDD team who regularly participated in SEEDS coaching and other activities.

Instrument 2 is a semi-structured discussion guide that will capture information on how grant recipients allocated resources to grant activities, including staff allocated to the FSSDD project, any other fiscal or in-kind resources leveraged for the project, and grants management. Respondents will include the FSSDD grant recipient organization leader(s) and/or fiscal staff who are most familiar with the grant’s resources.

Instrument 3 is a brief survey that will capture quantitative information that is less conducive to collecting during a semi-structured interview. The survey requests information about the supports grant recipients used during the grant period, their ratings of the utility of those supports, the research methods grant recipients used, a self-assessment of how grant recipients’ knowledge changed during the grant period, and feedback on the coaching grant recipients received. Respondents include any members of the grant recipient’s FSSDD team who regularly participated in SEEDS coaching and other activities.

B4. Collection of Data and Quality Control

Mathematica, an ACF contractor, and its subcontractor, The Adjacent Possible, will collect data. Contractor and subcontractor staff have held regular coaching calls with grant recipients under the auspices of the ACF-funded FSSDD grant for the past 18 months. However, because the information collection will be asking grant recipients to provide feedback on the support they received (as well as other topics), a different person than the grant recipient’s regular coach will collect data from the grant recipient. The goal of this approach is to elicit honest feedback from grant recipients. Respondents will be reminded that the feedback they provide will not be shared directly with their coaches and will not affect their prospects to receive additional funding from ACF in the future.

To conduct this information collection, staff from the contractor and subcontractor will reach out to the main point of contact at each grant recipient and identify a convenient time to facilitate each discussion. The discussions will happen via the WebEx videoconferencing platform. Contractor and subcontractor staff will send the survey to grant recipients electronically.

To ensure consistency and quality across data collection activities, contractor staff will be trained on how to implement the discussion guide by project leadership before conducting any discussions. Project leadership will also review all notes from grant recipient discussions and note any areas for follow-up. If needed, staff will reach back out to grant recipient teams via email or phone to clarify inconsistent or incomplete information.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection activities, including both the discussion guides and the survey, are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.

B6. Production of Estimates and Projections

The data are strictly qualitative and descriptive. They will not be used to generate population estimates, either for internal use or dissemination. The data will not be used to make policy decisions.

B7. Data Handling and Analysis

Data Handling

Each discussion will be conducted by two Contractor staff, with one staff member leading the conversation and the other taking notes. Discussions will also be audio-recorded (if permission is given by the grant recipient staff). Survey response data will be collected electronically via the QuestionPro platform. To minimize respondent error, questions will be designed to be categorical or multiple choice whenever possible and validation rules will be applied to responses.

Data Analysis

Data collected during this information collection will be combined with information from grant recipients’ initial capacity assessment (assessed at the start of the grant period), grant applications, and notes collected during coaching meetings. Contractor staff will extract key themes from discussion notes and audio recordings to assess the extent to which grant recipients developed their capacity to conduct research and evaluation activities during the grant period, grant recipients’ needs and capabilities for further evidence building, and grant recipients’ readiness for a future rigorous evaluation. Categorical survey data will be tabulated or averaged to supply summary statistics. Open-ended survey questions will be coded to enable frequency analysis and provide insight into common themes.

Data Use

This information will contribute to an internal report to ACF documenting the work of the FSSDD Evaluation Support team and sharing lessons learned for building grant recipients’ capacity to conduct research and evaluation activities through the provision of technical assistance to support future work. The data will also be used to develop summaries for FSSDD grant recipients to support their plans for after the grant period. These summary memos will recap each grant recipient’s plans for research and evaluation activities after the grant period and provide recommendations for how grant recipients can continue to build their evaluation capacity in the future. Contractor staff will deliver these summaries to each grant recipient within two months of the end of their grant periods. This information will also inform briefs for the field about grant recipient achievements, challenges, and experiences around specific topics related to evidence capacity building. These briefs, with anticipated publication in 2025, will be informational for TA providers who are working with community-based organizations to improve their capacity to conduct research and use evidence.

B8. Contact Persons

The OPRE representative is Emily Ross, Social Science Research Analyst, Division of Economic Independence, OPRE, ACF, HHS. Her email is Emily.Ross@acf.hhs.gov.

The Contractor representative is Julia Lyskawa, Senior Researcher, Mathematica. Her email is JLyskawa@mathematica-mpr.com.

Attachments

List of instruments

• Instrument 1_SEEDS Discussion Guide GR Final Research and Eval Capacity

• Instrument 2_SEEDS Discussion Guide GR Resource Allocation

• Instrument 3_SEEDS Survey to Collect GR Feedback

4