Supporting Statement B

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Next Steps for Rigorous Research on Two-Generation Programs

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

JULY 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer: Kathleen Dwyer

Part B

B1. Objectives

Study Objectives

The Next Steps for Rigorous Research on Two-Generation Programs (NS2G) project is designed to build the capacity of the two-generation field for future ACF research. Two-generation programs aim to improve outcomes related to child development and family economic security by intentionally combining services for parents and children in the same family. NS2G has three main study objectives:

1. To conduct formative research and provide evaluation technical assistance (TA) to better understand program implementation, strengthen promising programs, build their capacity to engage in data-informed continuous quality improvement, and prepare them for evaluations of effectiveness

2. To promote a better understanding of relevant processes and outcomes of two-generation programs for children in low-income families and their parents

3. To build the capacity of programs and researchers to conduct rigorous and meaningful evaluations of integrated approaches to supporting child development and improving family economic security

The information collection that is the subject of this request is relevant to objectives 1 and 2.

Generalizability of Results

This study is intended to present internally-valid descriptions of up to five specific two-generation programs and help these programs build capacity for future summative and effectiveness evaluations. The study does not intend to promote statistical generalization to other programs or service populations.

Appropriateness of Study Design and Methods for Planned Uses

NS2G builds on an earlier project, Exploration of Integrated Approaches to Supporting Child Development and Family Economic Security, which demonstrated that to prepare for effectiveness evaluations, two-generation programs will first need to use formative evaluation to refine their models (Sama-Miller et al. 2017). The goal of the NS2G formative evaluations and TA is to strengthen a group of up to five two-generation programs as they refine their models and move them closer to readiness for evaluations of their effectiveness.

The proposed information collection will support the individual programs’ formative evaluations and TA. The instruments are designed to build understanding across the NS2G team and the program itself of how each two-generation program is designed and implemented, the challenges and opportunities to strengthen its program design, and collect feedback on the implementation of specific strategies. Information collected from these instruments will be used to inform program development and enhancement. The proposed formative evaluations and TA are a collaborative process, and findings from administering the instruments will be shared with program staff.

Because this study aims to conduct formative research to better understand program implementation with up to five example programs, data collection for this study is not intended to promote generalizability of findings. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

NS2G defines two-generation programs as those that intentionally combine services intended to support both child development and family economic security. These programs intentionally serve parents and children (under age 12) from the same families. In 2016, the Exploration of Integrated Approaches to Supporting Child Development and Improving Family Economic Security (Integrated Approaches) project conducted a national scan to identify currently-operating two-generation programs. The scan identified 52 programs in operation. Of those programs, 21 programs were pilot programs or programs that had existed for less than three years, while only 2 programs had progressed to the point of participating in an outcomes evaluation. The relative newness of the field of two-generation programs suggested a need for formative evaluation and TA that would contribute to the continued development of the two-generation field and preparation of programs for evaluations of effectiveness.

Services provided by two-generation programs were influenced by the populations and areas that the programs served (Sama-Miller and Baumgartner 2017). All of the programs identified in the scan served low-income families; 19 served single mothers and their children only. Of the 52 programs, 44 served children ages birth to 3. The parent services generally focused on employment assistance and increasing skills and educational attainment through basic and postsecondary education, or individualized services such as case management (Sama-Miller and Baumgartner 2017). Services for children focused on school readiness and achievement, social-emotional and cognitive development, and physical and mental health. These programs frequently use ACF funding to provide early childhood services, such as TANF, Head Start, and Community Services Block Grants to support child services. Fourteen of the 52 programs provided Early Head Start or Head Start (Sama-Miller and Baumgartner 2017).

This will be a multisite study. ACF will select up to five two-generation programs to participate in the NS2G formative evaluations and TA. We plan to involve a maximum of 245 program staff and parents across all the sites.

Sampling and Site Selection

The NS2G project is using a purposive strategy to identify sites for NS2G. To narrow down the potential sites for selection, we considered programs that were (1) in the pool of programs identified in OPRE’s Integrated Approaches study, (2) nominated by ACF staff, or (3) recommended by one of the following two-generation program experts: Allison Holmes of the Annie E. Casey Foundation; Marjorie Sims of Ascend at the Aspen Institute; and Sharon McGroder, an independent consultant. From this narrowed list, the selected programs should:

• Be motivated and enthusiastic about participating in NS2G.

• Be implementing two-generation services, and not just be interested in providing services for low-income parents and their children.

• Serve populations of interest to ACF; at least one of the up to five programs should offer Head Start services

• Have already established partnerships to provide services, if partnerships with other organizations are necessary for them to provide two-generation services.

• Have relevant data available and have an appetite for using the data to inform program improvement. Ideally, programs will be able to determine whether parents and children in the same family are being served.

• Have stable and committed leadership.

The project team will involve up to five of the programs in the NS2G formative evaluations and TA, based on a deeper exploration of their services and needs. To conduct this deeper exploration, the project may reach out to fewer than 9 individuals to assess their interest in participating.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Mathematica designed the Program Confirmation Call Protocol (Instrument 1), the Site Visit Topic Guide (Instrument 2), the Participant Focus Group Protocol (Instrument 3), the Program Staff Survey (Instrument 4) and the formative evaluation feedback survey (Instrument 5) specifically for the NS2G project.

• Instruments 1 and 2 are informed by implementation science frameworks such as the Consolidated Framework for Implementation Research and the Active Implementation frameworks developed by the National Implementation Research Network (NIRN). These instruments contain semi-structured interview questions and topics that are designed to gather information about key components thought to contribute to strong implementation of evidence-based programs and practices, in order to document how two-generation programs are designed, implemented, and staffed.

• Instrument 2 includes a series of interactive, design thinking exercises intended to engage program staff in identifying program challenges and brainstorming creative solutions. These activities are informed by human centered design approaches developed by the LUMA Institute.

• Instrument 3 is intended to gather program participants’ experiences enrolling and participating in two-generation program services, and was informed by an instrument developed under and used in the Integrated Approaches study to collect information about the experiences of two-generation program participants during field work (Sama-Miller et al. 2017).

• Instrument 4 is intended to collect rapid feedback about program staff members’ use of a strategy, process, or tool, and was informed by instruments used on a number of other ACF projects that have used the LI² framework to conduct formative rapid-cycle evaluations (for example, McCay et al. 2017).

• Instrument 5 was informed by research on how to measure the effectiveness of TA (Rachidi 2019).

The project team reviewed the instruments to ensure that they ask only questions necessary to achieve the objectives of the information collection. All instruments were created specifically for the project, do not include any scales or items that measure constructs, and do not require psychometric testing.

B4. Collection of Data and Quality Control

Two Mathematica NS2G project team members (“TA liaisons”) will be assigned to work with each program; these project team members will collect the data. Data quality and consistency will be ensured by assigning team members with extensive experience conducting interviews and surveys, facilitating focus groups, and providing additional training on the relevant instruments.

Each site’s formative evaluation will begin with a program confirmation call (Instrument 1). In this call, two NS2G project team members will ask guided questions to confirm our understanding of each program’s services and capacity to participate in the formative evaluation and TA, including their data systems, the data they collect, how they use data, their past experiences with research and evaluation, and staffing. Using this information, Mathematica and the programs will co-develop individualized formative evaluation and TA plans that specify the types of activities that programs will engage in during their formative evaluations.

Following the program confirmation call and development of the formative evaluation and TA plan, members of the NS2G project team will conduct a two-day initial site visit. Day 1 of the site visit is part of the “Learn” phase, where we will work with the program and its partners to develop a comprehensive, shared understanding of their services and service challenges and deepen our understanding of the site’s TA needs and stage of development. TA liaisons will interview program staff partners to learn about the program’s community context, vision and goals, service delivery, staffing, and data management practices (Instrument 2). The TA liaisons will also conduct a focus group with program participants to collect feedback on the participants’ experiences in the program (Instrument 3).

On Day 2, the TA liaisons and program staff will co-design new strategies to address challenges in service delivery, such as improving the integration of services for parents and their children. First, to wrap up the “Learn” phase, TA liaisons will use human-centered design activities to help program staff develop a common understanding of their program’s intended outcomes, challenges, strengths, and opportunities, and articulate or refine a program logic model. In the second part of Day 2, staff will begin developing creative solutions to the implementation challenges they identified earlier (“Innovate”). At the end of Day 2, TA liaisons will ask participating staff to rate the design-thinking session by responding to a few short questions on a feedback card. TA liaisons will continue the “Innovate” phase during regular program TA calls using the lessons from the site visit to inform the rest of the formative evaluation and TA.

Once a program has developed a strategy to improve an aspect of its programming, TA liaisons will ask the program will complete a formative RCE to test its approach. The program staff survey (Instrument 4) will be used to pilot test a strategy to help the program “Improve.” In the formative RCE, the program will complete up to three sequential learning cycles. In each learning cycle, a small number of program staff (up to 10 staff per program) will pilot a strategy, tool, or process for a short period of time (up to eight weeks). For example, a program may develop and pilot a new process for sharing information about families between adult- and child-focused frontline staff. They will provide feedback through a 10-minute web-based survey administered about once per week. At the end of each learning cycle, TA liaisons will work with the programs to interpret data from the program staff survey to understand whether and how the approach was successful and what the program could do differently. Then, TA liaisons and the program will collaboratively refine the strategy, tool, or process and will begin another short learning cycle.

About two years after the start of their participation in the study, the sites will share what they learned from their formative evaluations and TA and develop action plans to continue their work through the end of the project.

Staff from the two-generation programs who are responsible for championing and leading the formative evaluation and TA work in their respective programs, will respond twice to a short feedback survey about the process of participating in the formative evaluations (Instrument 5). NS2G will administer the survey once at about the midpoint of the formative evaluations, in late 2021 or early 2022, and once at the end of the study, in early 2023. NS2G will review the feedback to understand whether and how program staff felt that their organization increased its capacity and functioning as a result of participating in the study.

While the plans described above include in-person activities (Instruments 2 and 3), we anticipate that TA liaisons may have to conduct activities virtually, using videoconferencing. The instruments have been designed for either in-person or virtual data collection.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The data collection is not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. The data collection will document examples of programs that provide services to parents, children, and families. Because data collection is not designed to be representative, response rates will not be reported.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, nonresponse bias will not be calculated.

B6. Production of Estimates and Projections

We will not produce estimates or weights for this project.

B7. Data Handling and Analysis

Data Handling

Qualitative data will be collected through note taking and photography during the initial site visit (Instruments 2 and 3). The survey data (Instruments 4 and 5) will be self-reported by program staff members. We will use audio recordings of interviews and focus groups (Instruments 2 and 3) to verify notes. As a part of the formative rapid-cycle evaluations, we will review summary information from Instrument 4 with program staff to confirm whether it aligns with their experiences and recollections of pilot-testing strategies. We will also review response rates for Instruments 4 and 5 to assess how much emphasis to put on the survey results for refining strategies to address implementation challenges.

Data Analysis

This project will not employ any complex analytical techniques. To analyze the data collected through interviews during the initial site visit, we will use standard qualitative analysis techniques such as thematic identification; the primary purpose of the site visit is to inform future TA. For the formative rapid-cycle evaluation staff survey, we will conduct standard qualitative analysis of responses to open-ended items and calculate ranges and averages for the remaining questions.

Data Use

This study is intended to provide descriptions of two-generation programs’ capacity building and improvements to their readiness for summative evaluations of their effectiveness. When necessary, results will be labeled as examples. NS2G will develop a summative report and up to three briefs about the formative evaluations and TA. Although the information collected under this clearance will not be the primary subject of any published ACF reports, the information may be included in published products to share insights from the formative evaluations and TA. Topics could include: the process of capacity building, the development of tools used in strengthening two-generation program models, descriptions of program models and the activities that programs engaged in through formative evaluations and TA, or recommendations for next steps in strengthening two-generation programs for future evaluations of effectiveness. The products will not share quantitative findings nor hard data about the programs. In sharing findings, we will describe the study methods and limitations to generalizability and as a basis for policy.

B8. Contact Person(s)

Emily Sama-Miller

Senior Researcher

Mathematica

1100 First Street NE

Washington, DC 20002

esamamiller@mathematica-mpr.com

202-484-4512

References

McCay, Jonathan, Michelle Derr, and Ann Person. “Using a ‘Road Test’ to Improve Human Services Programs.” OPRE Report #2017-107. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Rachidi, Angela. “Is Your Training and Technical Assistance (T/TA) Effective? Considerations for Measuring Performance.” Washington, D.C.: Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, 2019. Available at: https://aspe.hhs.gov/system/files/aspe-files/261886/measuring-t-ta-considerations-toolkit.pdf

Sama-Miller, Emily, Christine Ross, Teresa Eckrich Sommer, Scott Baumgartner, Lily Roberts, and P. Lindsay Chase-Lansdale. “Exploration of Integrated Approaches to Supporting Child Development and Improving Family Economic Security.” OPRE Report # 2017-84. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services. Available at: acf.hhs.gov/sites/default/files/opre/two_gen_final_report_final_clean_b508.pdf

Sama-Miller, Emily and Scott Baumgartner. Achieving Family Economic Security and Promoting Child Well-Being: Features of Contemporary Programs, OPRE Report #2017-49, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Aspen Institute. “National Network/2Gen Learning Community.” Available at https://ascend.aspeninstitute.org/two-generation/national-network2gen-learning-community. Accessed February 18, 2020.

Attachments

Instrument 1: Program Confirmation Protocol

Instrument 2: Site Visit Topic Guide

Instrument 3: Participant Focus Group Protocol

Instrument 4: Program Staff Survey

Instrument 5: Formative Evaluation Feedback Survey

8