PACT Appendix A - Description Eval Strategies

Parents and Children Together (PACT) Evaluation

APPENDIX A

Description of Evaluation Strategies

Description of Evaluation Strategies

The PACT Evaluation has three major, interrelated evaluation strategies: (a) experimental impact; (b) implementation; and (c) qualitative evaluations. Two types of multi-component evaluations will be undertaken:

• impact evaluations, complemented with implementation and qualitative evaluations, will be conducted in a subset of grantees to provide rigorous estimates of the effectiveness of the studied programs and information about their operating contexts; and

• implementation and qualitative evaluations (without impact evaluations) will be conducted in a separate subset of grantees which present some particular feature of program design or target population that warrants detailed study, but which would not support an impact evaluation (for example, if power analyses indicate that sample size is inadequate).

For ease of communication, these two types of evaluations are called “impact” and “implementation/qualitative only,” respectively. Following is a description of the three evaluation strategies to be undertaken in the two multi-component evaluations.

On April 20, 2012, an ICR was approved to hold discussions with RFHM grantees (0970-0403), in order to select grantees for the evaluations. On October 31, 2012, an ICR was approved for a RF baseline survey (as well as an introductory script to the baseline survey) and a RF study MIS.

Impact evaluation strategy. A primary goal of this component is to provide rigorous estimates of the effectiveness of the programs selected for the evaluation. It will address: (1) whether Responsible Fatherhood and Healthy Marriage (RFHM) programs are effective at improving multiple family-related outcomes of participants and their families; (2) whether program effectiveness varies by population served; and (3) whether effectiveness varies by program type (e.g., structure, services provided, how services are implemented, or community context).

This component will use an experimental design. Program applicants who are interested in and eligible for an RFHM program and who consent to be in the study will be randomly assigned to either a program group which will be offered participation in the program or a control group which will not be offered participation in the program. We expect that in most Responsible Fatherhood sites that the unit of random assignment will be individuals (fathers) rather than couples. In most Healthy Marriage sites the unit of random assignment will be couples rather than individuals. In all cases the unit of assignment will be the unit of analysis.¹

The plan is for the grantees to begin random assignment in 2013 and continue random assignment for about two years. About 400 fathers or couples are expected to be randomly assigned in each grantee program that is part of the impact evaluation.

Information will be collected twice for the impact component. First, a baseline survey will be conducted of fathers or members of couples, as appropriate, in both the program and control groups prior to random assignment. Second, follow-up data will be collected from sample members about 12 months after enrollment in the program. These follow-up data may include a survey of fathers or members of couples, as appropriate, in both the program and control groups; participant-completed diaries; observations in the study participant’s home; and administrative data (e.g., National Directory of New Hires).

Implementation evaluation strategy. The goal of the implementation study is to provide a detailed description of the grantee activities—how they are implemented, their participants, the contexts in which they are operated, and their operating practices. The detailed descriptions will assist in interpreting program impacts and identifying program features and conditions necessary for effective program replication or improvement.

Data on the programs will be collected from four main sources:

1. Site visits. During two rounds of multi-day site visits, data collection will involve interviews with managers and staff of the grantees and partner organizations, focus groups of participants, observations of program and staff activities, and reviews of participant case records.

2. On-line surveys. Program staff will be asked to complete an on-line survey to obtain more systematic and potentially more candid information than can be gained through interviews. An additional brief inventory (Working Alliance Inventory) will measure the quality of the working relationship between a program staff person (e.g., a case manager) and his/her client (participants) as reported by each party and will complement other implementation and qualitative information.

3. Telephone interviews with program dropouts. As engagement in the program is often a challenge for RFHM programs, interviews will be conducted with about 15 people at each study grantee who enrolled in a program but participated little or not at all.

4. Partner organization survey. A web-based survey of about 5 partner organizations in each site will occur at the beginning of the intake period and again near the end. This survey will describe the organizations that provide services to the population, details on services available, and the partnerships between the organizations.

5. Study Management Information System (MIS). Program staff will be asked to record information on the services offered and provided to study participants in a study MIS.

Qualitative evaluation strategy. The goal of the qualitative component is to obtain a richer and more nuanced understanding of the factors that influence the experiences, choices, and behavior of fathers or couples who are (or could be) served by RFHM grantees. While the impact and implementation strategies are focused on the RFHM grantee activities and services, the qualitative studies are more oriented to exploring how individual beliefs and experiences connect to and are affected by the communities in which participants live, their family relationships, and the programs and organizations that touch their lives.

At this point, only qualitative data collection is expected to be carried out in RF sites. Data collection for this component is expected to include:

1. In-depth, in-person conversations with program participants. At each site, up to 15 individuals in the program will be followed closely throughout the study. Interviews with these individuals, which will occur in the respondents’ homes if possible, will be guided by a semi-structured protocol, and will explore relationships with children and current and former partners, community involvement, and program experiences. These interviews will occur annually and be about two hours in length.

2. Check-in calls. Four check-in telephone calls will be made to the 15 program participants in each site who participate in the in-depth, in-person conversations.

1 However, this will be customized to reflect the target population (couples or individuals) of a specific site—some Responsible Fatherhood programs serve couples and some Healthy Marriage programs serve individuals.