Appendix E - FaMLE Implementation Analysis Plan Instructions

APPENDIX E

Instructions for Instrument 2:

Implementation Analysis Plan Template for HMRF Grantees

the importance of having an implementation analysis plan

The Administration for Children and Families (ACF), Office of Family Assistance (OFA) is requiring that all Healthy Marriage and Responsible Fatherhood (HMRF) grantees with local impact evaluations funded by OFA provide an implementation analysis plan for their evaluations, in addition to an impact evaluation plan (see additional template and instructions). In any rigorous impact evaluation, it is important to tell a clear, succinct story about program implementation. The implementation findings (1) describe what was offered to and received by people in both the treatment and control/comparison conditions, (2) contextualize the impact findings, and (3) generate hypotheses about why the program did or did not have a positive impact. Developing a structured implementation analysis plan before examining the implementation data will foster an efficient and effective approach for analyzing the data and reporting the findings.

Please complete this implementation analysis plan in addition to the impact analysis plan. This document provides instructions for completing the implementation analysis plan. Grantees must provide information on all required sections. Please do not use the format of the instructions presented below to complete your implementation analysis plan. Instead, please use the provided template (FaMLE_Implementation_Analysis Plan_Template).

Please email your implementation analysis plan (together with your impact evaluation analysis plan) to your Federal Program Specialist (FPS) and copy your Evaluation Team Technical Assistance (ETTA) liaison by [date]. For consistency, please use this common naming convention when submitting your implementation analysis plan:

[Grantee Name] Implementation Analysis Plan

Your FPS and ETTA liaison will review the analysis plan and provide comments and suggested edits and return it to you for revisions. Your analysis plan must be revised and approved by your FPS by [date].

Instructions for completing the implementation analysis plan template

For this implementation analysis plan, please describe the research questions you aim to answer about the implementation of the intervention, the data you will use to answer the research questions, and the methods you will use to analyze the implementation data and describe the findings. To the extent possible, please use tables to succinctly summarize the required implementation information.

The focus of this implementation analysis plan is to measure the intervention services received by the intervention group and the alternative services and services similar to the intervention that were received by the control/comparison group during the evaluation period (from study enrollment through the final follow-up interview), based on the data you have collected through surveys and nFORM. The analysis plan should focus on the services received by evaluation participants, which may be a subset of all services provided and populations served under the HMRF grant. Please discuss any questions about the focus of the implementation study with your ETTA liaison and FPS.

1. Research questions

The research questions articulate the main hypotheses of your implementation study. Research questions can refer to implementation elements such as fidelity, dosage, quality of implementation, engagement, and context, as the following examples show:

• Fidelity: Was the intervention (and each of its components, if applicable) delivered as intended?

• Dosage: How much of the programming (or components, if applicable) did the intervention group members receive?

• Quality: How well was the intervention implemented or delivered to intervention group members?

• Engagement: Did individuals, couples, and/or fathers in the intervention group engage in intervention services, and if so, how engaged were they?

• Context: What other types of services are available to intervention group members? What external events or unplanned adaptations may have affected implementation of the intervention as intended?

Table 1 provides examples (in italics) of research questions for each implementation element.

Table 1. Examples of research questions for each implementation element and study group

Implementation element	Research question
Intervention Group Questions
Fidelity	Were all intended intervention components offered and for the expected duration? What content did the intervention group receive? Who delivered services to the intervention group members? What were the unplanned adaptations to key intervention components?
Dosage	How often did the intervention group participate in the intervention on average?
Quality	What was the quality of staff–participant interactions?
Engagement	How engaged were intervention group members in the intervention?
Context	What other HM/RF programming was available to intervention group members? What external events affected implementation?
Control / Comparison Group Questions
Fidelity	If control/comparison group members received alternative services during the evaluation period: Were all intended service components offered to the control/comparison group and for the expected duration? What content did the control/comparison group receive from any sources during the evaluation period? Who delivered any services similar to the intervention to the control/comparison group members during the evaluation period? What were the unplanned adaptations to control/comparison group components offered as alternative services during the evaluation?
Dosage	How often did the control/comparison group members participate in alternative services or in services similar to the intervention from other sources, on average?
Quality	What was the quality of staff–participant interactions in alternative services or in services similar to the intervention received from other sources, if known?
Engagement	How engaged were control/comparison group members in the alternative services, or in services similar to the intervention received from other sources, if known?
Context	What other HM/RF programming was available to control/comparison group members?

2. Data sources

Describe the data sources you will use to answer the research questions (for example, nFORM, fidelity protocols, attendance logs). Describe the timing and frequency of each data collection effort (for example, during all sessions, once a week, annually), and the party responsible for collecting the data. Use a table to link the information on data collection to the research questions. Table 2 below presents an example of such table (sample text included in italics).

Table 2. Examples of data for addressing the research questions

Implementation element	Research question	Data source	Timing/frequency of data collection	Party responsible for data collection
Fidelity	Were all intended intervention components offered and for the expected duration?	Workshop sessions in nFORM	All sessions delivered	Intervention staff
Fidelity	What content did the clients receive?	Fidelity tracking log or protocol; attendance logs; session observations	Every session for fidelity tracking and attendance logs; two times a year for session observations	Intervention staff for fidelity tracking and attendance logs; study staff for session observations
Fidelity	Who delivered services to clients?	Staff applications; hiring records; training logs	One time X months after start of implementation; annually	Intervention staff
Fidelity	What were the unplanned adaptations to key intervention components?	Adaptation request; work plan; 6-month progress report; annual progress report	Annually; ad hoc	Intervention staff; study staff
Dosage	How often did clients participate in the intervention on average?	Workshop sessions and individual service contacts in nFORM; attendance logs	All sessions delivered	Intervention staff
Quality	What was the quality of staff–participant interactions?	Observations of interaction quality, using protocol developed by study staff	X percentage of sessions selected at random for observation	Study staff
Engagement	How engaged were clients in the intervention?	Observations of engagement, possibly using an engagement assessment tool; ratings from facilitator fidelity logs; engagement ratings from participant satisfaction surveys	Y percentage of sessions selected at random for observation	Study staff
Context	What other HM/RF programming was available to study participants?	Interviews with staff from partnering agencies in the community; survey items on baseline and follow-up assessments; websites of other agencies in the community providing HM/RF programming	Once a year; ad hoc	Study staff
Context	What external events affected implementation?	Interviews with community/county representatives; list of site/school closures	Once a year; ad hoc	Study staff

Note: We use “clients” in the questions above to avoid redundancy. These questions can be adapted to address the research questions posed in Table 1 about intervention group members and control/comparison group members separately.

3. Analysis approach

Describe the specific measures you will construct and your approaches to using those measures to answer the research questions. For example, describe whether you will calculate averages, percentages, and frequencies using the data you will collect for the implementation study. In addition, include information on your approaches to examine and summarize qualitative data from interviews, focus groups, and observations. Use a table to link the description of the measures to the research questions. Table 3 below presents an example of such table (sample text included in italics).

Table 3. Examples of measures for addressing the research questions

Implementation element	Research question	Measures
Fidelity	Were all intended intervention components offered and for the expected duration?	Total number of sessions delivered Average session duration, calculated as the average of the recorded session lengths (in minutes)
Fidelity	What content did the clients receive?	Total number of topics covered, calculated as the average of the total number of topics checked by each intervention facilitator in the daily fidelity tracking log or protocol Number of HMRF topics covered by other providers during the evaluation period, based on survey data
Fidelity	Who delivered services to clients?	Number and type of staff delivering services to study participants, such as the number of session facilitators and couples’ therapists Percentage of staff trained, calculated as the number of staff who were trained divided by the total number of staff who delivered the intervention
Fidelity	What were the unplanned adaptations to key intervention components?	List of unplanned adaptations, such as a change in setting, sessions added or deleted, and components cut
Dosage	How often did clients participate in the intervention on average?	Average number (or percentage) of sessions clients attended Percentage of the sample attending the required or recommended proportion of sessions Percentage of the sample that did not attend sessions at all Participation in services similar to those offered by the HM/RF program from other sources, and number of hours received, based on survey data
Quality	What was the quality of staff–participant interactions?	Percentage of sessions with high quality interactions, calculated as the percentage of observed interactions that study staff scored as “high quality”
Engagement	How engaged were clients in the intervention?	Percentage of sessions with moderate participant engagement, calculated as the percentage of sessions in which study staff scored participants’ engagement as “moderately engaged” or higher Average engagement rating, calculated as the average of engagement scale scores (ranging from 1–5, for example) across satisfaction surveys Reports of level of engagement in the intervention or in similar HMRF services, based on survey data
Context	What other HM/RF programming was available to study participants?	Percentage of the sample receiving HM/RF programming from other providers, constructed from clients’ survey data on experiences outside of the current intervention List of HM/RF programming available to study participants outside of the current intervention, as described on the websites from other agencies in the community
Context	What external events affected implementation?	Percentage and total number of anticipated study participants missed due to community issues, if any Number of sites/schools that were closed as a result of weather events or policy changes (unrelated to the HM/RF programming), if any

Note: Please adapt the questions above to measure the intervention services received by the intervention group and alternative services and services similar to the intervention that were received by the control/comparison group during the evaluation period (enrollment through the final follow-up interview), based on the data you have collected through surveys and nFORM.