Appendix D - FaMLE Impact Analysis Plan Instructions

APPENDIX D

Instructions for Instrument 1:

Impact Evaluation Analysis Plan Template for HMRF Grantees

The Importance of Having an impact Analysis Plan

The Administration for Children and Families (ACF), Office of Family Assistance (OFA) is requiring that all Healthy Marriage and Responsible Fatherhood (HMRF) grantees with local impact evaluations funded by OFA provide an analysis plan for their evaluations. An impact evaluation tests the effect of an intervention, or a component of an intervention, by comparing the outcomes of people randomly assigned to be offered the intervention with those randomly assigned to not be offered the intervention.^1,2 Developing an analysis plan before conducting any analysis is a way to demonstrate a team’s commitment to being objective with a pre-specified, systematic, and scientific approach. The analysis plan is a document that shows a funder, the program staff, and possibly future skeptics that the team pre-selected outcomes and analytic approaches to gauge program effectiveness.

Below are instructions for completing the analysis plan for an impact evaluation. Grantees must provide information on all required sections. Please use the provided template (FaMLE_Impact_Analysis Plan_Template) for this analysis plan. In addition, impact grantees must complete an implementation analysis plan. Instructions and a template for the implementation analysis plan are in accompanying documents. In addition to requiring completion of an analysis plan, ACF strongly encourages grantees to require their evaluators to share this analysis plan internally among their team, and perhaps even with grantee staff, so that everyone understands the plan and has an opportunity to discuss key decisions. The analysis plan can be considered an agreement between grantees and their evaluators on two key aspects of it: (1) the outcomes the evaluation will examine and (2) the approaches the evaluation team will use to assess program effectiveness on those outcomes.

The instructions presented below are organized as follows:

• Sections 1 to 3 describe the proposed research questions, program design and the counterfactual, and the study design. Explaining the intervention and evaluation design are critical to ensuring constancy of evaluation approach and to documenting any changes that may have occurred during the implementation of the intervention and evaluation.

• Sections 4 and 5 provide the blueprint for the benchmark and sensitivity analyses that the team will use to examine intervention effectiveness, and the exploratory analyses of secondary research questions.

These instructions have been created so that all grantees can fill out each section of the analysis plan regardless of the specifics of their evaluation, but some grantees may need to adapt some sub-sections to fit their design. In addition to documenting your approach to the impact analysis, this document is designed to assist you in detailing information that you can use in final evaluation reports or other dissemination products.

Under the direction of ACF, your Evaluation Team Technical Assistance (ETTA) liaison will review your analysis plan to provide input and support as you draft it. Please email your analysis plan to your Federal Program Specialist (FPS) and copy your ETTA liaison by [date]. For consistency, please use this common naming convention when submitting your analysis plan:

[Grantee Name] Impact Evaluation Analysis Plan.

Your FPS and ETTA liaison will review the analysis plan and provide comments and suggested edits and return it to you for revisions. Your analysis plan must be revised and approved by your FPS by [date].

Instructions for Completing the impact evaluation plan Template

ACF expects that evaluators will complete the analysis plans, with input from program directors and/or program staff as appropriate. For that reason, these instructions are mainly directed to evaluators and include a few technical terms.

1. Research questions that address intervention effectiveness on outcomes

a. Primary research question(s)

The primary research questions for HMRF impact studies focus on gauging an intervention’s effectiveness in improving healthy relationship and responsible fatherhood outcomes. Such outcomes may include the status and quality of the couples’ relationships, the quality of co-parenting or parenting, and economic stability and well-being. Outcomes might be measured in a variety of ways, such as surveys, direct assessments, and observations. A best practice is to focus each primary research question on how the intervention affects a specific outcome measure at a specific time point (for example, upon completion of the intervention, or one year after completing the intervention). This approach will clearly connect the outcome(s) and the time point(s) to the intervention’s logic model for the theory of change. An example of this practice is the question: “What is the impact of [intervention] relative to [counterfactual] on the support and affection that couples feel toward each other one year after the end of the intervention?” This approach can be followed for the outcomes and time periods of greatest importance to the local evaluation.

Because the likelihood of a false positive—that is, estimating a statistically significant impact when no causal effect actually exists—increases with the number of outcomes studied, another best practice is to limit the set of primary research questions. By setting priorities across the outcomes and time points that are most essential for conducting confirmatory tests of intervention effectiveness that align with the intervention objectives and logic model, the number of primary research questions can be reduced to the most essential ones.

Some impact research questions that are not critical as confirmatory tests for evaluating an intervention’s effectiveness in improving healthy relationships and responsible fatherhood outcomes are still important and of interest to the grantee, stakeholders, and researchers. These are generally considered to be secondary research questions, which are described next.

b. Secondary research questions

Secondary research questions help grantees examine other (non-primary) outcomes the intervention might influence, which are considered exploratory analyses. For example, secondary research questions could focus on the following:

1. Examinations of the outcomes specified in the primary research questions but at time points different from the one specified in the primary research questions (such as immediately after the end of the intervention).

2. Examinations of other outcomes different from those specified in the primary research questions (for example, precursors to the healthy relationships and responsible fatherhood outcomes of primary interest).

3. Examinations of the relationships among mediating variables and outcomes, such as the relationship between dosage or participation and outcomes.

2. Description of the intended intervention and counterfactual conditions

a. Intervention condition(s)

Describe the intended experiences of those in the intervention condition(s) (that is, what the intervention aimed to offer them). In particular, describe the following:

1. Intended components: Describe all of the key structural elements of the intervention (for example, group classes, workshops, one-on-one services) that the members of the intervention group(s) received and the comparison groups did not. If this is an intervention that includes multiple components, describe all of them. For example, “This is a multi-component intervention in which parenting couples receive classes in relationship skills, workshops on economic stability topics, case management, and booster sessions.” If the intervention consists of adding services to a particular program, describe the program and all the additional services that will be provided as part of the intervention. If the intervention consists of providing a number of services not related to a curriculum or program (for example, case management, counseling, home visits), describe each of the services.

2. Intended content: Provide the name of the curriculum used (if any) and describe the topics that the intervention covers and the resources/materials provided.

3. Target population: Describe the target population, that is, provide information on the characteristics of the population that the intervention intends to serve, such as age, gender, marital status, and socio-economic status. For example, “The intervention is intended to be delivered to low-income unmarried fathers and their partners.”

4. Planned dosage and implementation schedule: Describe the number of sessions and the duration of the intervention. Include the length of each session and how frequently they occur. For example, “This is an eight-month intervention, with sessions occurring once a week for two hours per session.” Describe variation in the frequency or length of sessions across sites, if applicable.

5. Intended delivery: Describe where the intervention takes place and who delivers the intervention. Include information on the required education, gender, cultural background, and hiring requirements of the intervention providers/facilitator, and on the training and technical assistance those providers will receive as part of this study.

Tables can be used to clearly and succinctly summarize intervention components. See Table 1 for an example (sample text included in italics). If there are multiple intervention conditions, separate tables can summarize each condition.

Table 1. Intended intervention components

Component	Curriculum and content	Dosage and schedule	Delivery
Relationship skills workshops	Healthy relationships curriculum: Understanding partner’s perspectives; avoiding destructive conflict; and communicating effectively	20 hours, with 2-hour sessions occurring twice a week, or 4-hour sessions occurring every Saturday	Group lessons provided at the intervention’s facilities by two trained facilitators in every session
Economic stability workshops	Resume preparation; interview and communication skills; appropriate work attire; financial literacy	Monthly 2-hour workshops	Workshops are provided by one facilitator

Note: The intervention intends to serve low-income married couples. Male and female facilitators at the X location will deliver the intervention. Facilitators hold varying educational degrees, including bachelor’s and master’s degrees, and receive ongoing training in the intervention’s curriculum from study staff.

b. Counterfactual condition

Describe the intended experiences of those in the control/comparison group, that is, those in the counterfactual condition. If the grantee (or partner) is not providing services to people assigned to the control/comparison group, describe any “business-as-usual” resources available to this group outside of the study. If the control/comparison group is receiving an alternative intervention, describe the following:

1. Intended components: Group classes, workshops, one-on-one services, and the like

2. Intended dosage: Total intended dosage, number of sessions and their length, frequency of sessions/services, time period during which services take place

3. Intended content: Curriculum, topics it will cover, and resources/materials participants will receive

4. Intended delivery: The setting where the alternative intervention will take place and who delivers it, the intended characteristics of the alternative intervention providers, and the training and technical assistance providers will receive

If the control group received a delayed intervention (also called a wait-list control design), describe any services or interactions with intervention staff that the control group received during the evaluation period while they were on the waitlist. You do not need to describe the delayed intervention this group received, because it occurred after all evaluation-related data collection ended.

c. Services actually received by the intervention and control/comparison groups

After describing the intended intervention and control/comparison group services, the analysis should provide information on what was actually received by people in the two study groups. Describe plans for measuring the services actually received by each group by following the instructions in the Implementation Analysis Plan document and template.

3. Study design

a. Sample formation

Describe the ways members of the target population become part of the impact study sample. Provide the information for the full sample (both intervention and control/comparison groups), including agencies and schools from which they were recruited, all service locations/sites, and any eligibility requirements for sample members. Include information on the following:

1. Sample eligibility criteria: Describe any required characteristics for sample inclusion (for example, age, marital status, involvement with the child support system, attending a particular school, geographical area, employment status).

2. Purposeful sampling: Describe any additional criteria for selecting the sample beyond the eligibility criteria, for example, if the sample is composed of a random selection of eligible participants, or if the sample includes only specific classrooms in eligible, participating schools.

(If a random assignment design) Random assignment process

Describe the following about the random assignment process:

1. What is the unit of randomization (for example, individual clients, couples, agencies, schools)?

2. Who randomly assigns units to the conditions (treatment or control), and when, how, and under what circumstances does this occur?

1. Do evaluation staff or intervention staff conduct the process to randomly assign units to conditions?

2. When does random assignment occur with respect to the timing of consent and baseline data collection? (If using nFORM data, this could include administering the applicant characteristics survey.) For clustered randomized controlled trials (RCTs), who, if anyone, learned of the outcomes of random assignment before consent and baseline data were collected, and for what purposes?

3. What is the method of random assignment (such as random number generation in Excel)?

4. Does randomization occur all at once (that is, the study randomly assigns a large number of units at a single point in time) or on a rolling basis (that is, the study randomly assigns small numbers of units at different points in time)? Describe the details of this process.

3. Describe any stratification/blocking you used to create separate instances of random assignment in the evaluation. For example, you might randomly assign people to a condition separately across service locations, such as schools; in this situation, the service locations are strata/blocks.

4. Report the intended probability of assignment to the treatment group. If it varies systematically (for example, across blocks/strata), report why and give the range of probabilities used.

5. If applicable, describe any sub-sampling that occurred after random assignment, the reason for doing so, the criteria used, and the manner in which you operationalized the sub-sampling.

(If a quasi-experimental design): Research group formation

Describe the process you will use for constructing the treatment and comparison groups, including whether you assigned clients or groups of clients to the treatment or comparison group. Specify when this assignment procedure occurred, relative to the timing of obtaining consent and collecting baseline data.

If you constructed the comparison group from an administrative data set, describe the source of the data and the criteria for identifying people similar to the clients in the treatment group, including characteristics and variables used to create comparable groups. Describe any services people in this group have received that are similar to the treatment services.

Consent process

Provide the name of the Institutional Review Board that approved the study design and data collection plans, and the date of approval (and the dates of any supplemental review approvals).

Describe, in detail, the consent process for both the treatment and control/comparison groups (and, if your study involved underage youth, the process for adult consent and youth assent). Include descriptions of similarities and differences between groups with respect to timing, process, and materials used (such as consent forms, incentives, and so on). If you randomly assigned people before the consent process, note whether you informed potential sample members of their condition before or after receiving their consent.

b. Data collection

Describe the data sources for the analyses. Describe the timing of each data collection point (for example, baseline and the follow-up periods used for primary and secondary research questions). Describe the modes and methods of collecting data at each data collection point (for example, in-person paper survey, online survey, the party responsible for collecting data at each time point and for each study condition). Thoroughly describe the process and the timing for data collection, by study condition (treatment and control/comparison). Clearly articulate similarities and differences across study conditions. Please use a table to clearly and succinctly summarize features of the data collection procedures for each study group (see Table 2 for an example; sample text included in italics). Finally, please provide a copy of your data collection instruments in an appendix to your analysis plan (at a minimum, provide the instruments that you will use to collect the outcome data that you will use to answer the primary research questions of your study).

Table 2. Key features of the data collection

Data source	Timing of data collection	Mode of data collection	Party responsible for data collection	Start and end date of data collection
Intervention participants	Enrollment (baseline) and end of intervention (8 months after enrollment)	In-person online survey	Evaluation staff	September 2016 through March 2021

c. CONSORT diagram

ACF requires that analysis plans include a CONSORT diagram. A CONSORT diagram is a flow chart that summarizes the number of clients in the study from initial enrollment through the final data collection point, separately for treatment and control/comparison groups. The CONSORT diagram serves two purposes. The first is to assess the estimated sample size and compare it to the target sample size that was the basis of the study’s power calculations. The second purpose is to assess the likelihood that the final analytic sample for key follow-up time period(s) would have high rates of overall or differential attrition.

The CONSORT diagram you will include in your analysis plan will be an interim version that will include the most current information about your sample. At a future date, you will need to revise the interim CONSORT diagram to create a final version with details on the final sample. Your final evaluation report will include this final version of the CONSORT diagram.

See the Appendix in these instructions for templates of the CONSORT diagrams. These templates are applicable to studies in which consent to study participation occurred (1) before assignment to treatment or comparison conditions (Figures A.1 and A.2), or (2) after assignment (Figures A.3 and A.4).

To complete the interim CONSORT diagram(s), use the most recent evaluation data available. Indicate the date through which you enrolled the sample and the date through which you collected and included survey data that are represented in the counts in this diagram. Keep in mind that participants may be at different stages of the study at any point in time due to rolling enrollment or multiple cohorts of implementation, so some participants might not yet have data at all data collection points.

1. CONSORT diagrams that track clusters as the unit of assignment (if applicable). We require specific information for cluster random assignment and quasi-experimental designs (that is, studies that involve assignment [random in RCTs and non-random in quasi-experimental designs] of service locations, community-based organizations, groups of clients, schools). See the following list for the required information (also see the Appendix for diagram templates, Figure A.1 for studies in which consent to participation happened before assignment to condition, and Figure A.3 for studies in which consent happened after assignment to condition):

   1. The date you are completing the CONSORT template; this indicates the time point the information reflects.

   2. If not yet described in Sections 3.b or 3.c above (as applicable), please include a brief paragraph (3 or 4 sentences) describing how you define clusters (for example, a group of clients or classroom of students who attended enrollment sessions and consented to participate in the evaluation). Use this paragraph to also describe what makes a cluster eligible for the evaluation, the number of clusters screened, the number of clusters determined to be eligible and the counts and reasons for those screened out, and whether and how you prioritized any clusters for inclusion in the study sample.

   3. The number of clusters (randomly, if the study is an RCT) assigned in total; to each condition (that is, treatment and comparison); and the start and end dates of cluster assignment.

   4. The number of clusters still participating in the study (that is, retained), by study condition, at each data collection time point. A participating cluster is one in which at least one person in the cluster completed the data collection effort.

      1. In addition, note any reasons for clusters dropping out, and the number of clusters each reason applies to.

   5. In addition to completing a CONSORT diagram for clusters, please complete a CONSORT diagram for the individual clients in participating clusters (see instructions in the next section; templates of the diagrams for individual clients are available in Figures A.2 and A.4 in the Appendix).

      1. The primary analysis sample you note in the CONSORT diagrams for both clusters and individual clients is the sample you will use to answer the primary research questions in the final report (that is, the analytic sample) after accounting for sample loss due to attrition, missing data, and any techniques used to establish an equivalent sample at the baseline.

2. CONSORT diagrams that track individual clients. Include this diagram for both individual-level and cluster-level designs (see the Appendix for templates, Figure A.2 for studies in which consent to participate occurs before assignment to study condition, and Figure A.4 for studies in which consent to participate occurs after assignment). In addition, provide the following information:

   1. The date you are completing the CONSORT template; this indicates the time point the information reflects.

   2. If not yet described in Sections 3.b or 3.c above (as applicable), please include a brief paragraph (3 or 4 sentences) describing what makes an individual client (or couple for couple-based interventions) eligible for the evaluation. Use this paragraph to also describe the number of individual clients screened and determined to be eligible and the counts and reasons for those who were screened out. In addition, describe the process for selecting study participants among those who were eligible.

   3. The number of individual clients assigned in total (randomly, if study is an RCT); the number assigned to each condition (that is, treatment and comparison); and the start and end dates of assignment

If consent to participate in the study occurs BEFORE assignment to condition, please skip steps 4 and 5 and continue with step 6. If consent to participate in the study occurs AFTER assignment to condition, please complete steps 4 and 5.

4. [If consent to participate in the study occurs AFTER assignment to condition]: The number of individual clients in the clusters at the time of random assignment, if the study is a cluster RCT

5. [If consent to participate in the study occurs AFTER assignment to condition]: The number of individual clients who consented to participate in the study (if you obtained individual consent)

6. The number of individual clients who have provided data, by study condition, at each data collection time point (baseline and subsequent follow-ups)

   1. If the evaluation uses a cluster design, then the number of people in each condition at any time point should reflect the number of people only in participating clusters at that time point. Exclude from these counts people in clusters that have dropped entirely from the study.

   2. At a given time point, a subset of people may not have been able to contribute data for a particular data collection effort. For example, people who are receiving services and have not yet completed the intervention would not be eligible to contribute follow-up data. Therefore, it is important to document the number of people who are eligible (that is, the number of people who could have contributed data) at a given time point, in addition to the number of people who actually did provide data.

   3. The number of respondents is the number with responses to the survey questions used to measure the primary outcomes specified as your primary research questions. This may be fewer individuals than the number who responded to the survey overall.

   4. Note all reasons for nonresponse, and the number of people each reason applies to.

7. The intervention start and end dates for the study period

4. Analysis

The analysis plan for evaluating impacts should lay out the outcome measures you will use to answer your research questions and a benchmark analysis for the final report. The benchmark analysis is the approach with which you will lead, in the summary of findings. You may want to perform additional analyses that alter one or more decisions that informed the benchmark analysis to understand how results depend on these decisions. These types of subsequent analyses are known as sensitivity analyses, because they can provide information on the extent to which certain results are sensitive to particular decisions you make for the main analysis. Generally, the level of confidence for findings that are similar in the benchmark and sensitivity analyses is higher than for findings that differ in the benchmark and sensitivity analyses.

a. Outcome measures

Describe the specific outcome measures you will use to answer the primary (and secondary, if applicable) research questions. If you will construct measures from multiple items or variables, describe the survey items you will use and how you will code them to create the measure.

1. Complete Table 3 (sample text included in italics), describing all measures that you will use to answer the primary research questions assessing the impact of the intervention. Include the time periods you will use to assess impacts for these questions.

2. Complete Table 4 (sample text included in italics) for all measures that you will use to answer secondary research questions used for non-confirmatory tests of the effect of the intervention. Include the time periods you will use to assess impacts for these questions.

Table 3. Description of outcome measures used to answer impact analysis primary research questions

Outcome name	Description of the outcome measure	Source of the measure	Timing of measure
Marriage status	The outcome measure is a yes/no response taken directly from the question in the survey, “Are you currently married?”	nFORM exit survey	At post-test (immediately after intervention ends)

Table 4. Description of outcome measures used to answer impact analysis secondary research questions

Outcome name	Description of the outcome measure	Source of the measure	Timing of measure
Level of affection	The outcome measure is a scale (value range 1 to 5) calculated from both partners’ responses as the average of five survey items measuring support, intimacy, commitment, trust, and friendship.	Local follow-up survey	6 months after intervention ends

b. Data preparation

Describe how you will clean the baseline and follow-up data and prepare them for analysis. This includes whether and how you will merge or combine data from different sources. Describe in detail how you will handle missing data (such as the process to impute missing data). This brief from the U.S. Department of Health and Human Services’ Office of Adolescent Health and the May 23, 2019 webinar on non-response bias in impact evaluations provides an excellent summary of how to deal with missing data in RCTs. In addition, describe your plans to identify and handle responses that are inconsistent with each other or are seemingly inaccurate data, across both baseline and outcome (at post-test and follow-up) surveys. For example, if you are administering surveys to both members of a couple separately, describe the strategies you will use to verify that the answers are consistent, such as checking that both people report the same marital status, and what you will do if they are not consistent.

c. Analytic sample

The analytic sample is the sample you will use to estimate the impacts of the intervention. Please describe how you will define the analytic sample (for each research question, if applicable) and use the CONSORT diagram as a guide in preparing this description. Clearly describe what data you require for a person to be part of the analytic sample and refer the reader to the specific sections of the CONSORT diagram where you present the number of individual clients participating in the study who meet those data requirements. For example, indicate whether the analytic sample for the study will be individual clients with complete baseline and outcome data for all variables of interest (that is, a complete-case sample), and refer the reader to the sections of the CONSORT diagram where you present the number of individual clients who completed the baseline, completed the immediate post-intervention follow-up (and subsequent follow-ups, as applicable), and have all the required data so they are included in the primary analysis sample. Alternatively, the analytic sample might be people who have complete outcome data but some missing baseline data, which you may impute. Imputing outcome data is generally not an acceptable evaluation data analysis practice, so ACF strongly discourages including people with missing outcome data in the analytic sample for primary analyses. However, this does not mean you should exclude from your analytic sample any people who do not complete services. Include these people in your analytic sample if they have outcome data (that is, if they complete the data collection efforts), even if they do not complete services.

Assessment of baseline equivalence

Quasi-experimental studies and random assignment studies that lose part of the sample at the follow-up time periods during which they will assess intervention impacts (that is, experience attrition) must verify that the study groups (treatment and comparison groups) are equivalent at baseline, because having well-matched treatment and comparison groups can minimize the risk of bias in the impact estimates when attrition occurs.

For the purposes of this template, please describe the following:

1. The measures you will use to examine the equivalence of the study groups at baseline (for example, demographic characteristics and baseline measures of the outcomes of interest). This is a required element in the analysis; plan to assess equivalence of the study groups in baseline measures, and describe your plans in detail in the analysis plan.

2. The methods you will use to test the significance of the differences at baseline between the study groups

Condition crossover

Describe how you will quantify and report the amount of crossover that occurs during the intervention. For example, describe how you plan to use enrollment rosters, workshop attendance data, and survey data to assess whether participants assigned to each study group (treatment and comparison) are receiving services or participating in the activities meant exclusively for or are substantially similar to those assigned to the other group. In addition, describe the approach to reporting those instances. For example, explain that you plan to report the percentage of participants in the comparison group who reported receiving relationship skills lessons from any organization.

Assessment of services received by the comparison group compared to the treatment group

Describe how you will use information on the dosage of services actually received by the treatment group and the receipt of any similar services by the comparison group to assess the differences between the types and amount of similar services received by the groups, on average.

d. Analytic approach

Analytic approach for primary research questions

Describe the manner in which the analysis will answer the primary research questions under an intent-to-treat framework. With the intent-to-treat approach, all the study participants who were assigned (randomly, if study is an RCT) to the study groups (treatment and comparison) are part of the impact analysis even if they did not receive the services they were assigned to receive, and you should analyze them in the groups to which you assigned them. Provide the following information:

1. Model specification: Describe the type of model you will use to estimate intervention impacts for each primary and secondary research question (linear regression, logistic regression, etc.).

1. Specify the statistical software package you will use.

2. Define the criteria you will use to assess the statistical significance of the study findings (for example, “findings are considered statistically significant based on p < .05, two-tailed test”).

3. Describe how the model will adjust for clustering (if applicable).

4. Describe how you will incorporate the intent-to-treat approach in your analyses. That is, describe your plans to include all the study participants who were assigned (randomly, if study is an RCT) to the study groups (treatment and comparison) in the impact analysis and examine them in the groups to which you had originally assigned them. In addition, describe your plans to include all participants who provide outcome data (that is, participate in the follow-up data collection) in the impact analysis, even if they do not complete services.

2. Covariates: List all potential covariates you plan to include in the analyses in a table (see Table 5 for an example, with sample text in italics), and justify your reason for including them. Generally, analysis models include measures of the outcome at baseline and demographic characteristics as covariates, because doing so may enhance the precision of the impact estimates.

1. If you have not determined the covariates yet, describe a plan for determining those that you will include in your analyses. Aside from the baseline version of the outcome of interest, specify whether any covariates will differ across the models used to answer the primary research questions. When appropriate, describe the blocking/stratification variables—for example, county, school size, and cohort—that you will incorporate as covariates.

Table 5. Covariates included in impact analyses

Covariate	Description of the covariate
Age	Age (in years) as of the baseline data collection
Baseline marital status	Marital status (1 = married; 0 = not married) as of the baseline data collection

3. Sample attrition: Attrition is the number of people in the baseline sample for whom follow-up was not completed or who are missing outcome data. For the purposes of this template, describe the approach you will use to report overall and differential (between study groups) attrition from the initially assigned sample.

4. Sensitivity analyses: Describe any analyses that you will conduct to test the robustness of the results or the appropriateness of the analytic model for the observed data. Include analyses that change potentially important research decisions. One such example includes analyses using procedures to prepare and handle missing or inconsistent data that differ from the procedures you use in the primary analysis approach (such as alternative methods to impute missing baseline data). Another example includes analyses that adjust for alternative sets of covariates. For instance, the main analysis approach might adjust for covariates at the individual level (such as age or race/education level) and at the cluster level (such as county, school district, service location size), yet the sensitivity analyses might adjust for covariates at the individual level only.

Analytic approach for secondary research questions

Describe the analytic approach you will use to address all secondary research questions, to the extent that it differs from the analytic approach for primary research questions (for example, you might not adjust for multiple comparisons for secondary research questions). Please address Sections 4.f.i. – 4.f.v above.

5. Additional planned analyses

Researchers can explore a broad range of possible associations among outcomes and mediating factors—without increasing the likelihood of false positives among primary research questions—by differentiating the secondary (exploratory) research questions from the primary ones before analysis begins. In reporting findings, do not highlight findings from the secondary research questions, which are exploratory analyses, even when they are statistically significant. Findings from exploratory analyses are not considered impact findings.

Identify all additional research questions that you plan to address using data from this evaluation. These questions might include secondary, non-experimental analyses on mediator variables, dosage and participation, and the relationship between implementation and impacts. Describe the outcome measures and the planned analytic approaches you will use. The following are examples of research questions for exploratory analyses:

• What is the association between receiving the intervention and outcomes considered precursors to the evaluation’s primary outcomes (for example, couples’ conflict resolution skills, a precursor to relationship satisfaction)?

• What is the association between receiving the intervention and other outcomes not considered primary, intended outcomes of the intervention (such as obtaining a GED or enrolling in college)?

APPENDIX

Figure A.1. CONSORT diagram for clusters (if applicable), for studies in which consent occurred before assignment

Complete based on pooled sample to date. Also complete diagram for individual clients.

Diagram date: _________

Describe what makes a cluster eligible for the evaluation; the number of clusters screened and the screening criteria used; the number of clusters determined to be eligible; the counts and reasons for those screened out; and whether and how any clusters were prioritized for inclusion in the study sample.

Date of enrollment data __________­­­­­­­

Date of survey data __________

Did not agree to be in study (n = __)

Did not pass screening criteria (n = __)

Other (n = __)

Move “completed baseline” to correct sequence given your processes.

Completed baseline data collection (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at immediate post

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at first follow-up

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n =__)

• ___ (n = __)

Completed data collection at second follow-up (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Primary analysis sample (n = __)

List reason(s) for cluster(s) being excluded

• ___ (n = __)

• ___ (n = __)

Assigned to Comparison (n = __)

Assigned to Treatment (n = __)

Primary analysis sample (n = __)

List reason(s) for cluster(s) being excluded

• ___ (n = __)

• ___ (n = __)

Completed data collection at second follow-up (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at first follow-up

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at immediate post

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ ( n= __)

Completed baseline data collection (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Clusters randomized (n = __)

Date(s) of cluster random assignment ______

Diagram date: ___________

Figure A.2. CONSORT diagram for individual clients, for studies in which consent occurred before assignment

Complete based on pooled sample to date.

Date of enrollment data __________­­­­­­­

Date of survey data __________

Describe what makes an individual eligible for the evaluation; the number of individuals screened and determined to be eligible; the counts and reasons for those screened out; and the process for selecting the study participants among those eligible.

Did not agree to be in study (n = __)

Did not pass screening criteria (n = __)

Other (n = __)

Move “completed baseline” to correct sequence given your processes.

Program start date(s):

Program end date(s):

Primary analysis sample (n = __)

List reason(s) for exclusion

• ___ (n = __)

• ___ (n = __)

Eligible for second follow-up (n = __)

Completed second follow-up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for immediate post (n = __)

Completed immediate post (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Primary analysis sample (n = __)

List reason(s) for exclusion

• ___ (n = __)

• ___ (n = __)

Eligible for second follow-up (n = __)

Completed second follow up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for first follow-up (n = __)

Completed first follow up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for immediate post (n = __)

Completed immediate post (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Completed baseline (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for first follow-up (n = __)

Completed first follow-up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Assigned to Comparison (n = __)

Assigned to Treatment (n = __)

Randomized (n = __)

Date(s) of random assignment ______

Completed baseline (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Figure A.3. CONSORT diagram for clusters (if applicable), for studies in which consent occurred after assignment

Complete based on pooled sample to date. Also complete diagram for individual clients.

Diagram date: _________

Describe what makes a cluster eligible for the evaluation; the number of clusters screened and the screening criteria used; the number of clusters determined to be eligible; the counts and reasons for those screened out; and whether and how any clusters were prioritized for inclusion in the study sample.

Date of enrollment data __________­­­­­­­

Date of survey data __________

Did not pass screening criteria (n = __)

Other (n = __)

Clusters randomized (n = __)

Date(s) of cluster random assignment ______

Consented to participate (n = __)

Assigned to Treatment (n = __)

Consented to participate (n = __)

Assigned to Comparison (n = __)

Completed baseline data collection (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed baseline data collection (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n =__)

Primary analysis sample (n = __)

List reason(s) for cluster(s) being excluded

• ___ (n = __)

• ___ (n = __)

Completed data collection at immediate post

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at first follow-up

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at second follow-up (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Primary analysis sample (n = __)

List reason(s) for cluster(s) being excluded

• ___ (n = __)

• ___ (n = __)

Completed data collection at second follow-up (n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at first follow-up

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Completed data collection at immediate post

(n = __)

List reason(s) for cluster(s) dropping out

• ___ (n = __)

• ___ (n = __)

Diagram date: ________

Figure A.4. CONSORT diagram for individual clients, for studies in which consent occurred after assignment

Complete based on pooled sample to date.

Date of enrollment data __________

Date of survey data __________

Describe what makes an individual eligible for the evaluation; the number of individuals screened and determined to be eligible; the counts and reasons for those screened out; and the process for selecting the study participants among those eligible.

Did not pass screening criteria (n = __)

Other (n = __)

Randomized (n = __)

Date(s) of random assignment ______

Consented to participate (n = __)

Completed baseline (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Completed baseline (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for immediate post (n = __)

Completed immediate post (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for immediate post (n = __)

Completed immediate post (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for first follow-up (n = __)

Completed first follow up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for first follow-up (n = __)

Completed first follow-up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for second follow-up (n = __)

Completed second follow up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Eligible for second follow-up (n = __)

Completed second follow-up (n = __)

Date(s) of data collection:

List reasons for non-completes

• ___ (n = __)

• ___ (n = __)

Primary analysis sample (n = __)

List reason(s) for exclusion

• ___ (n = __)

• ___ (n = __)

Primary analysis sample (n = __)

List reason(s) for exclusion

• ___ (n = __)

• ___ (n = __)

Assigned to Comparison (n = __)

Assigned to Treatment (n = __)

Consented to participate (n = __)

Move “consented to participate” to correct sequence given your processes.

Program start date(s):

Program end date(s):

1 Those not offered the intervention may receive no services or different services.

2 Assignment is most often at random, though other rigorous designs, such as high-quality quasi-experimental designs, may employ other methods.