Title IV-E Prevention Services Plan

Attachment B: Checklist for Program or Service Designation for HHS Consideration

Instructions:

Section I: The state must complete Section I (Table 1) once to summarize all of the programs and services that the state reviewed and submitted and the designations for HHS consideration.

Section II: The state must complete Section II (Tables 2 and 3) once to describe the independent systematic review methodology used to determine a program or service (listed in Table 1) designation for HHS consideration. Section II outlines the criteria for an independent systematic review. To demonstrate that the state conducted an independent systematic review consistent with sections 471(e)(4)(C)(iii)(I), (iv)(I)(aa) and (v)(I)(aa) of the Act, the state must answer each question in the affirmative. If the independent systematic review used the Prevention Services Clearinghouse Handbook of Standards and Procedures, the relevant sections must be indicated in the “Handbook Section” column. If other systematic standards and procedures were used, states must submit documentation of the standards and procedures used to review programs and services. States should determine the standards and procedures to be used prior to beginning the independent systematic review process. If the State cannot answer each question in Table 2 and Table 3 in the affirmative, ACF will not make transition payments for the program or service reviewed by the state using those standards and procedures.

Section III: The state must complete Section III (Tables 4 and 5) for each program or service listed in Table 1, and provide all required documentation. Section III outlines the requirements for the review of the program or service. States should complete Table 4 prior to conducting an independent systematic review to determine if a program or service is eligible for review. For a program or service to be eligible for review, the answer to both questions in Table 4 must be affirmative and the state must provide the required documentation. If a program or service is eligible for review, the state must conduct the review and identify each study reviewed in Table 5, regardless of whether a study was determined to be eligible to be included in the review.

Section IV: The state must complete Section IV (Tables 6-10) for each program or service (listed in Table 1) reviewed and submitted and provide all required documentation. Section IV lists studies the state determined to be “well-designed” and “well-executed” and outlines characteristics of those studies. Do not include eligible studies that were not determined to be “well-designed” and “well-executed” in Tables 6 -10. States should complete Table 6 with a list of all eligible studies determined to be “well-designed” and “well-executed.” States should complete Table 7 to describe the design and execution of each eligible “well-designed” and “well-executed” study. States should complete Table 8 to describe the practice setting and study sample. States must answer in the affirmative that the program or service included in each study was not substantially modified or adapted from the version under review. States must detail favorable effects on target outcomes present in eligible studies determined to be “well-designed” and “well-executed.” States must detail unfavorable effects on target and non-target outcomes present in eligible studies determined to be “well-designed” and “well-executed.”

Section V: The state must complete Section V (Table 11) for each program or service reviewed and submitted. Section V lists the program or service designation for HHS consideration and verification questions relevant to that designation. The state must answer the questions applicable to the relevant designation in the affirmative.

Section I: Summary of Programs and Services Reviewed and their Designations for HHS Consideration

Section I. Summary of Programs and Services Reviewed

Table 1. Summary of Programs and Services Reviewed

To be considered for transitional payments, list programs and services reviewed and provide designations for HHS consideration.

Program or Service Name (if there are multiple versions, specify the specific version reviewed)	Designations for HHS consideration (Promising, Supported, or Well-Supported)

Section II: Standards and Procedures for an Independent Systematic Review

Section II. Standards and Procedures for a Systematic Review

(Complete Table 2 and Table 3 to provide the requested information on the independent systematic review. The same standards and procedures should be used to review all programs and services.)

Table 2. Systematic Review

Sections 471(e)(4)(C)(iii)(I), (iv)(I)(aa) and (v)(I)(aa) of the Act require that systematic standards and procedures must be used for all phases of the review process. In the table below, verify that systematic (i.e. explicit and reproducible) standards and procedures were used. If the systematic review used the Prevention Services Clearinghouse Handbook of Standards and Procedures, indicate the relevant sections in the “Handbook Section” column. If other systematic standards and procedures were used, submit documentation of the standards and procedures used to review programs and services.

	to Verify	Handbook Section
Were the same systematic standards and procedures used to review all programs and services?		--
Were reviewers trained on systematic standards and procedures used to review all programs and services?		--
Were standards and procedures in accordance with section 471(e) of the Social Security Act?		--
Were standards and procedures in accordance with the Initial Practice Criteria published in Attachment C of ACYF-CB-PI-18-09?		--
Program or Service Eligibility: Were systematic standards and procedures used to determine if programs or services were eligible for review? At a minimum, this includes standards and procedures to: Determine if a program or services is a mental health, substance abuse, in-home parent-skill based, or kinship navigator program; and Determine if there was a book/manual or writing available that specifies the components of the practice protocol and describes how to administer the practice.
Literature Review: Were systematic standards and procedures used to conduct a comprehensive literature review for studies of programs and services under review? At a minimum, this includes standards and procedures to: Search bibliographic databases; and Search other sources of publicly available studies (e.g., websites of federal, state, and local governments, foundations, or other organizations).
Study Eligibility: Were systematic standards and procedures used to determine if studies found through the comprehensive literature review were eligible for review? At a minimum, this includes standards and procedures to: Determine if each study examined the program or service under review (as described in the book/manual or writing) or if it examined an adaptation; Determine if each study was published or prepared in or after 1990; Determine if each study was publicly available in English; Determine if each study had an eligible design (i.e., randomized control trial or quasi-experimental design); Determine if each study had an intervention and appropriate comparison condition; Determine if each study examined impacts of program or service on at least one ‘target’ outcome that falls broadly under the domains of child safety, child permanency, child well-being, or adult (parent or kin-caregiver) well-being. Target outcomes for kinship navigator programs can instead or also include access to, referral to, and satisfaction with services; and Identify studies that meet the above criteria and are eligible for review
Study Design and Execution: Were systematic standards and procedures used to determine if eligible studies were well-designed and well-executed? At a minimum, this includes standards and procedures to: Assess overall and differential sample attrition; Assess the equivalence of intervention and comparison groups at baseline and whether the study statistically controlled for baseline differences; Assess whether the study has design confounds; Assess, if applicable, whether the study accounted for clustering (e.g., assessed risk of joiner bias1); and Assess whether the study accounted for missing data; and Determine if studies meet the above criteria and can be designated as well-designed and well-executed.
Defining Studies: Sometimes study results are reported in more than one document, or a single document reports results from multiple studies. Were systematic standards and procedures used to determine if eligible, well-designed and well-executed studies of a program and service have non-overlapping samples?
Study Effects: Were systematic standards and procedures used to examine favorable and unfavorable effects in eligible, well-designed and well-executed studies? At a minimum, this includes standards and procedures to: Determine if eligible, well-designed and well-executed studies found a favorable effect (using conventional standards of statistical significance) on each target outcome; and Determine if eligible, well-designed and well-executed studies found an unfavorable effect (using conventional standards of statistical significance) on each target or non-target outcome.
Beyond the End of Treatment: Were systematic standards and procedures used to determine the length of sustained favorable effects beyond the end of treatment in eligible, well-defined and well-executed studies? At a minimum, this includes standards and procedures to: Identify (and if needed, define) the end of treatment; and Calculate the length of a favorable effect beyond the end of treatment.
Usual Care or Practice Setting: Were systematic standards and procedures used to determine if a study was conducted in a usual care or practice setting?
Risk of Harm: Were systematic standards and procedures used to determine if there is evidence of risk of harm?
Designation: Were systematic standards and procedures used to designate programs and services for HHS consideration (as promising, supported, well-supported, or does not currently meet the criteria)? At a minimum, this includes standards and procedures to: Determine if a program or service has one eligible, well-designed and well-executed study that demonstrates a favorable effect on a target outcome and should be considered for a designation of promising; Determine if a program or service has at least one eligible, well-designed and well-executed study carried out in a usual care or practice setting that demonstrates a favorable effect on a target outcome at least 6 months beyond the end of treatment and should be considered for a designation of supported; and Determine if a program or service has at least two eligible, well-designed and well-executed studies with non-overlapping samples carried out in usual care or practice settings that demonstrate favorable effects on a target outcome; at least one of the studies must demonstrate a sustained favorable effect of at least 12 months beyond the end of treatment on a target outcome; and should be considered for a designation of well-supported.
Reconciliation of Discrepancies: Were systematic standards and procedures used to reconcile discrepancies across reviewers? (applicable if more than one reviewer per study)
Author or Developer Queries: Were systematic standards and procedures used to query study authors or program or service developers? (applicable if author or developer queries made)

Table 3. Independent Review

The systematic review must be independent (i.e., objective and unbiased). In the table below, verify that an independent review was conducted using systematic standards and procedures by providing the names of each state agency and external partner that reviewed the program or service. States must answer all applicable questions in the affirmative. Submit MOUs, Conflict of Interest Policies, and other relevant documentation.

List all state agencies and external partners that reviewed programs and services
	to Verify
Was the review independent (conducted by reviewers without conflicts of interest including those that authored studies, evaluated, or developed the program or service under review)?
Was a Conflict of Interest Statement signed by reviewers attesting to their independence? If so, attach the statement.
Was a Memorandum of Understanding (MOU) signed by external partners (if applicable)? If so, attach MOU(s).

Sections III-V: Describe and Document Findings from Each Program and Service Reviewed and Submitted

Section III. Review of Programs and Services

(Complete Tables 4-5 for each program or service reviewed.)

Table 4. Determination of Program or Service Eligibility

Fill in the table below for each program or service reviewed.

	to Verify
Does the program or service have a book, manual, or other available documentation specifying the components of the practice protocol and describing how to administer the practice?
Provide information about how the book/manual/other documentation can be accessed OR provide other information supporting availability of book/manual/other documentation.
Is the program or service a mental health, substance abuse, in-home parent-skill based, or kinship navigator program or service?
Identify the program or service area(s).

Table 5. Determination of Study Eligibility

Fill in the table below for each study of the program or service reviewed. Provide a response in every column; N/A or unknown are not acceptable responses. The response in columns iii, v, vi, vii and ix must be “yes” or “no.” The response in column ix is “yes” only when the responses in columns iii, v, vi, and vii are “yes.”

i. Study Title/Authors	ii. Publicly Available Location	iii. Is the study in English? (Yes/No)	iv. Design (RCT, QED, or other). If other, specify design.	v. Did the intervention condition receive the program or service under review in accordance with the book/manual/documentation? (Yes/No)	vi. Did the comparison condition receive no or minimal intervention or treatment as usual? (Yes/No)	vii. Did the study examine at least one target outcome? (Yes/No)	viii. Year Published	ix. Eligible for Review? (Yes/No)
Example Title. Smith, A.B., Jones, C.D., and Doe, E.F.	https://www.acf.hhs.gov/opre	English	RCT	Yes	Yes	Yes	1997	Yes

Section IV. Review of “Well-designed” and “Well-executed” Studies

(Complete Tables 6-10 for each program or service reviewed.)

Table 6. Studies that are “Well-Designed” and “Well-Executed”²

Provide a copy of each of the studies determined to be eligible for review and determined to be “well-designed” and “well-executed.”

List all eligible studies that are “well-designed” and “well-executed’ (Study Title/Author)

Table 7. Study Design and Execution

For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below. Provide a response in every column; N/A or unknown are not acceptable responses for columns i, ii, iii, v, vi, and vii. The response in column ii must be “yes.”

i. Study Title/Authors	ii. Verify the Absence of all Confounds? (Yes/No)	iii. List Measures that Achieved Baseline Equivalence	iv. List Measures that did NOT Achieve Baseline Equivalence but were Statistically Controlled for in Analyses	v. Overall Attrition3 (for RCTs only)	vi. Differential Attrition4 (for RCTs only)	vii. Does Study Meet Attrition Standards?	viii. Notes, as needed
Example Title. Smith, A.B., Jones, C.D., and Doe, E.F.	Yes	-Center for Epidemiologic Studies Depression Scale (CES-D) -Child Behavior Checklist (CBCL)	-Income	2.0 percent	4.3 percentage points	Yes	N/A

Table 8. Study Description

For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below to describe the practice setting and study sample as well as affirm that the program or service evaluated was not substantially modified or adapted from the version under review. Provide a response in every column; N/A or unknown are not acceptable responses. The response in column v must be “yes.”

i. Study Title/Authors	ii. Was the study conducted in a usual care or practice setting? (Yes/No)	iii. What is the study sample size?	iv. Describe the sample demographics and characteristics	v. Verify that the program or service evaluated in the study was NOT substantially modified or adapted from the manual or version of the program or service selected for review (Yes/No)
Example Title. Smith, A.B., Jones, C.D., and Doe, E.F.	Yes	N=354 Caregivers, N = 177 Child, N = 177	Caregivers – Average age = 37 years old (SD = 5 years); 95% female; 35% Black or African American, 25% White, 30% Latino or Hispanic, and 10% other; and 78% of households living 200% below the federal poverty level. Children – Average age = 5 years old (SD=1.3 years); 47% female; 37% Black or African American, 27% White, 32% Latino or Hispanic, and 4% other.	Yes

Table 9. Favorable Effects

For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below listing only target outcomes with favorable effects. Provide a response in every column; N/A or unknown are not acceptable responses.

i. Study Title/Authors	ii. List the Target Outcome(s)	iii. List the Outcome Measures	iv. List the Reliability Coefficients for Each	v. Are Each of the Outcome Measures Valid?	vi. Are Each of the Outcome Measures Systematically Administered?	vii. List the P-Values for Each of the Outcome Measures	viii. List the Size of Effect for Each of the Outcome Measures	ix. Indicate the Length of Effect Beyond the End of Treatment (in months)
Example Title. Smith, A.B., Jones, C.D., and Doe, E.F.	Parent/Caregiver Mental Health (Depression)	CES-D	Cronbach’s alpha coefficient = 0.91	Yes	Yes	p = 0.04	d = 0.13	8 mos
	Child Behavioral and Emotional Functioning (Externalizing Behaviors)	CBCL (Aggressive Behavior Scale)	Cronbach’s alpha coefficient = 0.94	Yes	Yes	p = 0.03	d = 0.24	0 mos
	Child Behavioral and Emotional Functioning (Internalizing Behaviors)	CBCL (Anxious/Depressed Scale)	Cronbach’s alpha coefficient = 0.84	Yes	Yes	p = 0.23 (non-sig)	N/A	0 mos

Table 10. Unfavorable Effects

For each study eligible for review and determined to be “well-designed” and “well-executed,” fill out the table below listing only target outcomes with unfavorable effects. Provide a response in every column; N/A or unknown are not acceptable responses.

i. Study Title/Authors	ii. List the Target or Non-Target Outcome(s)	iii. List the Outcome Measures	iv. List the Reliability Coefficients for Each	v. Are Each of the Outcome Measures Valid?	vi. Are Each of the Outcome Measures Systematically Administered?	vii. List the P-Values for Each of the Outcome Measures	viii. List the Size of Effect for Each of the Outcome Measures	ix. Indicate the Length of Effect Beyond the End of Treatment (in months)
Example Title. Smith, A.B., Jones, C.D., and Doe, E.F.	Adult Height	Inches	Cronbach’s alpha coefficient = 0.99	Yes	Yes	p = 0.047	d = -0.05	0 mos

Section V. Program or Service Designation for HHS Consideration

Table 11. Program or Service Designation for HHS Consideration

Fill out the table below for the program or service reviewed. Only select one designation. Answer questions relevant to the selected designation; relevant questions must be answered in the affirmative.

	to Verify
There is NOT sufficient evidence of risk of harm such that the overall weight of evidence does not support the benefits of the program or service
	the Designation and Answer the Questions Relevant to that Designation
Well-Supported
Does the program or service have at least two eligible, well-designed and well-executed studies with non-overlapping samples5 that were carried out in a usual care or practice setting?
Does one of the studies demonstrate a sustained favorable effect of at least 12 months beyond the end of treatment on at least one target outcome?
Supported
Does the program or service have at least one eligible, well-designed and well-executed study that was carried out in a usual care or practice setting and demonstrate a sustained favorable effect of at least 6 months beyond the end of treatment on at least one target outcome?
Promising
Does the program or service have at least one eligible, well-designed and well-executed study and demonstrate a favorable effect on at least one ‘target outcome’?

1If a cluster randomized study permits individuals to join clusters after randomization, the estimate of the effect of the intervention on individual outcomes may be biased if individuals who join the intervention clusters are systematically different from those who join the comparison clusters.

2 For reference, the Prevention Services Clearinghouse Handbook Chapter 5, defines “well-designed” and “well-executed” studies as those that meet design and execution standards for high or moderate support of causal evidence. Prevention Services Clearinghouse ratings apply to contrasts reported in a study. A single study may have multiple design and execution ratings corresponding to each of its reported contrasts.

3 For reference, the Prevention Services Clearinghouse Handbook section 5.6 defines overall attrition as the number of individuals without post-test outcome data as a percentage of the total number of members in the sample at the time that they learned the condition to which they were randomly assigned.

4 For reference, the Prevention Services Clearinghouse Handbook section 5.6 defines differential attrition as the absolute value of the percentage point difference between the attrition rates for the intervention group and the comparison group.

5Samples across multiple sources of a study are considered overlapping if the samples are the same or have a large degree of overlap. Findings from an eligible study determined to be “well-executed” and “well-designed” may be reported across multiple sources including peer-reviewed journal articles and publicly available government and foundation reports. In such instances, the multiple sources would have overlapping samples. The findings across multiple sources with these overlapping samples should be considered one study when designating a program or service as “well-supported,” “supported,” and “promising.”

8