Supporting Statement B - CWCC Implemenation and Evaluation Templates

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Building Capacity to Evaluate Child Welfare Community Collaborations to Strengthen and Preserve Families (CWCC) Local Grantee Implementation Plan and Evaluation Plan Templates

Formative Data Collections for Program Support

0970 - 0351

Supporting Statement

Part B

February 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Mary Mueggenborg and Laura Hoard

Part B

B1. Objectives

Study Objectives

The objectives of this information collection are to collect systematic information about CWCC grantees’ implementation and evaluation plans via an implementation plan template (Instrument 1) and an evaluation plan template (Instrument 2) to:

1. Ensure grantees have complete and thorough implementation and evaluation plans that meet CB requirements;

2. Allow for a consistent and clear review process by which CB can approve grantees’ implementation and evaluation plans; and

3. Inform associated implementation and evaluation TA efforts to support CWCC program implementation and local evaluations.

Generalizability of Results

This information collection is intended to capture internally-valid descriptions of CWCC grantee-developed implementation plans and evaluation plans amongst the 9 CWCC grantees funded in September 2019, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The proposed templates allow for the systematic collection of information to ensure the grantees address the implementation and evaluation requirements of their grants.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. 

B2. Methods and Design

Target Population

This information collection will be completed by the 9 CWCC grantees who received funding in September 2019. Grantees can direct one or multiple staff to complete the templates, including assigning responsibility for the evaluation plan template to their contracted local evaluation firm or organization.

Sampling and Site Selection

All 9 CWCC grantees funded in September 2019 will be asked to complete the implementation plan and evaluation plan templates. Each grantee is developing their own implementation plan and evaluation plan which align with their specific grant activities, research questions, data capacity, and community and cultural context. We cannot select a sample of implementation and evaluation plans to review for implementation or evaluation TA purposes. In addition, CB needs to review and approve each grantee’s implementation and evaluation plan.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

The implementation plan template consists of 11 sections: (1) Introduction, (2) Problem Exploration and Target Area for Project Implementation, (3) Theory of Change, (4) Intervention, (5) Implementation Teams, (6) Implementation Readiness and Capacity Building, (7) Initiating and Sustaining Implementation, (8), Quality Assurance and Continuous Quality Improvement, (9) Communication and Dissemination, (10) Technical Assistance Needs, and (11) Work Plan and Timeline. The template includes tables, forms, and links to TA resources to assist grantees in completing their implementation plans. It also includes guidelines for formatting and structuring the plan. Grantees may enter the required information directly into the template but may also create a plan in a separate document, using the template as a guide.

An earlier version of the implementation plan template was used with a smaller (4), earlier cohort of CWCC grantees (fewer than 9 respondents were involved). Changes were made to the current template based on that experience, primarily to reduce redundancy with the evaluation plan template (for example, by removing sections on logic models and data collection).

The evaluation plan template consists of two components: 1) Categories of information contained in evaluation plans, including detailed recommendations of the specific details included in each category, and 2) Quality indicators for each category that describe elements of high quality evaluations.

Categories of information include research questions, sampling strategies, data security plans, methodologies, analysis plans, and logic models. These categories and levels of requested detail were developed through experience creating and reviewing evaluation plans. These categories also align with ACYF-CB-IM-19-04.

The quality indicators were drawn from several sources, including ACF’s Prevention Services Clearinghouse (PSC), the U.S. Department of Education’s What Works Clearinghouse (WWC), the U.S. Department of Labor’s Clearinghouse for Labor Evaluation and Research (CLEAR), and Abt Associates’ proprietary EVIRATER^TM standards. PSC, WWC and CLEAR standards focus primarily on comparison group designs such as randomized controlled trials and quasi-experimental designs, whereas Abt’s EVIRATER standards address the full spectrum of evaluation designs, including pre-post and interrupted time series. Additional community-level quality indicators were drawn from Abt’s experience on other projects including the TPP Scale-Up project for the Office of Adolescent Health.

An earlier version of this evaluation plan template was used with a smaller (4), earlier cohort of CWCC grantees (fewer than 9 respondents were involved). Changes were made to the current template based on that experience, such as removing a section on continuous quality improvement plans, and adding detail to some quality indicators. Grantees may enter the required information directly into the template but may also create a plan in a separate document, using the template as a guide.

B4. Collection of Data and Quality Control

Grantees (and/or their designees, such as contracted evaluation staff) will provide the information from their required implementation plans and evaluation plans. The contracted implementation TA provider and evaluation TA provider will review submitted implementation plan and evaluation plan templates, respectively, and in conjunction with CB, ask grantees to address any information gaps. Grantees will also receive group training on how to complete the implementation plan template and evaluation plan template, and the implementation TA provider and evaluation TA provider will be available to answer questions from grantees.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Grantees are required to complete and submit an implementation plan and evaluation plan as a condition of their cooperative agreement. CB will use the completed implementation plan and evaluation plan templates to make implementation and evaluation approval decisions, so we expect a complete implementation plan and evaluation plan template from all 9 grantees (100% response rate). Item response rates will not be calculated, but grantees with missing information will be asked to revise their templates.

NonResponse

Not applicable. We will not conduct any analysis of the information provided.

B6. Production of Estimates and Projections

No estimations will be conducted.

B7. Data Handling and Analysis

Data Handling

The contracted implementation TA provider and evaluation TA provider will ask grantees to clarify entries or provide missing information in the implementation plan template and/or evaluation plan template, as appropriate.

Data Analysis

Implementation plans will be assessed for their completeness and responsiveness to the information requested in each section of the template. CB will use the information provided in each grantee’s implementation plan to make a decision to approve implementation of the grantee’s project, approve implementation with a request for minor revisions, or to request major revisions and a resubmission of the implementation plan.

CB will use the information in the templates to make a decision to approve the evaluation, approve with minor revisions requested, or to request major revisions and a resubmission of evaluation plans.

Data Use

Respondents will be informed of all planned uses of data, and that their participation is voluntary. The submitted completed templates will not be shared beyond ACF and the TA contractors. Grantees may publicize their implementation and evaluation plans, so the information submitted within could potentially be made public by the grantees. TA contractors may also present aggregated details from grantees evaluation plans, such as evaluation designs or instruments used, via presentations or written materials at an evaluation conference as context for how and why evaluation TA was provided.

B8. Contact Person(s)

Name	Organization	Role on Contract	Phone/email
Mary Mueggenborg	OPRE, ACF, HHS	Co-Contracting Officer’s Representative	(202) 401-5689 Mary.Mueggenborg@acf.hhs.gov
Laura Hoard	OPRE, ACF, HHS	Co-Contracting Officer’s Representative	(202) 401-4561 Laura.Hoard@acf.hhs.gov
Allison Hyra	Abt Associates	Project Director (Evaluation TA)	(301) 347-5058 Allison_hyra@abtassoc.com
Michelle Blocklin	Abt Associates	Evaluation TA Task Lead	(617) 520-3597 Michelle_Blocklin@abtassoc.com
Beth Claxon	Children’s Bureau, ACF, HHS	Federal Project Officer	(202) 205-8221 Beth.Claxon@acf.hhs.gov
Chereese Phillips	Children’s Bureau, ACF, HHS	Contracting Officer’s Representative	(202) 594-3118 Chereese.phillips@acf.hhs.gov
Elliott Graham	James Bell Associates	Project Director (Implementation TA) and Implementation TA Task Lead	(703) 842-0958 Graham@jbassoc.com

Attachments

Instrument 1: Implementation Plan Template

Instrument 2: Evaluation Plan Template

6