Supporting Statement Part A

Cross-Site Evaluation of the Children’s Bureau’s Grantee Cluster:

Supporting Evidence-Based Home Visiting

to Prevent Child Maltreatment (EBHV)

Supporting Statement, Part A

For OMB Approval

September 23, 2009

A. JUSTIFICATION

The Administration for Children and Families (ACF) of the Department of Health and Human Services (DHHS) is requesting Office of Management and Budget (OMB) clearance for the data collection instruments and procedures, plans for data analysis, and reporting of findings to be used in a cross-site evaluation of its Children’s Bureau’s grantee cluster, Supporting Evidence-Based Home Visiting to Prevent Child Maltreatment (EBHV), a five-year project beginning in FY 2009-2010. The current submission requests clearance for the first three-year period of the Cross-Site EBHV Evaluation. ACF will submit an extension request at the beginning of the third year for the continuation of these activities for the last two years of the project. This section provides supporting statements for each of the points outlined in Part A of the OMB guidelines for the collection of information for this evaluation.

A.1. Circumstances Making the Collection of Information Necessary

a. Overview of Request

Child maltreatment is a serious problem. Approaches to reduce and prevent this maltreatment have been identified by researchers and practitioners in the form of evidence-based prevention. Learning when or how states and agencies can successfully implement and support such programs and their effect on maltreatment and other outcomes is an important issue for the Children’s Bureau (CB/ACF/DHHS) and other decision-makers. In 2008, CB/ACF/DHHS funded 17 cooperative agreements to support the development of infrastructure needed for high-quality implementation of existing evidence-based home visiting programs to prevent child maltreatment. Foremost, the grant initiative requires sites to build, expand, and/or connect the local or state systems supporting children and home visiting services. As part of this initiative, grantees will leverage their grants with other funding sources to support the implementation of evidence-based home visiting program models with high fidelity, the scaling up of implementation with high fidelity, and the sustainability of program model delivery. Grantees must also conduct local implementation and outcome evaluations along with an economic analysis. Appendix A lists the grantees, their locations, and the evidence-based home visiting program model(s) they are implementing.

CB/ACF/DHHS contracted with Mathematica Policy Research, Inc. and its subcontractor, Chapin Hall at the University of Chicago (CH), to conduct a participatory and utilization-focused cross-site evaluation of the grantees’ programs over the five-year period. The Cross-Site EBHV Evaluation will collect information from the 17 EBHV grantees to augment the existing evidence base by identifying successful strategies for adopting, implementing, and sustaining high quality home visiting programs to prevent child maltreatment. This will be done through support of rigorous local evaluations as well as using this data and cross-site research to assess participant, program, and systems change outcomes.

The Cross-Site EBHV Evaluation will provide information on integration of systems and performance of evidence-based home visiting with new target populations (e.g., Latinos). This data collection aims to build knowledge for both practice and policy making and could play an essential role in the future delegation of funds for programs supported by the Child Abuse Prevention and Treatment Act (CAPTA, P.L. 108-36).

b. Study Context and Rationale

In FY 2007, more than three million reports of suspected child abuse or neglect regarding six million children were made to child protective services agencies across the country. Close to one million (794,000) children were victims of substantiated maltreatment, and an estimated 1,760 children died as a result (USDHHS 2009). Despite recent declines in the number of substantiated cases of neglect and physical and sexual abuse (Finkelhor 2007; Finkelhor and Jones 2006), these large numbers demonstrate the need for the prevention of child maltreatment and effective evidence-based programs to improve outcomes for families and communities. The vast majority of reported maltreatment occurs within the context of family lives (relationships) that often include high degrees of parental depression and stress, low education levels, and/or limited financial resources (Wolfe 2004). Child well-being is deeply influenced by the adults who have primary responsibility for child rearing, typically their mothers and fathers. These caregiving relationships have been extensively studied and shown to be amenable to preventive intervention (an approach that outlines prevention goals rather than solely employing intervention or treatment after maltreatment has occurred). With improved relationships and child attachment to parents, at-risk families will be enabled to more adequately support their children’s early development and learning and thus improve the outcomes for thousands of young children (Appleyard and Berlin 2007; Daro 2006).

Addressing the needs of such families is a critical aim of emerging federally-funded maltreatment prevention efforts. In particular, there is a strong interest in the potential of home visiting programs to access and support these families. Well-designed and well-implemented home visiting programs help to improve the parent-child relationship and deepen attachment by providing support and models of positive parenting skills (Appleyard and Berlin 2007; Berlin et al. 2008; Daro 2006; Wolfe 2004). Interventions, however, cannot reach their full potential without taking into account the service delivery systems that complement these actions and support families (Foster-Fishman et al. 2007). These systems are important because they define who will be served and how they will receive services, and how they will be funded, monitored, and staffed. In order for home visiting interventions to have the greatest effect possible, these systems need to be integrated, supportive, and conducive to the delivery of services. Thus, the EBHV grant program focuses on the development of essential infrastructure, such as funding streams and staff workforce development, to support high-quality program implementation.

While several models of home visiting programs have been shown to be effective in reducing risk factors associated with child maltreatment and increasing protective factors such as attachment, the ability of state and local organizations to implement these programs with high fidelity is still uncertain. In particular, knowledge is needed about how to build the infrastructure and service systems necessary to implement and sustain evidence-based home visiting programs with fidelity to their models, and whether and how to adapt programs to new target populations. Therefore, the proposed Cross-Site EBHV Evaluation will examine four domains that are key to the implementation and monitoring of evidence-based home visiting programs—systems change³, fidelity to evidence-based models, costs of home visiting programs, and family and child outcomes. Analysis will link data and findings from these four domains to further strengthen the usefulness of the evaluation to the government and other key evaluation audiences. Finally, the Cross-Site EBHV Evaluation will include a process study that will provide information for understanding the context in which each grantee operated and how this influenced the grantees’ progress and achievements in all four domains. Research questions to be addressed by the study by these five evaluation components, as well as cross-domain questions, include:

• Systems Change. How did grantees build infrastructure capacity to support the goals of (1) implementing evidence-based home visiting interventions with fidelity to prevent child maltreatment, (2) scaling up these high-fidelity interventions, and
  (3) sustaining these high-fidelity interventions? What changes in infrastructure capacity occurred in service delivery systems over the course of the initiative? To what extent were system-wide EBHV goals achieved over the course of the initiative?

• Fidelity to Evidence-based Models. Were the home visiting programs initially implemented with fidelity? Were they delivered with ongoing fidelity? To what extent did grantees modify the evidence-based models to fit their target populations and local service delivery contexts? What contextual factors were associated with fidelity of implementation?

• Costs of Home Visiting Programs. What are the total costs of delivering and supporting evidence-based home visiting during a typical operating year? What does each program cost per participating family? How are costs allocated across key program components? How do costs vary by key grantee characteristics, such as program model, stage of implementation, and target population or by contexts, such as region of the country or urban/rural location?

• Family and Child Outcomes. Do evidence-based home visiting programs improve parent and child outcomes when they are implemented in the “real world” and supported by investments in infrastructure? How did grantees identify their intended outcomes? How and for what reasons did grantees adjust their perspective on achieving these outcomes as the initiative matured? How do outcomes vary across the different target populations grantees are serving? How do they vary across program models?

• Process. How did grantees plan and implement their EBHV grant projects? What is the context in which EBHV grantees planned and implemented their projects? Specifically, what are (1) facilitating factors or (2) barriers to the overall planning and implementation of the EBHV project? What initial and ongoing training and essential infrastructure support did EBHV grantees receive from their national program models? What technical assistance (TA) did grantees receive from the Children’s Bureau, its contractors, or other TA providers to support their planning and implementation efforts? How was it used?

• Cross Domain. Is systems change related to the fidelity of implementation? What is the nature of this relationship? Are greater investments, as measured by costs of home visiting programs, related to greater fidelity? Are resources devoted to particular program activities and essential infrastructure related to greater fidelity? Do program and essential infrastructure costs vary among grantees focusing on different systems change activities? What contextual factors were found to be barriers or facilitators to systems change and fidelity of the implementation?

A.2. Purpose and Use of the Information Collection

The 17 grantees are implementing six different home visiting program models in a variety of combinations (see Appendix A). Approximately 5,000 home visiting program participants and 425 direct service providers in 50 agencies implementing these programs will be part of local EBHV initiatives. Each grantee will evaluate how they implemented the program model as well as program costs and its outcomes for families and children. Methods for local evaluations range from descriptive studies to randomized control trials. Data being collected for local evaluations will be used in the cross-site evaluation, which will minimize grantee burden and increase cost effectiveness.

As mentioned previously, the cross-site evaluation aims to address five evaluation components: systems change, fidelity to evidence-based models, costs of home visiting programs, family and child outcomes, and process. Planned cross-site data collection will require both qualitative and quantitative methods. Seven instruments and two products will be utilized—one qualitative instrument through site visit interviews, six quantitative instruments, and two grantee-required product files. These data elements are outlined below and summarized in Table A.1.

Qualitative data will be collected during two multi-day site visits (spring of 2010 and 2012); these data will inform the domains of systems change, fidelity to evidence-based models, and costs of home visiting programs, as well as the process study. We will conduct semi-structured individual and small group interviews with EBHV grantees, key staff, and partners and focus groups with front-line home visiting staff members and supervisors. These activities follow an interview guide (see Table A.1).

TABLE A.1

INSTRUMENTs, PERIODICITY, and Evaluation components

			Evaluation Component
Instrument/Product	Mode	Periodicity	Systems Change	Fidelity to Evidence-Based Models	Costs of Home Visiting Programs	Family and Child Outcomes	Process
EBHV grantee and key staff-partner interview guide	In person	Spring 2010; Spring 2012	X	X	X		X
EBHV grantee systems web-based data entry	Web	Semiannual	X
EBHV agency fidelity/cost web-based data entry	Web	Monthly		X	X
EBHV grantee data quality progress table	Electronic file	Four times/year				X
Participant-home visitor relationship questionnaire	PAPI	Program entry and exit		X
Home visitor-participant relationship questionnaire	PAPI	Program entry and exit		X
EBHV grantee-partner network survey	Web	Spring 2010; Spring 2012; Spring 2014	X
Local analytic reports	Electronic file	As required by grant (no additional burden)				X
Data file for NDACAN	Electronic file	As required by grant (no additional burden)	X	X	X	X

NDACAN = National Data Archive on Child Abuse and Neglect

PAPI = paper-and-pencil instrument

Six data collection instruments will be used for quantitative data collection—grantee systems web-based data entry, agency fidelity/cost web-based data entry, a grantee data quality progress table, a participant-home visitor relationship questionnaire, a home visitor-participant relationship questionnaire, and a grantee-partner network survey. First, grantees will utilize the systems web-based data entry to answer tailored questions on systems change goals, activities, and progress. Second, EBHV agencies implementing the evidence-based home visiting programs will complete fidelity/cost web-based data entry. To assess fidelity to the evidence-based models, agencies will provide information on their home visiting program(s), provider(s), and participants for the cross-site evaluation.⁴ Much of this data will be one-time entries on a rolling basis as new families enroll, with some monthly updating, with the exception of a home visit encounter form that captures services received at each home visit (occurring approximately every two weeks). This agency web-based system will also be a source for obtaining yearly financial data to evaluate costs of home visiting programs. Third, grantees will complete a data quality progress table that will capture information such as response rates and missing data for the family and child outcomes data collected by grantees.⁵ The fourth and fifth quantitative data instruments are related; for the Cross-Site EBHV Evaluation, grantees will collect paper-and-pencil questionnaires completed by participants and home visitors separately on the home visiting relationship they have established, shortly after beginning services and then at program exit. And sixth, a grantee-partner network survey will be completed twice in the requested three-year clearance period to examine each grantee’s relationships with its key partners and how these relationships change over time.

Additionally, two products required for the local evaluations will be gathered for the Cross-Site EBHV Evaluation—analytic reports of family and child outcomes and local evaluation data files. These will also support future secondary data analyses available to other researchers via the National Data Archive on Child Abuse and Neglect (NDACAN). Since these products are grant requirements, they impose no additional burden to grantees.

The Cross-Site EBHV Evaluation will inform CB/ACF/DHHS and the 17 EBHV grantees regarding approaches, challenges, and barriers to implementing, scaling up, and sustaining evidence-based home visiting programs while building infrastructure and systems to support such programs with fidelity. As a utilization-focused evaluation, this study will provide timely and useful information to key stakeholders, including the 17 grantees, other operators of EBHV programs, state and county agencies, other EBHV funders, and developers of national evidence-based home visiting program models, on an ongoing basis. The main uses of the data collected will be to:

• Describe the systems within which EBHV grantees acted and identify interrelated and interdependent participants (both individuals and organizations) who worked together to implement, scale up, and/or sustain high-fidelity evidence-based home visiting programs to reduce child maltreatment.

• Describe the extent to which grantees demonstrated fidelity to their selected home visiting program model including the extent to which it was implemented as intended by its designers, and how fidelity is maintained over time.

• Examine the costs of resources devoted to delivering an effective evidence-based home visiting program as well as the essential infrastructure for supporting it, such as training, supervision, and program management.

• Present a systematic review of the evidence from grantees’ local evaluations of associations between supporting evidence-based home visiting initiatives and family and child outcomes.

• Describe how grantees implemented systems change and home visiting program activities and why they took these approaches.

A.3. Use of Improved Information Technology and Burden Reduction

The proposed data collection will use web-based data entry systems to collect information in a uniform manner across all grantees. The systems will be designed to be user-friendly and efficient. For example, the systems will be customized for each grantee through strategies such as pre-loading, grantee-specific information for interactive displays and updates. These modifications will reduce burden and improve quality and usefulness of evaluation data in subsequent reporting periods. To use the systems, each grantee and agency will need computer access with an Internet connection. Users will enter the systems through a logon screen by entering a user-specific password stored in the system. Grantees and agencies will designate the individuals needing access, and the Cross-Site EBHV evaluation team will designate user names and passwords and program them into the system. The system will accommodate multiple user names and passwords for each site to facilitate multiple users per program.

A.4. Efforts to Identify Duplication and Use of Similar Information

The Cross-Site EBHV Evaluation design aims to align with the grantee’s local evaluations to minimize burden or duplication of data collection. Potential duplication of data exists in the proposed collection effort with the site visit interviews and agency fidelity/cost web-based data entry. To address this possibility, for the site visits, we will identify grantees with interview plans as part of their local evaluation that are similar to the cross-site evaluation’s plans and will assess the feasibility of coordinating efforts based on periodicity and goals. Then, to reduce duplication, we will offer grantees two options: (1) Cross-Site EBHV Evaluation staff conduct joint interviews with EBHV grantee local evaluators, or (2) Cross-Site EBHV Evaluation staff conduct the interview on behalf of the grantee and share notes afterwards.

For the web-based data entry systems, there is overlap between data already collected as part of one of the national program models being implemented by numerous grantees and the proposed data collection for the Cross-Site EBHV Evaluation. Where overlap exists, we will obtain data grantees provide to the program’s national office through secure transmittals from the national office, instead of requiring dual data entry by these grantees.

A.5. Impact on Small Businesses or Other Small Entities

A potential exists to affect small entities within a grantee site, depending on the local community partners and funders with which EBHV grantees engage. EBHV grantee partners, funders, and direct service providers would be included as part of site visit interviews and grantee-partner network survey. Additionally, EBHV grantee agencies would be directly involved to enter data into the web-based system on fidelity and cost. Proposed data collection for these three efforts will impose minimal burden on all organizations involved including small business and entities and discussions will be as streamlined as is feasible, about 90 minutes for interviews, 25 minutes for the survey, and approximately 9 hours a month for the fidelity/cost web-based data entry.

A.6. Consequences of Collecting Information Less Frequently

The proposed data collection for the Cross-Site EBHV Evaluation is critical to understanding how grantees implement the initiative. If these data are not collected, or are collected less frequently, evaluators would not be able to reliably describe the strategies used to support the adoption, implementation, and sustainability of high quality evidence-based home visiting programs with fidelity, how grantees evolved over the five-year grant period, or implementation successes and challenges of grantees. The proposed frequency of data collection varies by instrument to ensure high quality data, compliment grant requirements, and capture change over time. For each data collection mode, periodicity is limited to the minimum needed to obtain high quality data and address the proposed research questions.

Site visit interviews will occur at two time points during the grant initiative, during spring of 2010 and 2012. The first site visit enables the cross-site evaluation team to provide utilization-focused reports to the grantees early in the grant period. The second site visit captures change over time and each grantee’s evolution from early to later implementation. The frequency of the grantee-partner network survey aligns with the planned site visits to capture how a partner’s perspective changes over the course of the initiative, with a third round of data collection after the EBHV grantees’ local evaluation concludes to assess partnership sustainability. Less frequent collection of these data would substantially reduce the analytic value of the study.

The grantee systems web-based data entry will require semiannual entry. This timing will match the timing of grant-required progress report preparation and allow for regular updating on goal progress.

The agency fidelity/cost web-based data entry system will provide the opportunity for monthly uploads but the actual frequency will be ongoing with variation across data elements. The cross-site evaluation will require that grantees be up-to-date on a monthly basis for data entry in the fidelity to evidence-based models. Cost-related data will be entered annually. Data collection will occur throughout the year to ensure the receipt of high quality data and support responsive intervention by the cross-site evaluation team when concerns are detected. Additionally, asking for monthly data will ensure that grantees remain current with their collection and entry of data. Similarly, ongoing completion of the data quality progress table at approximately four time points will ensure grantees are monitoring local evaluation data collection and data quality.

The participant-home visitor relationship questionnaire (completed by participants) and the home visitor-participant relationship questionnaire (completed by home visitors) will capture information at two time points, when participants enter and exit the program, to assess the extent to which relationships are built while participating in home visiting services.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances requiring deviation from these guidelines.

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995 (P.L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection. The first notice for the Cross-Site EBHV Evaluation data collection was published in the Federal Register, Volume 74, No. 119, Page 29701 on June 23, 2009 (Reference number FR E9–14624). No public comments were received during the 60 days following that announcement. A copy of the 60-day notice is included in Appendix B.

Members of the Cross-Site EBHV Evaluation expert panel have been contacted for advice on various aspects of the study design and the data collection instruments. Their feedback was obtained through in-person meetings, telephone conversations, and written comments. Members of the Cross-Site EBHV Evaluation Expert Panel are listed in Table A.2.

TABLE A.2

Cross-Site EBHV Evaluation Expert Panel Membership

Name	Affiliation
Phaedro Corso	University of Georgia
Diane DePanfilis	University of Maryland, Baltimore
Kenneth Dodge	Duke University
Glenda O’Eyang	Human Systems Dynamics Institute

A.9. Explanation of Any Payment or Gift to Respondents

No payments or gifts will be provided to respondents as part of data collection.

A.10. Assurance of Privacy Provided to Respondents

This study is being conducted in accordance with all relevant regulations and requirements, including the Privacy Act of 1974 (5USC 552a), the Privacy Act Regulations (34 CFR Part 5b), and the Freedom of Information Act (5 CFR 552) and related regulations (41 CFR Part 1-1, 45 CFR Part 5b, and 40 CFR 44502). All interviewers and data collectors will be knowledgeable about privacy procedures and will be prepared to describe them in detail or to answer any related questions raised by respondents. During the introduction to each interview, site visit respondents will be told that none of the information they provide will be used for monitoring or accountability purposes and that the results of the study will be presented in aggregate form only. The web-based systems manuals will also contain an introductory statement to this effect.

To further ensure privacy, identifying information will be maintained in separate tables in the database, which will be linked to the data entry screens only by a sample identification number. Personal identifiers that could be used to link individuals with their responses will be removed from all completed questionnaires and stored under lock and key at the research team offices. Access to the file linking sample identification numbers with identifying information will be limited to a small number of individuals who have a need to know this information. Additionally, all Mathematica staff will be required to sign a confidentiality statement (see Appendix C). We are obtaining a NIH certificate of confidentiality to help ensure the privacy of study participants. We are in the process of applying for the IRB clearance needed prior to applying for the certificate.

Access to hard-copy documents will be strictly limited. Documents will be stored in locked files cabinets and discarded material will be shredded. Any computer files that contain this information also will be locked and password-protected, and access will be limited to specific users. Data stored and transmitted as part of the web-based system will be kept tightly secure through passwords, encryption, and server security. Password security will be utilized to make sure that only authorized users of the system can gain access. The use of secure transmission technologies will ensure that data will be encrypted as it is sent across the Internet. The website collecting the data will be hosted on one of Mathematica’s secure web servers accessible through the Internet, but the data will not be stored directly on this web server. Data will be stored in a database residing on Mathematica’s local area network that is protected from unauthorized outside access using industry-standard hardware and software firewall protection.

A.11. Justification for Sensitive Questions

There are no personally sensitive questions in this data collection.

A.12. Estimates of Annualized Burden Hours and Costs

The proposed data collection does not impose a financial burden on respondents nor will they incur any expense other than the time spent participating.

The estimated annual burden for study respondents—EBHV grantee staff, evaluators, partners, funders, direct service providers, and home visiting participants—is listed in Table A.3. The total annual burden is expected to be 11,355 hours for collection of data in the various instruments, as noted below. We base time estimates on our experience using similar site visit protocols, web-based entry techniques, or questionnaires in previous projects including the Head Start Oral Health Initiative Evaluation, the evaluation of the Early Head Start Enhanced Home Visiting Pilot Project, and the Building Strong Families Demonstration and Evaluation.

To compute the total estimated annual cost, total burden hours were multiplied by the average hourly wage for each adult participant, based on median weekly wages from the Bureau of Labor Statistics, Current Population Survey estimates (first quarter of 2009 provided for full-time employees over the age of 25). For most instruments, the respondent will be a member of the EBHV grantee staff, local evaluators, or key stakeholders. For these respondents, we used the median salary for individuals with a bachelor’s degree ($25.60 per hour) as the most common minimum education level of a diverse group (to include program directors, community agency leaders, and local evaluators and research assistants).

For home visiting participants, we used the median hourly rate for individuals with less than a high school education ($11.25 per hour) because home visiting program models target high-risk families whose members generally have low educational attainment. For home visitors, we used the median salary for individuals with a bachelor’s degree ($25.60 per hour) based on one program model (which 12 of 17 grantees are implementing) requiring college-educated nurses. Another model trains paraprofessionals but notes a preference for individuals with a college education. Selecting the higher wage level produces a conservative (upper-end) estimate of the potential annual cost across all EBHV grantee site respondents.

TABLE A.3
ESTIMATED ANNUAL RESPONSE BURDEN AND ANNUAL COST

Instrument	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours	Median Hourly Wage	Total Annual Cost
EBHV grantee and key staff-partner interview guide	249	2	1.60	797	$25.60	$20,403.20
EBHV grantee systems web-based data entry	17	2	1.00	34	$25.60	$870.40
EBHV agency fidelity/cost web-based data entry	50	12	9.00	5,400	$25.60	$138,240.00
EBHV grantee data quality progress table	17	4	4.25	289	$25.60	$7,398.40
Participant-home visitor relationship questionnaire	4,716	2	0.25	2,358	$11.25	$26,527.50
Home visitor-participant relationship questionnaire	4,716	2	0.25	2,358	$25.60	$60,364.80
EBHV grantee-partner network survey	142	2	0.42	119	$25.60	$3,046.40
Estimated Total				11,355		$256,850.70

A.13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

Not applicable.

A.14. Annualized Cost to Federal Government

The total cost to the federal government for the Cross-Site EBHV Evaluation under the terms of the contract to Mathematica is $3,764,167 for the three-year period. In particular, the estimated cost of the study’s information collection to the federal government—including designing and administering all data collection instruments, processing and analyzing the data, and preparing reports—is $2,573,600. This OMB package requests clearance for a three-year period during which data collection and reporting activities will occur. The average annual cost of data collection and analysis during this period is $857,866.67.

A.15. Explanations for Program Changes or Adjustments

None; this is a new collection.

A.16. Plans for Tabulation and Publication and Project Time Schedule

a. Analysis Plan

Grantees are implementing a range of activities to support different home visiting program models, targeting a range of populations, to be responsive to the needs of their local communities. Further, grantees are planning to use a variety of evaluation approaches to examine the impacts of the home visiting programs on children and families. As a result, grantee plans vary greatly across the 17 sites; thus, the Cross-Site EBHV Evaluation will use a mixed-method analysis approach.

Systems Change, Fidelity to Evidence-Based Models, and Costs of Home Visiting Programs Domains. Data related to the systems change, fidelity to evidence-based models, and costs of home visiting programs domains will be analyzed in order to describe the activities of grantees over the course of the project. The analyses will not focus on drawing causal conclusions but rather on systematically and clearly describing each domain and associations among domains. Grantee outcomes for each domain will be summarized using means, percentages, and ranges at a point in time (for example, annually) and over time. For example, in the systems change domain, we will examine the percentage of grantees working to change state-level systems. Additionally, using simple statistics (such as means and percentages), as well as more sophisticated techniques (such as factor analysis), we will create typologies and categories within each domain at a point in time as well as over time. Using these typologies, additional analyses will examine the associations among these three domains. More specifically, we will analyze the relationship between infrastructure changes; system attributes; program characteristics, including costs; and fidelity of implementation, accounting for differences in other grantee and program characteristics (such as risk level of client, region, economic circumstances). We will use statistical measures of association, such as correlations, to examine whether the outcomes within domains are associated with these characteristics of grantees. We will also use multilevel hierarchical linear models to perform growth curve modeling to investigate change in fidelity over time as a function of grantee, program model, service delivery site, and system level characteristics.

Family and Child Outcomes. To understand whether evidence-based home visiting programs impact families and children, the Cross-Site EBHV Evaluation proposes a systematic review of evidence of effectiveness of the programs as implemented within grantees’ local communities. Every grantee will measure family and child outcomes for common constructs such as parental depression, child health, and parenting, though the exact measures may vary by grantee. Then, the cross-site evaluation team will review evaluation designs to determine the local study’s level of evidence based on two criteria: (1) quality of the family and child outcome measures, and (2) evaluation study design and its implementation (to include data quality indicators). Each grantee’s evaluation will be grouped into one of three evidence groups:
(1) strong (such as well-implemented randomized controlled trials), (2) moderate (such as quasi-experimental designs), and (3) exploratory (may not meet the other two but provides information about whether results are consistent with the study hypotheses). Next, the systematic review will standardize effects across grantees to provide consistent measures across differing scales. Results will be presented across grantees to describe the level of evidence of the effects of home visiting programs on family and child outcomes.

Process Study. We will organize and synthesize qualitative data from site visits, the web-based data entry system, and grantees’ plans for the process study. Using the Atlas.ti (Scientific Software Development 1997) software package, the cross-site evaluation team will develop a structured coding system for categorizing the data, enter data into a database according to the coding scheme, and retrieve data that are linked to primary research questions and subtopics to facilitate theme identification. Our process will first use a within-case perspective followed by a cross-case perspective to identify themes and patterns that are discernable to an individual grantee, a set of grantees, or all grantees. We will also explore relationships across themes—for example, the kinds of implementation challenges grantees face and their staffing patterns and partnership arrangements. We will examine data for subgroups according to grantee characteristics (for example, home visiting model, population served) to compare, for example, how implementation varied (1) across grantees implementing different home visiting models, or (2) among grantees newly implementing a model, or (3) among grantees not offering direct services. We will also compute descriptive statistics (for example, means, percentages, ranges) to characterize implementation, such as the level of consensus among partners on the targeted outcomes (on a scale from one to five).

b. Time Schedule and Publications

During the time covered by this clearance, three project reports are planned. They will include key findings as data and analyses become available from each year of data collection. The first report, planned for late 2010, will present findings from the first round of site visits and available data concerning home visit program enrollment, service delivery, implementation, and evidence-based model fidelity. In late 2011, the project’s second report will present an analysis of systems change and implementation data along with analyses of home visiting program enrollment, service delivery, and costs. The third report (planned for late 2012) will present information from the second round of site visits with a detailed description of implementation, ongoing assessment of evidence-based model fidelity, and, if available, descriptive information on grantee-collected family and child outcome data.

A.17.   Reason(s) Display of OMB Expiration Date is Inappropriate

Approval not to display the expiration date for OMB approval is not requested.

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this data collection.

REFERENCES

Appleyard, Karen, and Lisa J. Berlin. “Supporting Healthy Relationships Among Young Children and Their Parents: Lessons from Attachment Theory and Research.” Durham, NC: Center for Child and Family Policy, Duke University, 2007. Available at [http://www.childandfamilypolicy.duke.edu/publications/policybriefs/files/eca/Attachment-final.pdf].

Berlin, Lisa J., Charles H. Zeanah, and Alicia F. Lieberman. “Prevention and Intervention Programs for Supporting Early Attachment Security.” In Handbook of Attachment: Theory, Research, and Clinical Applications, edited by Jude Cassidy and Philip R. Shaver. New York: Guilford Press, 2008.

Daro, Deborah. “Home Visitation: Assessing Progress, Managing Expectations.” Chicago, IL: Chapin Hall Center for Children, 2006.

Finkelhor, David. “Prevention of Sexual Abuse Through Educational Programs Directed Toward Children.” Pediatrics, vol. 120, no. 3, 2007, pp. 640–645.

Finkelhor, David, and Lisa Jones. “Why Have Child Maltreatment and Child Victimization Declined?” Journal of Social Issues, vol. 62, no. 4, 2006, pp. 685–716.

Foster-Fishman, Pennie G., Branda Nowell, and Huilan Yang. “Putting the System Back into Systems Change: A Framework for Understanding and Changing Organizational and Community Systems.” American Journal of Community Psychology, vol. 39, no. 3-4, 2007, pp. 197–215.

Scientific Software Development. “Atlas.ti: Visual Qualitative Data Analysis, Management, and Model Building in Education Research and Business.” Berlin, Germany: Scientific Software Development, 1997.

U.S. Department of Health and Human Services, Administration on Children, Youth, and Families. “Child Maltreatment 2007.” Washington, DC: U.S. Government Printing Office, 2009.

Wolfe, David A. “Services and Programs Proven to be Effective to Prevent Child Maltreatment and its Impact on the Social and Emotional Development of Young Children (0-5).” In Encyclopedia on Early Childhood Development, edited by Richard E. Tremblay, Ronald G. Barr, and Ray DeV Peters. Montreal, Quebec: Centre of Excellence for Early Childhood Development, 2004.

3 The EBHV initiative is designed to help grantees develop the infrastructure needed to support evidence-based home visiting programs. Infrastructure development involves building capacity in the areas of planning, operations, workforce development, funding, collaboration, communication, political support, and quality assurance. To develop infrastructure, EBHV grantees will work at multiple systems levels, including core home visiting operations, local organizations, and state or national agencies. To evaluate grantees’ efforts in the domain of systems change domain, the cross-site evaluation will track change over time and progress towards grantees’ intended changes in infrastructure capacity and achievement related to infrastructure development at multiple systems levels.

4 A subset of grantees implementing the Nurse-Family Partnership model will have the majority of the fidelity data transmitted by the NFP National Service Office’s Central Information System database.

5 The grantees’ data quality information will alert the cross-site evaluators to possible technical assistance needs concerning the family and child outcomes data collection. A grant requirement for EBHV grantees is to submit their local evaluation data to the National Data Archive on Child Abuse and Neglect (NDACAN), which is regular practice for CB/ACF/DHHS grants to facilitate ongoing research through data collection supported by federal dollars. In order for such data to be useful, it must be high quality and well measured. An important responsibility of the cross-site evaluation is to ensure the quality of this data and make it available for archiving.