Health Resources and Services Administration
Maternal and Child Health Bureau
Evaluation of the State Early Childhood Comprehensive Systems
Grant Program
A. Justification
Circumstances of Information Collection
The Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) requests OMB approval for the evaluation of the State Early Childhood Comprehensive Systems (ECCS) Grant Program. This is a new information collection activity. ECCS was created to help States and jurisdictions develop and strengthen mechanisms to promote a systems approach to the design and delivery of services needed by young children and their families. Specifically, ECCS was designed to help public and private agencies, organizations, and programs that deliver services to young children and their families work together more effectively within a cohesive structure or system. ECCS builds on MCHB’s long history of supporting systems approaches through its Community Integrated Service Systems (CISS) Initiative. CISS was designed to provide funding to States and jurisdictions to promote the development of statewide networks of comprehensive community-based systems of services to ensure family-centered, culturally competent, coordinated care for all children and their families. The lessons learned from CISS and the release of MCHB’s Strategic Plan for Early Childhood Health in 2002, citing greater evidence of the ability of children’s early experiences to shape long-term outcomes, prompted MCHB to provide leadership in the development of cross-service systems offering comprehensive, integrated services to promote healthy child development.
Comprehensive systems are intended to help families seeking information, resources, or services to access them more easily and in a more timely way. ECCS grantees are required to work towards the development of systems comprised of five key components:
(1) Early care and education;
(2) Social-emotional development/mental health;
(3) Family support services;
(4) Parenting education; and
(5) Medical homes/health care.
Each participating grantee forms a multi-agency ECCS State Team that provides leadership and oversight in the design, implementation, and monitoring of a statewide ECCS initiative. These State Teams include representatives from both public and private agencies along with family members of young children. The grant guidance does not require that particular agencies be represented but instead it requires that the State Team include representatives from each of the five component areas and that efforts are made to include key partners. Grantees receive guidance, technical assistance, and other resources from:
HRSA’s Maternal and Child Health Bureau (MCHB);
John Snow Inc. the technical assistance contractor for HRSA/MCHB;
Project Thrive at the National Center for Children in Poverty which provides support on policy issues; and
The Federal Partners Work Group (Work Group) comprised of representatives from HRSA/MCHB and other Federal child and family service agencies.1
The Federal Partners Work Group meets monthly to discuss how to coordinate Federal early childhood programs and initiatives. The main goal of the Work Group is to develop a more integrated approach to early childhood across the participating agencies at the state level.
ECCS grantees are expected to focus on seven elements of system building with the goal of developing a system that encompasses the five component areas described above. These system building elements are:
Governance, which encompasses the structure and function of leadership of the State’s/jurisdiction’s early childhood system and the ECCS initiative, the initiative’s vision and goals, and representation of local entities;
Financing, blending or braiding funding sources, efforts to develop sustainable funding, and efforts to establish results-based funding;
Communications, efforts to increase policymakers’ and the public’s engagement in early childhood issues and tools for educating the public and providers about taking a systems approach;
Family leadership development, parents and families playing a lead role in policy and program planning, development, and implementation;
Provider/practitioner support, provision of training or technical assistance as well as encouraging supportive policies to improve the effectiveness and efficiency of the early childhood workforce;
Standards, includes the development of individual state standards for programs and providers against which progress is measured, and the creation of state-based streamlined administrative rules and processes across agencies; and
Monitoring/accountability, includes using indicators to monitor the well-being of children and families across the five component areas and addressing issues such as inter-agency data sharing and data development. These indicators are approved through the Title V Information System, OMB #0915-0172.
This implementation evaluation of the ECCS program will take place during the second round of implementation funding, which will cover the 2009-2012 period. The objectives of the evaluation are to:
Identify and study the strategies that grantees and partners are using to build comprehensive early childhood systems.
Measure the level of progress grantees have made towards addressing the overarching Federal goals and objectives for ECCS grantees and those of their statewide plans.
Assess the effectiveness of grantees’ implementation of early childhood systems development activities.
Assess the support offered by MCHB, ECCS technical assistance providers, and the Federal Partners Workgroup.
In addition, information from the evaluation will outline the achievements of ECCS and identify potential areas for improvement which will inform program planning and operational decisions for HRSA/MCHB and the ECCS grantees.
The evaluation is designed to collect data from all 52 grantees through five methods:
(1) Abstraction of administrative data from grant documents such as progress reports and continuation applications to establish understanding of each site, approval is not being sought for collection of data from administrative grant documents;
(2) Web-based survey of ECCS Coordinators;
(3) Web-based survey of selected State Team members;
(4) Key informant interviews with ECCS Coordinators;
(5) Web-based form for collecting data on grantee-selected indicators.
MCHB is seeking approval from the Office of Management and Budget (OMB) for four of these data collection methods. A brief description of the data collection activities for which OMB approval is being sought is included below.
Web-based survey of ECCS Coordinators (Attachment A). ECCS Coordinators will be asked to answer questions on the structure and function of their ECCS State Team, the nature of activities, and perceptions of progress made in achieving their outcomes. The survey consists of primarily close-ended questions that are designed to collect data from each grantee.
Web-based survey of selected State Team members (Attachment B). Five members of each State Team will be asked to complete a similar but shorter survey than the one completed by the ECCS Coordinator; the same content areas will be covered using fewer questions. The five team members will consist of two specified participants for each state and three chosen by the ECCS Coordinator. The specified participants are the State Head Start Collaboration Coordinator and the State Community Based Child Abuse Prevention (CBCAP) Initiative Coordinator. These participants are specified because they are charged with coordinating key aspects of the early childhood system and increased collaboration between these leaders and ECCS Coordinators has been a key focus of the ECCS initiative over the past two years. Head Start Collaboration Coordinators and CBCAP Coordinators participated with ECCS Coordinators in a national partnership meeting designed to increase collaboration between the programs. The ECCS Coordinator will be asked to identify three additional State Team members to participate in the survey. These will be State Team members who are most knowledgeable about the initiative and can best report on the functioning of the State Team and the nature of the Team’s activities.
Key informant interviews with ECCS Coordinators and selected State Team Members (Attachment C). Telephone interviews with ECCS Coordinators will be conducted to collect more descriptive information on how the State Team functions, what steps have been taken to integrate early childhood services in each of the seven system building elements, challenges and successes of implementation, and how the activities are designed to improve the lives of children and families.
Web-based Data Collection Form for Grantee-selected Indicators (Attachment D). ECCS Coordinators will also be asked to give information on three early childhood and family outcome indicators from previously approved measures from the Title V Information System (TVIS, OMB #0915-0172). States have also created their own indicators for early childhood and family outcome indicators; these indicators are not collected from HRSA/MCHB, they are at the discretion of the State to create and measure. The indicators are used to track the well-being of children and families in their state. In addition, they will also be asked to explain how the strategies they are carrying out under their ECCS initiative are intended to produce a change in the indicator. The purpose of this data collection form is to provide information on the extent to which grantees have been able to link their system-building strategies to these indicators of child and family well-being.
Data for the web-based surveys, key informant interviews, and indicators will be collected in two phases during 2010 and 2012. This two-stage process will provide a baseline understanding of each State’s Team and activities, as well as the initial status of early child and family outcomes. The 2012 data collection will capture any changes made to the structure and function of State Teams, the nature of activities, and progress made in achieving outcomes.
The data collection is authorized under Title V, Section 501(a)(3) of the Social Security Act as amended, (42 USC 701(a)(3)). The authorizing statue is included as Attachment E.
Purpose and Use of Information
This evaluation is designed to contribute to a comprehensive understanding of the planning, implementation, and effectiveness of the ECCS Grant Program. The evaluation will determine how the ECCS Grant Program has contributed to the development of comprehensive, integrated early childhood systems across participating States/Jurisdictions. In order to answer this overarching question, the evaluation will focus on identifying results of the established evaluation questions and sub-questions provided in Exhibit 1.
Exhibit 1: Key Evaluation Questions
1. What is the structure and function of each grantee’s State Team and ECCS initiative? |
|
2. How do the ECCS initiatives address each of the seven key elements of early childhood comprehensive systems? |
|
3. How have ECCS initiatives contributed to progress towards greater interagency collaboration and integration of services? |
|
4. What evidence have ECCS grantees collected that their initiatives are improving early childhood and family health and development outcomes? |
|
The results should describe whether and how the ECCS State Teams are progressing towards the initiatives overarching goals and the grantees’ and their partners’ perceptions of whether and how the support provided by HRSA/MCHB and the Federal Partners Workgroup is contributing to grantees’ progress toward these goals. The support that is being asked about includes funding for TA and resource centers, efforts to better coordinate TA support across agencies, and Federal efforts to improve overall coordination of early childhood programs. The Federal Partners Workgroup members will be informed about how their efforts to better coordinate Federal early childhood programs and technical assistance is being perceived at the state level. The findings will also have implications for other Federal agencies interested in launching similar systems-building efforts in the future.
Use of Improved Information Technology
The data collection methods that will be utilized include two web-based surveys, telephone interviews, and a web-based data collection form for indicators. Every effort has been made to structure the collection of data so as to minimize the burden on respondents:
Web-based surveys. The two surveys, one for ECCS Coordinators and one for selected State Team members, will be administered through a secure Web site that will permit respondents to complete each instrument at a time that is convenient for them. Respondents who begin the survey and are unable to complete it will be able to save their responses and resume work on the survey at a different time. Also, the survey questions will be primarily close-ended with categorical responses which will help decrease the amount of time required to complete the surveys. The web-based format will incorporate skip patterns that ensure that respondents automatically skip past sections of the survey that are not relevant to their experiences.
Telephone interviews. The interviews are designed to collect more detailed, nuanced information about ECCS planning and implementation from ECCS Coordinators than will be captured by the web-based survey. Training will be provided to the team of skilled interviewers to help probe for additional information when needed and to help progress through the questions as quickly as possible. In addition, interviewers will provide respondents with discussion topics in advance of the call and accommodate their schedules to conduct calls at convenient times.
Web-based Data Collection Form. The data collection form for indicators will also be administered via a secured website that allows respondents to exit the form at any point and return later to complete the instrument. Grantees are only being asked to report on three indicators because the purpose of this part of the data collection is to illustrate how grantees have linked their system-building strategies to indicators of child and family well-being.
Efforts to Identify Duplication
The data requested for this evaluation are unique to the ECCS programs, required by statute, and are not available elsewhere. The exception is the data used from state level indicators previously approved by OMB # (0915-0172).
Involvement of Small Entities
This project does not have a significant impact on small entities.
Consequences if Information Collected Less Frequently
Without collecting these data, MCHB will not have access to a comprehensive assessment of the current planning, implementation, and monitoring of the ECCS Grant Program, as well as key drivers of comprehensive systems-building. Specifically, the evaluation will provide an analysis of the progress grantees have made during the second phase of funding and the role that Federal partners played in achieving the goals of ECCS. In addition, the results will identify key challenges and areas of weakness that the MCHB, Federal Partners Workgroup and grantees can use to inform future program planning efforts.
Consistency with the Guidelines of 5 CFR 1320.8(d)
This data collection request is fully consistent with the guidelines in 5 CFR 1320.8(d).
Consultation Outside the Agency
The notice required in 5 CFR 1320.8(d) was published in the Federal Register on July 23, 2009, Volume 74, Number 140, pages. 36490-36491. The 30 day notice was published in the Federal Register on March 24, 2010. There were no public comments.
The instruments were shared with a Grantee Advisory Group consisting of five ECCS Coordinators. These coordinators were selected based on their longevity in the position and their knowledge and experience of the initiative. The names of the participants on the Grantee Advisory Group are included in Attachment F.
Two coordinators completed a pretest of the coordinator survey and interview and two ECCS State Team members completed the Team Member Survey. The comments and suggestions of these groups were incorporated into revised instruments. Most of the changes consisted of clarification of question wording. Some questions were cut from the interview protocol because they were determined not to produce useful information. Some questions were also cut from the Team Member Survey because it was determined that they were not appropriate for a team member. More specific wording was needed which involved creating separate questions for different types of standards.
Payment or Gift to Respondents
Respondents will not be offered any gift or payment.
Assurance of Confidentiality
For each instruments respondents are being informed that their agency’s name and location, and general job title (e.g. State ECCS Coordinator) will be identified in reports prepared for this study and in data files provided to HRSA. They are also being told that none of their responses will be released in a form that identifies them or any other staff member by name. These respondents are being interviewed as part of their professional position, therefore personally identifiable information related to any health information will not be collected.
Respondents will be asked to provide their name solely for the purposes of following up on answers that are unclear or confusing. These names will not be included in any data files or tables provided to HRSA and will only be available to the contractor.
Questions of a Sensitive Nature
There are no questions of a sensitive nature. The questions are designed to solicit information solely regarding the planning, implementation, and effectiveness of ECCS.
Estimates of Annualized Hour Burden
In Exhibit 1, estimates of the collection burden on participants from each category of respondent are provided. The ECCS Coordinator web-based survey will take approximately 45 minutes. This is based on an estimate of completing approximately 3 questions per minute. The pretests confirmed the initial burden estimates in that coordinators reported taking less than 45 minutes; therefore the burden estimate remains at 45 minutes to include any preparation time to take the survey.
The State Team member survey is estimated to take approximately .30 of an hour. This assumes that they will be able to complete 3 questions per minute. This estimate was confirmed by the pretest.
The telephone interview was estimated to take approximately 1.5 hours based on the contractor’s experience with similar instruments. This was confirmed by the pretest. Respondents will be provided with the topics covered by the questions ahead of time and are expected to spend .25 hours reviewing materials ahead of time. The Web-based Data Collection Form for Grantee Selected Indicators is expected to take 1.5 hours to complete. This includes .5 to .75 hours to enter the information into the form along with additional time to organize or compile the requested data. This estimate is based on the contractors experience with similar data collection tools and was confirmed by the comments of the Grantee Advisory Group that reviewed the draft instrument. Draft protocols may be found in Attachments A-D. The total number of burden hours for all data collection efforts is 286.
The following table identified the annualized burden estimate:
Exhibit 2. Estimated Burden Hours
Form |
Type of Respondent |
No. of Respondents |
No. of Responses per Respondent |
Average Burden Hours Per Response |
Total Burden Hours |
Web-based Survey with ECCS Coordinators |
ECCS Coordinators |
52 |
1 |
0.75 |
39 |
Web-based Survey with Selected Team Members |
State Team members |
260 |
1 |
0.3 |
78 |
Telephone Interview |
ECC Coordinators |
52 |
1 |
1.75 |
91 |
Web-based Data Collection Form for Grantee Selected Indicators |
ECCS Coordinators |
52 |
1 |
1.5 |
78 |
TOTAL |
-- |
416 |
--- |
--- |
In Exhibit 3, we present the estimated burden cost for the data collection effort. The total estimated annualized cost to the respondents is $11,033.88. This cost estimate was calculated based on the total respondent hour burdens noted in Exhibit 1. The wage rate is the median wage rate for medical and health service managers according to the Bureau of Labor Statistics Occupational Wage Statistics.
Exhibit 3. Estimated Burden Cost to respondent
Type of Respondent |
Total Burden Hours |
Average Hourly Wage Rate |
Total Hour Cost |
ECCS Coordinators (Web-based Survey) |
39 |
$38.581 |
$1,504.62 |
Selected ECCS State Team Members (Web-based Survey) |
78 |
$38.581 |
$3.009.24 |
ECCS Coordinators (Telephone Interview) |
91 |
$38.581 |
$3,510.78 |
ECCS Coordinators (Web-based Data Collection Form for Grantee Selected Indicators) |
78 |
$38.581 |
$3,009.24 |
TOTAL |
286 |
--- |
$11,033.88
|
1 Based on median hourly wage for medical and health service managers, “May 2008 National Occupational and Wage Estimates”, United States, U.S. Department of Labor, Bureau of Labor Statistics.
Estimates of Annualized Cost Burden to Respondents
There are no capital or startup costs associated with data collection.
Annualized Cost to the Government
The total value of the contract between MCHB and the contractor is $983,630 over the course of one base year and three option years. Of that amount, approximately $328,292 will be used to cover the costs of the data collection and analysis, which includes direct labor, fringe, overhead, and fees. This is an annualized cost of $109,430.78 for contracted data collection. In addition, the estimated costs are for several Federal employees involved in the oversight and analysis of information collection, amounting to an annualized cost of $6,500 for Federal labor. The total annualized cost for the assessment is therefore $334,792.
Changes to Burden
This is a new data collection activity.
Plans for Tabulation and Publication and Project Time Schedule
Data for the evaluation will be collected in two phases during Option Years 1 and 3. Initial analysis will begin shortly after data is collected from the web-based surveys, and will continue periodically throughout the project period to produce two interim briefings after the first and second years of the evaluation and the final report upon completion of the evaluation. The final report will be shared directly with the grantees. Information will be collected over a 29-month period following OMB approval. Exhibit 4 provides a schedule of data collection, analysis, and reporting following OMB approval. The remainder of this section describes the analytic techniques that will be employed.
Exhibit 4. Timetable for Data Collection, Analysis, and Publication
Activity |
Estimated Start Date |
Estimated End Date |
PHASE I DATA COLLECTION: OPTION YEAR 1 |
||
Web-based survey with ECCS Coordinators |
1 month following OMB approval |
3 months following OMB approval |
Web-based survey with selected ECCS State Team members |
1 month following OMB approval |
3 months following OMB approval |
Tabulation of all Web-based survey results |
4 months following OMB approval |
5 months following OMB approval |
Telephone interviews |
3 months following OMB approval |
5 months following OMB approval |
Indicator Data Reporting Form data entry |
1 month following OMB approval |
3 months following OMB approval |
Analysis of all phase I data |
6 months following OMB approval |
8 months following OMB approval |
PHASE II DATA COLLECTION: OPTION YEAR 3 |
||
Web-based survey with ECCS Coordinators |
25 months following OMB approval |
28 months following OMB approval |
Web-based survey with selected ECCS State Team members |
25 months following OMB approval |
27 months following OMB approval |
Tabulation of all Web-based survey results |
28 months following OMB approval |
29 months following OMB approval |
Telephone interviews |
27 months following OMB approval |
29 months following OMB approval |
Indicator Reporting Form data entry |
25 months following OMB approval |
27 months following OMB approval |
Analysis of all phase I and II data |
30 months following OMB approval |
32 months following OMB approval |
DEVELOPMENT OF REPORTS |
||
Development of first interim briefing report |
9 months following OMB approval |
10 months following OMB approval |
Development of second interim briefing report |
21 months following OMB approval |
22 months following OMB approval |
Development of final report |
30 months following OMB approval |
34 months following OMB approval |
Data analysis will be descriptive. More details on the analysis are provided in Section B.
The data analysis results will be presented in a final report. This report will include:
Descriptive data on the activities undertaken by State Early Childhood Teams, categorized by one of the five core elements (page 1) of an early childhood system they fall under;
An assessment of progress that State Early Childhood Teams made during the second phase of ECCS funding;
An assessment of the activities of the Federal Partners Early Childhood Workgroup, including the grantees and State Team perceptions of its success in promoting the delivery of integrated technical assistance services and in developing collaborative initiatives designed to promote early childhood systems building;
Analysis of the extent that States are able to provide information on progress implementing improvements in the core elements and to what degree this progress varies by element with possible explanations for why this is the case; and
An analysis of the indicator data provided (previously approved indicators through TVIS OMB #0915-0172). This will include descriptive information on the types of indicators that States selected, and how State systems-building strategies were designed to improve results for children and families.
Exception for Display of Expiration Date
All data collection materials will display the OMB expiration date.
Certifications
HRSA certifies that the collection of information encompassed by this request complies with 5 CFR 1320.9 and the related provisions of 5 CFR 1320.8(b)(3).
Attachments for Supporting Statement
Authorizing Statue
Grantee Advisory Group
1 The FPWG includes: HRSA’s Maternal and Child Health Bureau; the Administration for Children and Families’ Child Care Bureau and the Office of Head Start; the Center for Mental Health Services Mental Health Promotion Branch; Office of Assistant Secretary for Planning and Evaluation; Office of Special Education Programs Early Childhood Team; the Centers for Disease Control’s Child Development Studies Team; and Office of Juvenile Justice and Delinquency Prevention.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | “Evaluation of the State Early Childhood Comprehensive Systems Grant (ECCS) Program” |
| Author | Amy Brown |
| File Modified | 0000-00-00 |
| File Created | 2021-02-02 |