Introduction

EVALUATION OF THE EARLY CHILDHOOD EDUCATOR
PROFESSIONAL DEVELOPMENT PROGRAM

OMB SUBMISSION PACKAGE
CLEARANCE REQUEST

MAY 9, 2007

Prepared By:
Child Trends
4301 Connecticut Ave NW, Suite 350
Washington, DC 20008

Prepared For:
Policy Program Studies Service
U.S. Department of Education
COR: James Maxwell

ORGANIZATION OF THIS DOCUMENT
The Policy and Programs Studies Service (PPSS), in the Office of Planning,
Evaluation and Policy Development in the Office of the Secretary of the U.S. Department
of Education (ED), requests clearance of new data collection instruments and notification
materials (see Appendices I through VII) for the evaluation of the Early Childhood
Educator Professional Development (ECEPD) program (see Appendix VI). This Office
of Management and Budget (OMB) submission package includes the following sections
in support of this request:
•
•
•
•
•
•
•
•
•
•

Overview of Study
Section A: Justification
Section B: Collection of Information Employing Statistical Methods
Appendix I: Project Director’s Interview, Project Characteristics Form, and
Evaluation Description Form
Appendix II: In-Depth Project Team Interview
Appendix III: In-Depth Evaluation Team Interview
Appendix IV: notification materials
Appendix V: matrix showing the constructs of the evaluation and the sources
proposed to collect the data
Appendix VI: legislation authorizing the ECEPD program and evaluation
Appendix VII: memo describing pilot results

OVERVIEW OF STUDY: STATEMENT SUPPORTING PART A
This overview provides a brief description of the study and the questions being
addressed in the evaluation, a description of the three data collection instruments being
submitted for clearance, and an outline of the study design and data collection
procedures.

Description of the Study
The Early Childhood Educator Professional Development (ECEPD) program is a
federal discretionary program that provides grants to partnerships working to improve the
knowledge and skills of early childhood educators working in communities with high
concentrations of children living in poverty. The ECEPD program is described in Public
Law 107-110, Title II, Part A, Subpart 5 Section 2151 (e) of the Elementary and
Secondary Education Act (ESEA) as amended by the No Child Left Behind Act of 2001
(see Appendix VI). The purpose of the ECEPD program is to enhance the school
readiness of young children, particularly disadvantaged young children, and to prevent
young children from encountering difficulties once the children enter school.
An ECEPD-funded partnership is made up of one or more entities from each of
the following categories: (1) institutions of higher education, or other public or private
entities that provide professional development for early childhood educators that work

with children from low-income families in high-need communities; (2) one or more
public agencies (including local educational agencies, State educational agencies, State
human services agencies, and State and local agencies administering programs under the
Child Care and Development Block Grant Act of 1990, Head Start agencies, or private
organizations); and (3) to the extent feasible, an entity with demonstrated experience in
providing training to educators in early childhood education programs concerning
identifying and preventing behavior problems or working with children identified as or
suspected to be victims of abuse. The ECEPD grants support activities that are designed
to deliver ongoing, intensive professional training, education and support for early
childhood educators. Activities may include, for example, professional development for
early childhood educators on the application of recent research on child, language, and
literacy development and on early childhood pedagogy or to work with children who
have limited English proficiency, children with disabilities, and children with other
special needs; professional development related to the selection and use of screening and
diagnostic assessments to improve teaching and learning; or, provision of training for
coaches or consultants who work with educators to help them apply what has been
learned in professional development activities to their daily practices in a classroom or
home-based care setting.
Child Trends has been contracted by ED to conduct an evaluation of the ECEPD
program. The purpose of the study is to describe, using a variety of data sources, the
structure of ECEPD partnerships, the characteristics and content of professional
development activities supported by ECEPD grants, and the characteristics of participants
in ECEPD activities. The study will also describe outcomes for participants and
outcomes for children to the extent that they can be summarized from grantee
evaluations. Finally, by aligning the descriptive findings with the five achievement
indicators that have been established for the ECEPD program, the study will identify
promising practices in professional development for early childhood educators. The
study will focus on ECEPD grantees from three of the cohorts (2003, 2004, and 2005).
Exhibit 1 presents the conceptual framework that will be used for the ECEPD
Evaluation. This framework specifies the main components of ECEPD projects and the
hypothesized mechanisms through which they will affect the stated outcomes of the
projects. The framework also provides a visual picture of how the ECEPD Achievement
Indicators developed by the Department of Education are aligned with the various
components of the evaluation.
At the far left side of the framework, the ECEPD Partnership Structure and
Characteristics provide the setting within which the Project Goals and
Implementation are developed. The evaluation will describe the types of ECEPD
partnerships and will examine how features of the partnerships relate to the various goals
that are set for the projects. Implementation of the project will also be assessed. For
example, issues such as success in recruiting educators from targeted groups, hiring and
training of project coaches, timely development of project materials, and fidelity of
instruction (that is, the degree to which the instruction or coaching matches the stated
goals and content of the curriculum) will be reviewed. The conceptual framework
acknowledges that functioning of the partnership and the goals and implementation of the

ECEPD projects will be influenced, in part, by the State and Local Policy Context
which will also be examined in the evaluation. These contextual features include the
demographic characteristics of the community or region included in the project scope as
well as policies providing incentives (or barriers) to professional development. For
example, state quality rating systems provide financial awards to programs (or directly to
educators) who meet certain requirements for professional development qualifications.

Exhibit I: Conceptual Framework for the Evaluation of the
Early Childhood Educator Professional Development Program

The next block of the framework specifies the Professional Development
Activities that are delivered to educators. The evaluation will focus on describing the
range of activities that are offered by the ECEPD projects. The following features are of
interest:
•

Format – the mechanism for delivering professional development to
educators, including classroom-based workshops or course with lecture and
discussion, on-site consultation or coaching, on-line workshops or courses,
supervised practicum, or a combination of two or three of these formats

•

Content – the substantive material that is delivered to educators, focusing in
particular on the degree to which the material is based in research, the breadth
and depth of material, and the coherence of the topics that are covered

•

Intensity – a combination of the number of hours of instruction offered as
well as the course/professional development requirements that are specified
for educators

•

Instruction – the characteristics and qualifications of those who provide
professional development and their ability to effectively engage those they
serve

•

Application to practice – the degree to which educators are supported in
translating what they have learned to their actual practice and skills in
classrooms or home-based programs

•

Accessibility – provisions offered by the ECEPD project to facilitate
educators’ participation in professional development including availability of
scholarships, career counselors, translation of materials, and instruction in
languages other than English.

The next block in the framework is Educator Outcomes. As shown in the
framework, educator outcomes are directly affected by the Professional Development
Activities as well as the characteristics of the educators themselves. The evaluation will
examine five types of educator outcomes:
•

Knowledge – understanding of the terminology and concepts of specific
content areas and of children’s development within and across content
areas/developmental domains

•

Skills – practices within the classroom or group, including implementation of
curriculum, set-up of the physical environment, routines and activities, and
interactions with children

•

Efficacy – a sense of enjoyment and competence with instructional practices
and content related to different developmental domains

•

Professional status – hours or credits of education attained, degrees and
certification achieved, wages and/or benefits received, or professional
recognition in a career lattice or quality rating system

•

Sustained Change – the degree to which positive outcomes are sustained
over time

The final block in the framework is Children’s Outcomes. These outcomes are
affected simultaneously by the educator’s knowledge and skills as well as by
characteristics of the children and their families. The evaluation will examine children’s
outcomes across developmental domains and will attempt to understand any subgroup

differences across projects in the children who are most likely to experience
improvements in their outcomes as a result of educator professional development.

Evaluation Topics
This section describes the questions being addressed in the ECEPD Evaluation.
These questions have been developed with input from PPSS and the Technical Working
Group convened for the ECEPD Evaluation.
Æ ECEPD Partnership Structure and Characteristics
•

Project Structure – What partners are included in ECEPD projects, and what
are their roles? What are the infrastructures for delivering professional
development that have been extended or created by the ECEPD projects?
How are ECEPD projects monitoring and evaluating the impact of their
projects? What strategies are ECEPD grantees using to sustain their projects
after the end of the grant? How are relationships across ECEPD partnerships
structured? What structural factors contribute to successful partnerships?
Which evaluation designs are most feasible and informative?

Æ Project Goals and Implementation
•

Project Goals – What are the major goals of the ECEPD projects?

•

Project Implementation – How do ECEPD projects adapt over time? What
common issues arise that require changes from the original implementation
plan? What strategies do ECEPD projects use to recruit educators? Are they
able to meet their targets? What are the sources of difficulty in meeting
targets when they occur? How do ECEPD projects recruit and retain staff?
What supports are most effective for full implementation of ECEPD projects,
and what factors are associated with delays and incomplete implementation?
How are projects documenting fidelity of implementation?

Æ State and Local Policy Context (Moderator) – What are the demographic
characteristics of the community/region? What state and/or local policies that
need to be taken into account to understand implementation and outcomes for
ECEPD grantees?
Æ Professional Development Activities
•

Format of Activities – What sets of professional development activities are
offered by ECEPD grantees? Can the various sets of activities be grouped to
identify distinct types of professional development strategies? How are the
professional development activities chosen or developed? How do projects
access the research in making their choices?

•

Content of Activities – What is the content of professional development
activities being implemented by ECEPD grantees? How is content
monitored?

•

Intensity of Activities – What is the duration and sequence of professional
development activities? Can a dosage metric be developed and compared
across

•

Instruction – Who provides professional development in ECEPD projects?
What is their level of training and education? To what extent is their
teaching/training monitored by the ECEPD project?

•

Opportunities for Application to Practice – What steps do ECEPD projects
take to ensure that new knowledge will be applied to practice?

•

Improving Accessibility – What supports are ECEPD grantees including in
their projects to help educators enroll and progress in their programs?

Æ Educator/Participant Characteristics (Moderator) – What is the age,
education, and experience level of participants in ECEPD projects? In what types
of early education and care settings do the educators work? What are the
characteristics of the children they serve? Does the program track/document
individual participants' progress through the program? What are the major
challenges faced by educators? How do ECEPD projects address and support
educators with particular challenges?
Æ Educator/Participant Outcomes
•

Knowledge – Do ECEPD projects change educators’ understanding of the
terminology and concepts in specific content areas and of children’s
development within these content areas? Are certain educators more likely to
improve their knowledge than others?

•

Skills – Do ECEPD projects change educators’ practices within the classroom
or group, including implementation of curriculum activities, making changes
in the physical environment and routines, and changes in the quality of
interactions? Are certain educators more likely to improve their skills than
others?

•

Efficacy – Do ECEPD projects change educators’ enjoyment of and sense of
competence in teaching specific content areas as well as understanding of the
importance of children’s learning in a particular area? Are certain educators
more likely to improve their efficacy than others?

•

Professional Status – Do ECEPD projects change educator’s professional
development status, including changes in the hours or credits of education
attained, degrees and certification achieved, wages and/or benefits received,

and professional recognition in terms of levels on a career lattice, a quality
rating system, or system of tiered reimbursement?
•

Sustained Change – Are changes in educators’ knowledge and/or practice
sustained over time? Which educators are most likely to improve their
knowledge, skills and/or professional development status?

Æ Child and Family Characteristics (Moderator) – What are the characteristics of
children/families that have implications for the implementation and outcomes of
the ECEPD projects?
Æ Children’s Outcomes
•

Cognitive – Do ECEPD projects improve children’s cognitive development?
Which children are most likely to experience improvements in their cognitive
outcomes? Is cognitive development more likely to be affected by educators’
professional development than other child outcomes?

•

Language and Literacy – Do ECEPD projects improve children’s language
and literacy development? Which children are most likely to experience
improvements in their language and literacy outcomes? Is language and
literacy development more likely to be affected by educators’ professional
development than other child outcomes?

•

Social/Emotional – Do ECEPD projects improve children’s social/emotional
development? Which children are most likely to experience improvements in
their social/emotional outcomes? Is social/emotional development more
likely to be affected by educators’ professional development than other child
outcomes?

•

Math – Do ECEPD projects improve children’s math? Which children are
most likely to experience improvements in their math outcomes? Are math
skills more likely to be affected by educators’ professional development than
other child outcomes?

Æ Promising Practices – What are the emerging patterns in the existing literature
regarding the professional development practices that have positive outcomes for
educators' knowledge and skills and child outcomes? How can the evidence from
ECEPD grantee evaluations be integrated with the existing literature in
professional development for early childhood educators to extend our knowledge?
What promising directions are emerging? Are certain professional development
strategies more likely than others to be linked to improved educator and/or child
outcomes?

Data Collection Instruments and Materials for Which OMB Approval Is
Being Sought

ED proposes to use semi-structured interviews to collect data from ECEPD
Project Directors and other partnership participants who can provide details about various
aspects of the ECEPD project. Child Trends will also request that a subset of projects
provide examples of materials that are used in their professional development activities.
These materials will be reviewed and coded by Child Trends, and a summary of findings
will be presented in the final evaluation report.

(1) Project Director’s Interview
The purpose of the Project Director’s Interview is to collect in-depth information
about aspects of the ECEPD project that are not readily available in other project
materials (such as the Annual Performance Report). Topics covered in the Project
Director’s Interview include Project Goals, Project Structure, Project
Implementation/Adaptation, Project Activities, Characteristics of Participants, and
Promising Practices (see Appendix I). All ECEPD Project Directors in the 2003–2005
grantee cohorts will be interviewed by Child Trends’ researchers via telephone to collect
the interview information.
The Higher Education Grantees (HEG) Project Director’s Interview of the Head
Start Training and Technical Assistance Quality Assurance Study served as a guide for
the structure and length of the ECEPD Project Director’s Interview. The two programs
are similar in that they ask questions of directors about the functioning of a program
aimed at increasing the knowledge and skills of early childhood educators. Several items
from the HEG Project Director’s Interview were selected to be reproduced in the ECEPD
Project Director’s Interview because of their previous clearance by OMB, their quality
and reliability, and because researchers may be able to compare participant responses to
these items to the responses of participants in programs from other national samples.
The evaluation construct matrix that Child Trends is using to develop the
interviews in this study is in Appendix V. The purpose of this matrix is to link each item
of the data collection instrument to the evaluation questions and, ultimately, to the
reporting requirements and programmatic goals of the ECEPD program.

(2) Project Characteristics Form
Child Trends staff will review all grantee’s Annual Performance Reports and fill
in a summary chart of project characteristics for each ECEPD grantee (see Appendix I).
Specifically, this form summarizes in a consistent manner information regarding the
demographics of the staff that provide professional development to early childhood
educators in each ECEPD program, the demographics of the program participants, and
the demographics of the children the early childhood educators serve. Given that
grantees emphasize different things in their performance reports, it is unlikely that review
of existing materials alone will result in comparable data across projects. Consequently,
at the conclusion of the Project Director’s Interview, Child Trends will ask Project
Directors a few follow-up questions to clarify and complete the Project Characteristics
Form, thereby ensuring that all projects have complete and current data about their

projects represented in the final report. Language is provided at the end of the Project
Director’s Interview to review and complete the Project Characteristics Form.

(3) Evaluation Description Form
Child Trends staff will review all grantee’s Annual Performance Reports and
Evaluation Reports and fill in a summary chart of descriptive features of each program
evaluation (see Appendix I). Specifically, this form summarizes in a consistent manner
information regarding research questions, research design, current analysis plan, and
study findings for key ECEPD constructs (e.g., educator knowledge, educator skills, etc.).
For each key construct, the form summarizes what measures were used to examine the
construct, including any adaptations for non-English speakers; the overall results of each
study’s findings with regard to the construct thus far; whether the study addresses
subgroup differences and if so the patterns for subgroups thus far; and whether the study
is examining sustained change over time in the construct, and if so the patterns that are
emerging thus far. Given that grantees emphasize different things in their Annual
Performance Reports, it is unlikely that review of existing materials alone will result in
comparable data across projects. Consequently, Child Trends will contact Project
Directors to clarify and complete the Evaluation Description Form, thereby ensuring that
all projects have complete and current data about their project evaluations represented in
the overall evaluation report. Language is provided at the end of the Project Director’s
Interview to review and complete the Evaluation Description Form.

(4) Project Team Interview
For a subset of up to nine grantees, Child Trends will conduct a group interview
with key members of the ECEPD project teams. Team members might include
curriculum developers, trainers, teaching faculty, coaches, and liaisons between
universities and programs. The key members to include in the interview will be
identified by the Project Director at the end of his/her interview. Topics covered in the
Project Team Interview include Project Structure, Project Implementation/Adaptation,
Project Activities, Characteristics of Participants, Educator Outcomes, and Promising
Practices (see Appendix II). An emphasis is placed on gaining more in-depth knowledge
on each of these topics, especially with regard to project activities, than can be obtained
from reviewing Annual Performance Reports.

(5) Evaluation Team Interview
For a subset of up to nine grantees, Child Trends will conduct a group interview
with key members of the ECEPD evaluation teams. (These grantees will correspond to
the same projects for which Project Team Interviews will be completed.) Topics covered
in the Evaluation Team Interview include Project Structure, Project Activities, Educator
Outcomes, Child Outcomes, and Promising Practices (see Appendix III). As with the
Project Team interview, the aim of the Evaluation Team interview is to gain in-depth
knowledge on each of these topics than can be obtained from other sources, especially
information about monitoring of the fidelity of implementation of the ECEPD program

and patterns of sustainability and subgroup differences they are finding with regard to
educator and child outcomes.

(6) Notification Materials
Child Trends has prepared notification materials addressed to ECEPD Project
Directors and partnership participants (see Appendix IV). The notification materials
consist of a letter and an information brochure about the ECEPD Evaluation. The letter
emphasizes that participation in the evaluation is specifically indicated in the program
application and requirements, and assures the participants that their involvement is
important to the study. The letter briefly describes the purpose of the evaluation,
highlights the OMB number, explains that clearance was obtained, provides contact
information for the ED staff and the contractor, and assures the participants that
individual responses will not be reported..
The information brochure is designed so that Project Directors and other
partnership participants can easily find general information about the purpose of the
evaluation and the data that are being collected. The brochure includes the name and
purpose of the program, the duration of the evaluation, Child Trends and ED contact
information, and the anticipated timeline of the evaluation activities and final report.

(7) Request for Project Materials
Child Trends will collect and review training materials used by a subset of
ECEPD projects selected for in-depth review. The training materials review will
supplement the interview data that are collected and will provide details about how
professional development activities are structured, the content of the activities and the
expectations for educators.
ECEPD project directors will be sent a letter and request form (see Appendix IV)
explaining the types of materials and information that they are being asked to send to the
Child Trends research team for coding and analysis. The letter provides an overview of
the purpose of the request and the types of materials that should be submitted. Project
Directors will be asked to complete a coversheet for each submission that notes the type
of material (for example, a course syllabus or degree requirements) and the source of the
material. These coversheets will help organize the submission and will provide the Child
Trends research team with the details needed to classify and review the submission.

Data Collection Plan
Child Trends plans to collect data from ECEPD projects in two waves (see
Exhibit II for an overview and schedule of data collection activities). Child Trends has
already pilot tested the interviews for length and clarity of the questions (see below for
further details). In the first wave of data collection, (scheduled to take place in the
months of July, August and September 2007, after clearance from OMB) Child Trends
will administer the Project Director’s Interview to all Project Directors from the 2003,

2004, and 2005 cohorts (n=18). The Project Directors will also be asked to verify that
information summarized about their project in the Project Characteristics Form and the
Evaluation Description Form is accurate and, when necessary, to complete missing
information. In the second wave of data collection, (scheduled to take place in
September and October, 2007) a smaller group of ECEPD projects (n=6-9) will be
intentionally selected for two In-Depth Interviews, one with members of the Project
Team (including, for example, curriculum coordinators, instructors, coaches/consultants,
and/or career counselors) and one with the Evaluation Team and collection of project
materials. Further details about data collection are described below.
Members of the Child Trends team who will be conducting the interviews and
taking notes will participate in an 12-hour training module (designed for experienced
interviewers) that involves 4 hours of general training/review on interviewing techniques
and 8 hours of project-specific training/review on the purpose and content of each
question on the interview protocols. Question-by-question descriptions will be developed
and reviewed in depth during the project-specific training to ensure that the interviewers
(all senior members of the team) have a common understanding of the purpose of each
question and probe. Once data collection begins, the team will conduct weekly
debriefing meetings to discuss responses to interview items and address any concerns
about the reliability and quality of interview items.

Exhibit II: Overview and Schedule of Data Collection Activities
Activity

Feb
2007

Mar

Apr

May

Jun

Jul

Pilot Testing of
Interviews
Memo on pilot testing
results
Interviewer Training
Notification Materials
sent to sites
Project Directors
Interviews, Completion
of Forms, 18 sites
Identification and
notification of sites for
in-depth interviews
In-depth Interviews,
6-9 sites
Materials Collection
6-9 sites

z

Aug

Sep

Oct
2007

Project Director Interview
During the months of March and April, 2007, the Project Director Interview was
pre-tested with 4 individuals who were past or current ECEPD Project Directors to test
for the clarity of interview questions, adequacy of probes, the length of the interviews and
to make any corrections prior to beginning interviews with ECEPD Project Directors. A
memo to ED was delivered in the beginning of May describing the outcome of the pilot
interviews and recommending edits and changes to the interview protocol (see Appendix
VII).
When protocols are finalized and clearance has been received, interviews will be
conducted over the phone. Project Directors will receive notification about the
evaluation and the request for an interview via email and U.S. mail. The research team
will call the Project Director to confirm receipt of the materials and to schedule a
convenient time for an interview.
Interviews will be conducted by senior members of the research team. A junior
staff will also be in attendance during each interview to assist with note-taking and, upon
completion of the interview, to debrief with the interviewer about the content of all
recorded answers. With the consent of the respondent, interviews also will be recorded
using a digital recorder. All interview notes will be completed within one day of the
interview and will be reviewed for clarity by another senior member of the interview
team. If it is determined that certain responses are unclear or incomplete, the Project
Director will be contacted to clarify responses or to provide further information.
After the interview responses are cleaned and edited for clarity, they will be coded
for key themes related to each of the evaluation topics. This first round of coding will be
used to identify and select 6-9 sites for the next phase of data collection. The selection of
projects will begin in mid August, 2007 and will continue until the completion of Project
Director Interviews. Projects will be selected for the follow-up data collection depending
on the rigor of their evaluation design and methods, the availability of evaluation data,
and identification of the project as one that can provide unique insights into the
evaluation topics because of the project structure, activities or feature of implementation.
Child Trends will confirm all choices of sites for in-depth data collection with ED prior
to contacting the sites.

In-Depth Interviews (Project Team and Evaluation Team) and
Materials Collection
During April and May, 2007, the In-Depth Interviews were pre-tested with one
Project Team and one Evaluation Team to test for the clarity of interview questions,
adequacy of probes, the length of the interviews and to make any corrections. A memo to
ED was delivered at the beginning of May describing the outcome of the pilot interviews
and recommending edits and changes to the interview protocols.

The In-Depth Interviews will also be conducted over the telephone using the
method described above for the Project Director’s Interviews though they will likely be
conducted with more than one person from the ECEPD project (with a maximum of 6
people on the call). Staff (and contact information) to be included in the In-Depth
Interviews are requested at the end of the Project Director’s Interview. Staff will be
contacted by telephone and email to set up the conference calls used to complete the
group interview (see Section B 2.a below for further details of data collection
procedures).