supporting statement part B July 07

SECTION B: COLLECTION OF INFORMATION EMPLOYING STATISTICAL
METHODS
1.

Universe of Respondents

The universe of respondents for this study includes staff in the 18 ECEPD
projects funded in 2003, 2004, and 2005. Data collection will focus on the Project
Directors for all 18 Partnerships. In a subset of up to nine ECEPD projects, selected for
availability and rigor of evaluation data, additional staff will be asked to participate in
interviews with the research team.

2.

Procedures for Collection of Information
a. Data Collection

As noted earlier, Child Trends proposes to collect the data from all 18 ECEPD
Project Directors and key members of up to nine ECEPD grantee project and evaluation
teams using telephone interviewing.
Child Trends plans a comprehensive approach to data collection, tracking, and
follow-up:
(1)

Prepare materials—letter and notification packet. The materials will
include information about the telephone interviews with the directors
as well as a brochure that provides more information about the
evaluation.

(2)

Call all ECEPD Project Directors to confirm the receipt of the
materials and to set up a time for telephone interviews.

(3)

Send email reminders to ECEPD Directors to confirm date and time of
telephone interview and to provide copies of the Project
Characteristics and Evaluation Description Forms.

(4)

Conduct telephone interviews with ECEPD Directors (including a
discussion about how to complete the Project Characteristics Form and
the Evaluation Description Form). At the end of these interviews,
names and contact information are requested for key members of both
the project and evaluation teams.

(5)

Initiate follow-up activities.
(a)

With input from ED, Child Trends will select up to nine ECEPD
projects that will be asked to participate in in-depth interviews
with key project and evaluation team staff and from whom
project materials will be requested.

(b)

Initial email contact will be made with key project and
evaluation team members to find an appropriate time to set up a
conference call/interview with each team. The brochure about
the ECEPD evaluation will be attached to the email to provide
orientation to the project and their role in it. Contact information
for Child Trends’ research team will be provided.

(c) Telephone calls will be made to key staff who have not
understood the process, do not have email, or need
encouragement.
(d)

Email or phone call reminders will be sent prior to the
conference call, which will include the call-in number for the
conference call/interview.

(e)

Complete the In-depth Project Team and In-Depth Evaluation
Team interviews at times convenient to the team members.

(f)

Project Directors for the programs selected for in-depth study
will be sent the letter requesting additional project materials for
review. Telephone reminder calls will be made to any Project
Directors who do not send back the requested materials by the
deadline noted in the letter.

During the Director Interview administration and follow-up activities, Child
Trends will provide biweekly updates by email to the COR. Each update will list the
number and percent of respondents by ECEPD grantee and will document each step that
has been taken to assure that a minimum of 15 programs participate (though the goal is to
secure participation among all 18 grantees). Child Trends will seek approval from the
COR to begin coding data for analysis after when at least six Project Director Interviews
have been completed so that selection of sites for the in-depth interviews can begin.
When all interview data have been collected, Child Trends will prepare a summary on the
quality of the data for the COR before qualitative analysis begins.

b. Data Analysis
The ECEPD Evaluation uses a comparative case study design to examine the
research questions described above in the Overview. The study relies primarily on data
from interviews with key informants in ECEPD projects as well as from a review of
select materials collected from each project. Data will be organized and analyzed using
QSR N6 software for qualitative data analysis. Because the interview data gathered for
the ECEPD Evaluation will be structured by the interview process itself (each interview
will follow the same set of questions), initial coding of the data will be a straightforward
entry of information by site and by interview question topic. In addition, using the data
from the first few Project Director Interviews as a guide, further codes for emerging
themes will be developed and applied to the data. One of the important features of N6 is
the ability to modify and apply new coding of throughout the period of data analysis.
Reliability meetings will be conducted frequently while coding is launched to ensure that

key themes are coded similarly across researchers. Once codes have been applied
systematically to the data, N6 will be used to retrieve data on the evaluation questions
and to facilitate analysis and writing. Senior researchers will look in particular for links
between outcomes for educators and for children that vary by features of the project
structure, implementation context and type(s) of professional development activity
offered.

3.

Method for Maximizing Response Rate

Because of the small number of ECEPD projects (18) in the 2003, 2004 and 2005
cohorts, the evaluation team will employ a number of strategies to ensure that all ECEPD
Project Directors participate in the first phase of the data collection and that those sites
selected for further data collection (6-9 sites) continue to fully participate in the study.
Prior to the onset of data collection, in June, 2007, the evaluation team will send
notification materials to Project Directors that introduce the study and highlight the
importance of full participation. One week after the delivery of notification materials, the
Child Trends team will make personal phone calls to establish contact with the Project
Directors and will follow up with electronic mail messages to Project Directors to
encourage their participation. Reminders of interview dates and participants will be sent
via email. All interview participants will receive an electronic copy of the interview prior
to the interview date so that they can prepare responses and facilitate completion of the
interview in a timely manner. If email contact is not effective in scheduling interviews,
the Child Trends team will make phone calls offering to schedule the interviews at any
time that is convenient for the respondent (including evenings and/or weekends, if
necessary). Phone messages and electronic mail communication will be cordial and
respectful and will convey appreciation of the time that is spent on the study.

4.

Test Procedures

The Project Director Interview, In-Depth Project Team Interview and In-Depth
Evaluation Team Interview were piloted in March and April, 2007 with 4 individuals
who are past or current ECEPD Project Directors, 1 Project Team and 1 Evaluation Team
to test for the clarity of interview questions, adequacy of probes, and the length of the
interviews. The pilot interviews were conducted by a senior member of the evaluation
team who has experience conducting semi-structured interviews. Modifications were
made to the interview protocol based on feedback from pilot participants and from a
review of the responses. A memo on the pilot testing results is included in Appendix VII.

5.

Names of Statistical Consultants

The Evaluation of the ECEPD Program is being conducted by Child Trends. Dr.
Martha Zaslow serves as Project Director, Dr. Kathryn Tout serves as Project Manager, and
Dr. Tamara Halle serves as a Senior Task Leader on the Evaluation. The data collection
instruments and data analysis plan were developed by the Child Trends’ team. Child Trends
has also consulted with Dr. Jodi Sandfort, Associate Professor at the Hubert H. Humphrey
Institute of Public Affairs on the design and qualitative data analysis strategy used in the
Evaluation.