Pilot Results

Appendix VII
Memo Describing Pilot Results

MEMO
TO:

JAMES MAXWELL

FROM:

MARTHA ZASLOW, KATHRYN TOUT, TAMARA HALLE

SUBJECT:

REVISIONS OF ECEPD EVALUATION INTERVIEWS BASED ON PILOT WORK

DATE:

6/7/2007

CC:

This memo details the revisions we have made on the Director, Project Team, and
Evaluation Team Interviews based on the piloting of the measures we conducted. The
goal of these pilot interviews was to confirm that understanding of items by respondents
was as intended, and to minimize the burden on participants.
The Piloting Process. We conducted pilot interviews with four ECEPD Project
Directors: one from the 2003 cohort, and the three directors representing the most recent
ECEPD grantees (i.e., the 2006 cohort). We selected the 2006 ECEPD Project Directors
for the pilot so that we did not further reduce our participant pool for the evaluation study
itself. In addition, we interviewed the Project Team and the Evaluator that corresponded
to the 2003 ECEPD project for which we had completed a Director Interview. All
interviews were audio taped on a digital recording device. After completing each
interview, we asked the interviewees to evaluate the interview as a whole and also to
identify items that were redundant, difficult to respond to, and/or would require obtaining
information not readily at hand to answer the question. We used the version of the
interviews submitted with the OMB clearance package for this pilot work. Detailed
interview notes were created by taking notes on a computer during the call; researchers
went back to listen to the digital recordings of the interviews to clarify these notes and fill
in details, as necessary.
Findings of the Pilot Work. Our pilot work revealed that our interviews were taking
about 80 minutes to complete, which matched our assumptions and expectations. All
participants indicated that, for the most part, the questions we included in the interviews
were relevant to their experiences with the ECEPD programs. However, there were some
instances of redundancies across items, and need for clarification of terminology on a few
items. Where redundancies occurred, we eliminated unnecessary items. Where
clarification was needed, we inserted additional language or “probes” that the interviewer
could use if necessary during the telephone interviews.
We found that the timing of the interview made it difficult for some grantees to fully
answer the questions we had developed. Specifically, all three Directors in the 2006
cohort found it difficult to answer questions about the “lessons learned” from their
project because it was just too early in the process for them to provide good responses.
Nevertheless, we will be keeping these items in the interviews, because we feel that the
cohorts that comprise the evaluation will be able to adequately answer such questions (as

did the one Director from the 2003 cohort in our pool of pilot participants). This issue of
timing, however, made us aware that the final Evaluation Report will need to report
findings by cohort, precisely because the timing of the projects may affect their responses
to the questions.
When we spoke to the Evaluator from the 2003 ECEPD project, he found it difficult to
remember details of the evaluation report in order to respond to questions about the “most
surprising finding” in their study. He suggested that we ask the Evaluation Team to
review the most recent (or final) Annual Performance Report (APR) or Evaluation Report
prior to the phone interview with our team. We intend to include this additional request
in our protocol.
Revisions to Interviews. Specifically, the revisions we made to the interviews were as
follows:
ƒ

Eliminated questions that were redundant or did not seem to elicit the
information we were seeking, based on the review of the detailed interview
notes of the pilot interviews

ƒ

Reworded questions to avoid yes/no responses

ƒ

Made sure that all questions linked back to our conceptual model for the
evaluation and added a “rationale” statement after each question to identify
the linkage to the conceptual model

ƒ

Reordered questions, where necessary, to improve the flow of the interview

ƒ

Eliminated items that requested information we could possibly get from other
sources (e.g., review of APRs) with one exception – we kept the first question
that asks the Director to describe in his/her own words the goals of the
ECEPD project. We felt that there was a need to have an “icebreaker” that
allows them to start talking about their program, and this question provides
the needed impetus to discuss the project. We do, however, convey our
familiarity with the programs in the introduction to the call by informing the
interviewees that we have recently reviewed all APRs and Evaluation Reports.

An additional change we made to the interviews was eliminating the section on
“Finalizing Project Characteristics Form and Evaluation Description Form” that we had
placed at the very beginning of the interview. This section was to help fill in gaps in
summary documents we will be creating based on the review of grantee documents
provided to the Department of Education. We decided to eliminate this section because it
obstructed the flow of the interview and could potentially add up to 15 minutes to the
overall interview time, depending on the amount of clarification needed. Instead, we will
email the forms to the Director prior to the interview. However, rather than completing
them during the interview, we will briefly discuss the forms with the Director at the close
of the interview to answer any questions, to clarify what we are collecting and to
highlight the information that is still needed to complete the form. We will then ask that
the Director complete the form and email it back to the Child Trends team.