Final Macro Report

Summary of Findings and
Recommendations:

Cognitive Testing of
Proposed Items for the
Beginning Teacher
Longitudinal Survey
(BTLS)
Submitted to:
United States Census Bureau

August 2009

Table of Contents
Background ....................................................................................................................................1
Summary of Methodology .............................................................................................................1
Summary of Participant Feedback and Recommendations ......................................................2
First Phase of Testing:
Item 1 (Return to 2007-08 school).............................................................................................4
Item 2 (Changes in current school) ............................................................................................5
Item 3 (Teaching in same state) .................................................................................................6
Item 4 (Private/public school) ...................................................................................................7
Item 5 (Factors influencing decision to return) .........................................................................8
Item 6 (Most important reason for returning) ..........................................................................12
Item 7 (Completed a teacher preparation program) .................................................................13
Item 8 (Completed an alternative certification program) ........................................................14
Item 9 (Renewed teaching certificate) .....................................................................................15
Item 10 (Receieved endorsement to certificate) ......................................................................16
Item 11 (Completed advanced degree) ....................................................................................17
Item 12 (Specify degree field) .................................................................................................18
Item 13 (Assigned as a mentor) ...............................................................................................20
Item 14 (Number of teachers mentoring) ................................................................................21
Item 15 (Training related to mentoring) ..................................................................................22
Item 16 (Preparedness to be a mentor) ....................................................................................23
Item 17 (Certification activities in teaching assignement) ......................................................24
Second Phase of Testing:
Item 1 (Reason for reduction in force) ....................................................................................27
Item 2 (Agreement with statements about principal) ..............................................................28
Item 3 (Satisfaction with principal) .........................................................................................31
Item 4 (Input in choosing mentees) .........................................................................................32
Item 5 (Extent to which principal works with teachers to choose mentees) ...........................33
Item 6 (Extent to which principal discusses mentoring relationship with teachers) ...............34
Item 7 (Frequency with which principal discusses mentoring relationship with teachers) .....35
Item 8 (Existence of strong support system) ...........................................................................36
Item 9 (Teacher leadership activities)......................................................................................37

Appendix A: Interview Protocol for First Phase of Testing
Appendix B: Interview Protocol for Second Phase of Testing

Background
In the spring of 2009, the Census Bureau contracted with ICF Macro, a research and evaluation
company in Calverton, MD, to plan and carry out a series of cognitive interviews with current teachers
in K-12 schools. The purpose of these interviews was to gather feedback on a number of proposed
questions for the Beginning Teacher Longitudinal Survey (BTLS), which is a national survey that will
be administered for the first time during the 2009-10 school year by the National Center for Education
Statistics (NCES) and the Census Bureau.
This report is a summary of the feedback that ICF Macro received from participants, as well as the
methodology that was used in conducting the interviews. The report also provides Macro’s
recommendations for revisions to the proposed BTLS items.

Summary of Methodology
Research Timeline
The interviews were carried out in two separate phases. The first phase focused on testing a set of 17
items with teachers who had returned to teaching after a short time away from the field. The second
phase tested an additional 9 items with teachers who had either a) served as a mentor teacher or b)
taught at a school affected by a reduction-in-force.
The first phase of interviews began in May, and findings were presented to NCES and the Census
Bureau on June 16. The second phase began in early June, and ICF Macro presented findings on July
7.
Description of Recruiting Process
To facilitate recruitment of participants, the Census Bureau provided Macro with a list of schools that
were included in the sampling frame of the Schools and Staffing Survey (SASS) but had not been
sampled for SASS. Macro then created a stratified random list of schools by region and grade level,
and researched the names and emails of five teachers in each school using the school’s online website.
If a school did not have a website, the school was removed from the sample and replaced with another
school in the region. The five teachers in each school were then sent an email which indicated that
they should forward the message to any colleagues that they believed might be qualified to participate
in the study. To qualify for the first phase of the study, teachers had to be current K-12 teachers, have
left teaching for three years or less, and returned to teaching within the last four years.1 To qualify for
the second phase, teachers had to be current K-12 teachers and have either a) served as a mentor
teaching or b) taught at a school affected by a reduction-in-force.
In addition to using the list provided by the Census Bureau, email invitations were sent to an ICF
Macro internal database of educators. The email sent to these participants also asked them to forward
the message to colleagues who they thought might qualify for the study.
1

The recruiting criteria were later updated to also require that participants have less than three years of teaching experience.

ICF Macro, an ICF International Company

Page 1

Description of Participants
During the first phase of research, ICF Macro conducted a total of 16 interviews with returning
teachers. Seven of these teachers taught at the high school level, four at the middle school level, two at
the elementary/middle level, and three at the elementary level. Five of the 16 teachers had been
teaching for three years or less. Participants represented eight different states: California, Georgia,
Louisiana, Maryland, New Jersey, Pennsylvania, South Carolina, and Virginia.
During the second phase, ICF Macro conducted interviews with 12 additional teachers. Of these
teachers, all but one had experience as a mentor teacher, and five had taught at a school that had
experienced a reduction in force. Five taught at the high school level, one at the middle school level,
and six at the elementary level. Eight different states were represented (California, Connecticut,
Georgia, Maryland, Massachusetts, New Jersey, North Carolina, and Virginia), as well as the District
of Columbia.
Structure of Interviews
Each interview was approximately 45 minutes long and was conducted by phone. Prior to each
interview, the participant was emailed a copy of the proposed items and told to print them out but not
to read them. During the interview, participants were asked to answer each of the proposed items as
they normally would if they were answering a paper survey. As they answered each item, they were
asked to “think aloud”—that is, to describe out loud what they were thinking as they read and
answered the question and point out anything that surprised or confused them. After the participant had
answered a set of items, the interviewer would then ask a series of follow-up questions or probes, as
well as asking for clarification of responses as necessary. If a skip pattern caused a participant not to
respond to an item, he or she was later asked to review the item and provide feedback. A copy of the
protocols for the two phases of the study are included as Appendices A and B to this report.

Summary of Participant Feedback and Recommendations
The following section of this report summarizes the results of this study. For each item, we provide the
wording of the question being tested, a list of relevant findings, and then our recommendations for how
the item could be improved.

ICF Macro, an ICF International Company

Page 2

First Phase of Testing
(May/June 2009)

ICF Macro, an ICF International Company

Page 3

ITEM 1

You were teaching in [school] during the 2007-08 school year. Did you return to that school?
__ Yes  GO TO Item 2.
__ No  GO TO Item 3 below.

Findings:


All respondents thought this question was clear.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 4

ITEM 2
Have there been any changes in your current school, such as a change in name, location, or
grades offered?
__ Yes  GO TO Item 5 below.
__ No  GO TO Item 5 below.

Findings:


Almost all the teachers who answered this question thought it was clear. One teacher hesitated
in answering this question as she wasn’t sure if the question referred to a change in grade levels
or in the grading system. She eventually decided the question referred to grade levels.



Only one teacher answered this question incorrectly. She responded “Yes” to this question
even though there had been no changes in grade levels offered at her school. She incorrectly
thought the question was asking if there had been any changes in the grade levels she taught.

Recommendations:


Change the word “grades” to “grade levels”—“Have there been any changes in your current
school, such as a change in name, location, or grade levels offered?”

ICF Macro, an ICF International Company

Page 5

ITEM 3
Are you currently teaching in the SAME STATE as you were during the 2007-08 school year?
__ Yes  GO TO Item 4.
__ No  GO TO Item 4.

Findings:


All respondents thought this question was clear.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 6

ITEM 4
Which of the following describes the change from your 2007-08 school to your current school?
(For this question, all charter and Bureau of Indian Education [BIE]-funded schools are
considered public schools.)
*Mark (X) only one box.
__ Moved from a PUBLIC school to another PUBLIC school in the SAME SCHOOL
DISTRICT.
__ Moved from a PUBLIC school to another PUBLIC school in a DIFFERENT SCHOOL
DISTRICT.
__ Moved from a PUBLIC school to a PRIVATE school.

Findings:


Almost all respondents who answered this question thought it was clear. One teacher was
confused about whether a charter school was a public school, and did not see the sentence in
the question that addressed this issue. As this teacher had never taught at a charter school, her
misunderstanding did not affect the accuracy of her response.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 7

ITEM 5
Indicate the level of importance EACH of the following played in your decision to return to
the position of a pre-K-12 teacher.
*Mark (X) one box on each line.
The following scale is used for the items below: 1) Not at all important, 2) Slightly important, 3)
Somewhat important, 4) Very important, 5) Extremely important
If any of the reasons for returning to teaching do not apply to you, mark “1” as not at all
important.
I returned to the position of a pre-K-12 teacher:
1

2

3

4

5

a. Because I had a change in residence or wanted to take a job more
convenient to my home.
b. Because my maternity/paternity leave ended or I no longer needed to
stay at home with my children.
c. Because my health or the health of a loved one no longer required
me to be out of teaching.
d. Because I decided I was not ready to stop teaching.
e. Because I passed the required test(s).
f. Because I was offered the grade level or subject area that I wished to
teach.
g. Because I needed the income to meet my financial obligations (e.g.,
rent, loans, credit card payments).
h. Because my current school or district offered at least partial
forgiveness of my student loans.
i. Because I needed the health benefits.
j. Because I wanted the retirement package.
k. Because I could continue receiving my teacher retirement benefits.
l. Because I was given a housing incentive by my current school.
m. Because I wanted a higher standard of living than my household
income provided.
n. Because I wanted job security.
o. Because I decided to pursue a pre-K-12 teaching position as a career.
p. Because I completed the coursework I was pursuing.
q. Because a part-time teaching assignment became available.
r. Because I liked the school schedule/calendar.
s. Because I obtained a position in a school with desirable
characteristics.
t. Because I was able to maintain my seniority.
Other factors
u. Because of other factors not included in previous items a – t
 please specify:

ICF Macro, an ICF International Company

Page 8

Findings:
Instructions and Layout


The instructions for the question asked respondents to select “1” if a particular reason did not
apply to them. However, two respondents commented that assigning the value of “1” to two
different values (“not at all important”, and “does not apply”) seemed inappropriate.

Individual Items


Part (d): Respondents thought this question meant that an individual took a break from
teaching and now wanted to return.



Part (i): Respondents understood this item, and did not feel it was too specific. They felt that
other important teacher benefits, including retirement benefits and the school
schedule/calendar, were already covered in other parts of the question.



Part (j): All respondents understood that this question referred to the desirability of the state’s
retirement package for future use.



Part (k): Only half (8) of all respondents understood that this question was intended for
teachers who officially retire but then come back to teach in some capacity. Four respondents
were not currently receiving retirement benefits, but provided a rating for this factor greater
than 1 (which was clearly not what the item intended).



Part (l): All respondents understood the term “housing incentive.”



Part (m): Respondents understood this question to mean that they wanted additional income
above and beyond what they require to meet their basic financial obligations. They saw this
part of the question as being distinct from part (g).



Part (o): Respondents thought this question referred to an individual who left teaching to
pursue another career, who now decided to return to teaching to pursue it as a career.



Part (r): Respondents had no difficulty with this item. When asked what they thought were the
aspects of the school schedule/calendar being referenced, they mentioned the time off during
school breaks and the early end to the school day.



Part (s): Respondents had no difficulty with this item. When asked what characteristics they
thought of when they saw the term “desirable characteristics,” they mentioned the location,
type of students, and characteristics of their administration, colleagues, and students’ parents.



Part (t): Respondents had different ideas of what “seniority” meant. About half of all
respondents stated that seniority was related to pay scale, while the other half associated
seniority with privileges given to teachers who have been at a school for a long time (e.g.,
selecting their own class).

ICF Macro, an ICF International Company

Page 9



It was not clear that participants distinguished between parts (d) and (o) when answering the
question. In 9 of 16 interviews, respondents gave the same ratings to parts (d) and (o). The
situations of these 9 respondents are as follows:
Had always planned to return
o This respondent returned to teaching after taking sick leave. She rated both (d) and (o)
as 5. Her most important reason for returning was listed as (i).
o This respondent always planned to return to teaching when your youngest child started
kindergarten. She rated both (d) and (o) as 5. Her most important reason for returning
was listed as (b).
o This respondent planned to return to teaching after completing her Master’s. She rated
both (d) and (o) as 5. She rated her most important reason for returning as (o).
o This respondent returned to teaching as had she finished her graduate work and wanted
to get additional teaching experience so she could move into administration. She rated
both (d) and (o) as 4. Her most important reason for returning was listed as (u) (in
which she explained exactly why she wanted teaching experience).
Changed mind about leaving teaching
o This respondent left teaching to go back to her work as a magazine editor. While there
she realized she missed teaching. She rated both (d) and (o) as 5. Her most important
reason for returning was listed as (d).
o This respondent left teaching to be in politics, but returned to teaching because she felt
“dead-ended.” She rated both (d) and (o) as 4. Her most important reason for returning
was listed as (d).
o This respondent returned to teaching after trying to pursue another career. He rated both
(d) and (o) as 4. He rated his most important reason for returning as (r). When probed,
he said he would change his answer to (d).
Other reasons
o This respondent moved to be closer to family and teaching was the first job she was
able to get. She rated both (d) and (o) as 2. Her most important reason for returning was
listed as (g).
o This respondent returned when she was able to get the position she wanted. She rated
both (d) and (o) as 5. She rated her most important reason for returning as (o).



One respondent had left teaching to become an actuary, but then decided to go back to
teaching. He rated (d) as 5, (o) as 4, and listed his most important reason for returning as (d).

ICF Macro, an ICF International Company

Page 10

Recommendations:


Modify part (k) so that participants understand this question applies only if they are currently
receiving teacher retirement benefits.



Consider adding a “Not Applicable” column to reduce confusion among teachers if items do
not apply to them.



In Macro’s briefing on this research, NCES indicated that the part (d) was intended to refer to
teachers who had retired and were coming back, while part (o) was intended to refer to teachers
who were returning to teaching from another career. This distinction was clearly not evident to
the respondents, many of whom answered (d) even though they had not retired. Therefore,
parts (d) and (o) should be clarified so that their meanings are distinct and clear to respondents.

ICF Macro, an ICF International Company

Page 11

ITEM 6
From the items above, which do you consider the one most important reason in your decision
to return to the position of a pre-K-12 teacher?
*Enter the letter from Item 5 above.
|___| Most important

Findings:


In general, respondents did not have any difficulty responding to this question.



Prior to beginning the interview, each respondent was asked to describe their primary reason
for returning to teaching. The purpose of this question was to determine if respondents’
answers to Item 6 would be consistent with the answer given at the start of the interview. In all
but two interviews, respondents either readily selected reasons for returning to teaching from
the list that closely matched their true reason, or entered their own reason for returning using
the “other” option.



One of the two respondents whose responses appeared inconsistent said at the beginning of the
interview her primary reason for returning was because her health had improved (which would
seem to most closely match option (c)). For Item 6, however, she selected option (i): “Because
I needed the health benefits.” When asked about this inconsistency, she indicated that to her
both items meant the same thing.



The other respondent stated at the beginning of the interview that she returned to teaching
because she decided that she missed it. When answering Item 6, however, she initially chose
option (g): “Because I needed the income to meet my financial obligations (e.g., rent, loans,
credit card payments).” She commented that this was the primary purpose of any job, since if
she had no need for money she would not work. After thinking about it, however, she decided
that this was not in the spirit of what Item 6 was asking, and she changed her answer to option
(d): “Because I wasn’t ready to stop teaching,” which was more consistent with her explanation
at the beginning of the interview.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 12

ITEM 7
Since December 31, 2008, have you completed a teacher preparation program leading to your
initial teaching certificate?
__ Yes  GO TO Item 8.
__ No  GO TO Item 9 below.

Findings


Most respondents found this question to be clear. There was no confusion about the definition
of a “teacher preparation program” or an “initial teaching certificate.”



Three respondents were initially confused as to whether the word “since” was to be interpreted
as prior to December 31, 2008 or after December 31, 2008. After some thought these three
respondents correctly concluded that the questions should be interpreted as after December 31,
2008.



One respondent who completed her teacher preparation program since December 31, 2008 but
did not yet have an initial teaching certificate answered “Yes” to this question.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 13

ITEM 8
Was the teacher preparation program a part of an alternative certification program?
(An alternative certification program is a program that was designed to expedite the transition of
non-teachers to a teaching career, for example, a state, district, or university alternative
certification program, Teach For America, or National Teacher Project).
__ Yes
__ No

Findings:


Almost all respondents found this question clear. Two respondents obtained their certification
through alternative means (Teach for America and Maryland’s Resident Teacher Certificate
program) and found the definition to accurately reflect their own understanding.



After reading the definition, only one respondent remained unclear as to what an alternative
certification program entailed. This respondent thought an alternative certification program was
one that prepared you for teaching through online courses.



One respondent who possessed a Masters of Arts in Teaching (MAT) commented that in some
states the MAT is considered an alternative program, while in other states it is considered a
traditional program. She classified her program as traditional.

Recommendations:


If appropriate, consider adding language to this question that clarifies how MAT programs
should be classified.

ICF Macro, an ICF International Company

Page 14

ITEM 9
Since December 31, 2008 have you renewed your teaching certificate?
__ Yes
__ No

Findings:


Most respondents were able to answer whether they had renewed their teaching certificate since
December 31, 2008. Two respondents again struggled to determine whether “since” meant
prior to or after December 31, 2008. Again, these respondents eventually correctly concluded
that the question should be interpreted as after December 31, 2008.



Three respondents that were fairly new to teaching were not sure what it meant to “renew their
certificate.” Since these respondents were not yet required to renew their certificates, this did
not affect the accuracy of their responses.



One respondent had moved to a new state, and therefore had to obtain a new teaching
certificate for that state. She answered “no” to this question, because she did not consider this
to be “renewing” her certificate.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 15

ITEM 10
Since December 31, 2008 have you received an endorsement to your certificate?
__ Yes
__ No

Findings:


Four respondents did not understand the meaning of the word “endorsement.” Of these four:
o One left the question blank because he did not know what it meant.
o One responded “No,” which turned out to be the correct answer.
o One responded “Yes,” but after the definition of “endorsement” was explained to her it
became clear that she actually did not have one. She stated that she had initially answered
“Yes” because she had completed her Masters and had moved up on the pay scale.
o The final respondent responded “No.” She was initially certified to teach multi-categorical
special education grades 6-8. Due to changes in her state’s criteria for “highly qualified”
teachers, she was required to re-certify to teach grades K-12. She completed the
requirements to become highly qualified after December 31, 2008.



One respondent commented that she had lost some of her endorsements. This teacher was
initially certified to teach in grades K-6. When she changed states and needed to become recertified, she was only able to obtain certification for grades K-3.

Recommendations:


Provide a definition for “endorsement” as part of this question.

ICF Macro, an ICF International Company

Page 16

ITEM 11
Since December 31, 2008 have you completed a master’s degree, educational specialist or
professional diploma, doctoral degree (Ph.D., Ed.D), or professional degree (M.D., J.D.,
D.D.S.)?
__ Yes  GO TO Item 12.
__ No  GO TO Item 13.

Findings:


All respondents thought this question was clear.

Recommendations:


For consistency consider inserting the word “degree” after the word “educational specialist.”
Educational Specialist (Ed.S.) programs are generally considered to be degree programs.

ICF Macro, an ICF International Company

Page 17

ITEM 12
Please specify the field of the master’s degree, educational specialist or professional diploma,
doctoral degree (Ph.D., Ed.D), or professional degree (M.D., J.D., D.D.S.) that you have
completed.
*Record one of the assignment field codes and names listed in Table 1 on page 7.
|___|
Note: The list of field codes and names that was given to participants is provided on the following
page.
Findings:


Several respondents with Master’s degrees experienced difficulty identifying the field of their
degree from the table. One respondent commented that the list more closely resembled a
Bachelor’s degree list than a Master’s degree list, and that the majority of educational fields
were not reflected.



Several respondents found themselves trying to combine degrees, such as Administration (131)
and Policy Studies (134).



One respondent had multiple graduate degrees, and was unsure which to use in his response to
Item 12.

Recommendations:


Replace the current degree list with a Master’s degree list that includes programs with
concentrations in areas such as Administration and Supervision, and Curriculum and
Instruction. This list should also include Masters of Arts in Teaching (MAT) and Masters in
Business Administration (MBA) degrees.



For consistency consider inserting the word “degree” after the word “educational specialist.”
Educational Specialist (Ed.S.) programs are generally considered degree programs.

ICF Macro, an ICF International Company

Page 18

Table 1. Major Fields of Study Codes
Elementary Education
101
102

Early childhood or Pre-K, general
Elementary grades, general

Secondary Education
103
104

Middle grades, general
Secondary grades, general

Arts and Music
141
142
143
144
145

Art or arts or crafts
Art history
Dance
Drama or theater
Music

English and Language Arts
151
152
153
154
155
156
157
158
159

Communications
Composition
English
Journalism
Language arts
Linguistics
Literature or literary criticism
Reading
Speech

English as a Second Language
160
161
162

ESL or bilingual education: general
ESL or bilingual education: Spanish
ESL or bilingual education: other languages

Foreign Languages
171
172
173
174
175

French
German
Latin
Spanish
Other foreign language

General Education
Special Education
110

Other Education
131
132
133
134
135
136

Health education
Physical education

Mathematics and Computer Science
190
197

Mathematics
Computer science

Natural Sciences
211
212
213
214
217
218

Biology or life sciences
Chemistry
Earth sciences
Engineering
Physics
Other natural sciences

Administration
Counseling and guidance
Educational psychology
Policy studies
School psychology
Other non-subject matter specific education

Subject Matter Specific
Social Sciences
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235

Anthropology
Area or ethnic studies (excluding Native American Studies)
Criminal justice
Cultural studies
Economics
Geography
Government or civics
History
International studies
Law
Native American studies
Political science
Psychology
Sociology
Other social sciences

Vocational, Career, or Technical Education
241
242
243
244
245
246
247
249
250
253

Health Education
181
182

Special education, any

254
255
256

Agriculture and natural resources
Business management
Business support
Marketing and distribution
Health occupations
Construction trades, engineering, or science technologies
(including CADD and drafting)
Mechanics and repair
Manufacturing or precision production (electronics,
metalwork, textiles, etc.)
Communications and related technologies (including design,
graphics or printing; not including computer science)
Personal and public services (including
culinary arts, cosmetology, child care, social work,
protective services, custodial services, and
interior design)
Family and consumer sciences education
Industrial arts or technology education
Other vocational, career, or technical education

Miscellaneous
261
263
264
265
266
267

Architecture
Humanities or liberal studies
Library or information science
Military science or ROTC
Philosophy
Religious studies, theology or divinity

Other
268

Other

ITEM 13
This school year (2008-09) have you been assigned as a mentor to other teachers in your
school or district?
__ Yes  GO TO Item 14.
__ No  GO TO Item 17.

Findings:


Of all the respondents who answered this question, one was currently a mentor and two were
former mentors.



Almost all respondents assumed that the question referred to being formally assigned as a
mentor. Only one thought the question could also refer to being an unofficial mentor. As this
respondent did not operate in either capacity, this incorrect assumption did not affect the
accuracy of his response.



Two respondents who worked with student teachers wondered whether the question included
this type of mentoring. They eventually decided it did not, and responded “No.”

Recommendations:


Consider modifying the question to read “…formally assigned as a mentor…” if this is the
intent of the question.

ICF Macro, an ICF International Company

Page 20

ITEM 14
How many teachers are you mentoring in your school or district this year?
|__|

Findings:


Three respondents provided answers to this question—one who was currently a mentor and two
who were former mentors. None had any difficulty answering the question.



One respondent, in addition to mentoring one teacher in her school, was a part of a district-wide
mentoring program through which she works with 20-25 teachers. She did not consider her
district-wide role when answering this question, and responded “1.”



Respondents who were not mentors were also asked to review this question. One respondent
was not sure if the question referred to the current calendar year, or the current school year.
Two respondents who had never been mentors thought that mentors might have difficulty
reporting the number of mentees with whom they worked—although as noted above, this did
not appear to be the case.

Recommendations:


Consider modifying the question to read “…this school year.”

ICF Macro, an ICF International Company

Page 21

ITEM 15
Were you given any training related to mentoring by your school or district prior to
becoming a mentor?
__ Yes
__ No

Findings:


The respondent who currently serves as a mentor responded “Yes” to this question. However,
she commented that she would have wanted to be able to provide more detail about her
experience, rather than just answering “Yes” or “No.” She recommended the following
response options: “formal mentor training program,” “informal mentor training,” and “no
training.”



One of the former mentors who answered this question indicated that she received training
while she served as a mentor, but not prior to becoming a mentor. Therefore, she would have
answered “No” to Item 15.

Recommendations:


Consider modifying the question so that it also captures information about training that is
provided while teachers are serving as mentors, not only before.

ICF Macro, an ICF International Company

Page 22

ITEM 16
To what extent do you feel prepared to be a mentor?
__ Not at all prepared
__ Somewhat prepared
__ Well prepared
__ Very well prepared

Findings:


The three respondents who were current or former mentors had no difficulty rating their level
of preparedness on this scale. The current mentor felt “somewhat prepared”, while the former
mentors felt “well prepared” and “very well prepared.”



One respondent commented that there were two distinct aspects to mentoring—content
(mentoring on how to teach a specific subject) and process (mentoring on how to handle other
teacher issues, such as classroom management and time management). This respondent
indicated that it would be possible to have different levels of preparedness in these two aspects,
and therefore felt that perhaps the question should ask about both separately.

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 23

ITEM 17
For your main teaching assignment have you:
*Mark (X) one box on each line.
(Please answer yes or no to the following):
Y
a.
b.
c.
d.
e.
f.

N

Received an in-state teaching certificate in this field
Received an in-state teaching certificate for the grades you teach
Received a National Board Certification in this field
Earned a bachelor’s or higher degree in this field
Taken and passed a state exam showing subject competency in this field
Taken and passed the HOUSSE (High Objective Uniform State Standard of
Evaluation) in this field

Findings:


Three respondents initially thought that the question was asking about a specific time frame
(either “this school year” or “since December 31, 2008”). This was likely because previous
questions were limited to specific time frames. All three participants eventually understood
that this question did not include a time frame, so this misconception did not impact their
responses to the questions.



Part (a): Three participants were unsure what to consider their “field” for the purposes of this
question. One, who possessed a certificate to teach in elementary grades but specifically taught
math, was not sure whether her “field” was elementary education or mathematics (she did not
possess any certification in mathematics). She eventually decided to respond to the questions
using elementary education as her field. Another reported that she had certification in both
social studies and in elementary education but was unsure which of the two was her “field” (she
eventually answered the question based on her certification in social studies). A third teacher
was also initially unsure of her “field,” but eventually concluded that she was certified to teach
kindergarten.



Part (b): Respondents had no difficulties with this question.



Part (c): National Board Certification was familiar to all respondents.



Part (d): Respondents had no difficulties with this question.



Part (e): Two respondents were unsure of whether they had taken a state exam showing subject
competency in their field. These respondents had taken the PRAXIS content exam, but they
were not sure if this was also considered the state test. Both answered “No” to this question.
One respondent who had taken the PRAXIS test did respond “Yes” to this question, because he
knew his state had adopted the PRAXIS as the state exam.



Part (f): Only one of 16 respondents had ever heard of HOUSSE. This participant had recently
taken and passed the HOUSSE in her state.

ICF Macro, an ICF International Company

Page 24

Recommendations:


While two interview participants had difficulty answering part (a) because they were unsure
what to consider their “field,” this may have been in part because no context was provided for
this item. If it is preceded by other items that ask respondents to specify their field, this
confusion should be reduced. This potential confusion may be more frequent among
elementary school teachers.



Consider clarifying whether PRAXIS II content exams should be considered “state exams.”

ICF Macro, an ICF International Company

Page 25

Second Phase of Testing
(June/July 2009)

ICF Macro, an ICF International Company

Page 26

ITEM 1
What was the primary reason for this reduction in force?
 Budget cuts or budget shortfalls
 Reduced pupil enrollment
 School and/or district merger
 Reason unknown
 Other reason – please specify ________________________

Findings:


Of the twelve teachers interviewed, three had failed to have their contract renewed due to a
reduction in force. However, all remaining teachers had taught in schools affected by a
reduction in force.



Seven teachers gave the first response to this question (“budget cuts or budget shortfalls”),
while five gave the second response (“reduced pupil enrollment”).



Although all teachers were knowledgeable about the reasons for the reduction in force at their
school, several said they could not answered confidently as to which was the primary reason.
In fact, about half reported that they would like to be able to select more than one option from
the list as their reduction in force was due to a combination of several factors. One specifically
noted that since budget cuts are often due to reduced pupil enrollment, the first two response
options will often overlap.



Teachers were not able to identify any additional reasons for reductions in force that should be
included in the list.

Recommendations:


As teachers may not know the primary reason for the reduction in force, consider allowing
respondents to check all responses that apply to their situation.

ICF Macro, an ICF International Company

Page 27

ITEM 2
To what extent do you agree or disagree with each of the following statements?

a) My principal supports
me in classroom
management issues.
b) My principal supports
me in my interactions
with parents.
c) My principal is readily
available to me.
d) My principal listens to
my concerns.
e) My principal takes
time to visit my
classroom.
f) My principal supports
my professional
development.
g) My principal provides
me with useful
feedback about my
teaching.
h) My principal has
respect for teachers.
i) My principal has
respect for students.
j) My principal treats
teachers fairly.
k) My principal
encourages
collaboration among
teachers.
l) My principal supports
student teachers in my
school.
m) My principal is readily
available to student
teachers in my school.

 There are no
student teachers in
my school
 There are no
student teachers in
my school

ICF Macro, an ICF International Company

Strongly
Agree

Somewhat
Agree

Somewhat
Disagree

Strongly
Disagree









































































































Page 28

Findings:
Specific Items:


Part (a): Two respondents had difficulty answering this part of the question because although
their principal is generally helpful, they do not need support in classroom management issues.
One commented that disagreeing with this statement seemed to be a negative statement about
the principal, when in fact it could mean that no support is necessary.



Part (c): Two teachers had difficulty answering this question. One teacher was unsure how to
interpret the question as her principal is readily available by email, but face-to-face contact is
minimal. The other teacher thought the question was ambiguous as her principal is frequently
required to be out of the building and this teacher wasn’t clear if the question seemed to suggest
that the principal’s lack of availability was due to him neglecting his responsibilities.



Part (e): Teachers had differing interpretations of what “takes time to visit my classroom
meant.” Some felt that a brief visit lasting a few minutes would qualify, while others only
counted incidents when the principal observed an entire class.



Part (e): Two principals commented that at their school it is not the principal’s job to visit
classrooms; they both selected “somewhat agree.”



Part (f): One teacher was unsure of how to answer this question as her principal supports
teachers’ required professional development, but not necessarily their preferred professional
development (e.g. enrolling in a graduate program). She indicated that her principal “checked
the box” with respect to professional development, but was not particularly helpful. This
teacher eventually selected “somewhat agree.”



Part (g): Two teachers stated that as it was not the principal’s job to provide feedback (it is her
supervisor’s responsibility). One said that she would have preferred a “not applicable” option,
because disagreeing with this statement seemed to reflect badly on her principal. Both
respondents eventually selected “somewhat disagree.”



Part (h): Three teachers thought the term “respect” was too broad. Two thought a distinction
should be made between personal and professional respect for teachers, while one commented
that there were many different levels of respect, such as listening to teachers’ concerns vs.
seeking teacher input in decision making.



Parts (l) and (m): Teachers were split almost evenly on whether they thought respondents to
this survey would know the extent to which their principal supported or was available to
student teachers.

Overall:


One respondent questioned the efficacy of having “somewhat agree” and “somewhat disagree”
options instead of “agree” and “disagree.” She would have preferred the latter response
choices.



When asked what additional aspects of a principal’s job could be added to the list, teachers
suggested:
o Principal’s support for new teachers.
o Principal’s support or enforcement of school-wide discipline.

ICF Macro, an ICF International Company

Page 29

o Principal’s interaction with the community.
o Principal’s effective use of teacher time (especially with regard to scheduling and
planning meetings).
Recommendations:


Reword parts (a) and (b) as “My principal is able to provide me with support in classroom
management issues/in my interactions with parents.”



It is likely that in larger schools principals delegate to other administrators the job of visiting
classrooms and providing feedback. Consider including a “not applicable” column for parts (e)
and (g).



Consider including a “don’t know” column in parts (l) and (m).

ICF Macro, an ICF International Company

Page 30

ITEM 3
How satisfied are you with the principal at your current school?
 Very satisfied
 Somewhat satisfied
 Somewhat dissatisfied
 Very dissatisfied

Findings:


Most teachers (9 of 12) had no difficulty responding to this question.



Two teachers felt the question was ambiguous, because there were some things with which she
was satisfied and some with which she was not. They indicated that they had to “average out”
their opinions to select their responses (one ended up selecting “somewhat satisfied,” while the
other selected “somewhat dissatisfied”).



Another teacher reported that she always has a hard time answering these types of questions as
she hates to give anyone a “bad review.” She also expressed concern that a teacher might rate a
principal poorly on the basis of one or two experiences.



When asked what factors they considered in determining their level of satisfaction, teachers
stated:
o Respect for teachers (6 of 12)
o Principal availability (5 of 12)
o Support (5 of 12)
o Leadership skills (2 of 12)
o Student discipline (2 of 12)

Recommendations:


No changes are necessary for this item.

ICF Macro, an ICF International Company

Page 31

ITEM 4
Did you have input in choosing your mentee(s)?
 Yes
 No

Findings:


All teachers were either current mentors or had previously been mentors.



Only one teacher reporting having had input in choosing their mentee(s). However, this may
have been because several respondents were actually “mentors” for student teachers, not other
faculty members at their school.



Teachers were asked if in their general experience, mentor teachers had input in selecting their
mentees. Almost all said that in their experience mentor teachers did not have input, although
again several may have been thinking of how their school treats student teachers, not new
teachers.



When asked what type of “input” they thought the question referenced, teachers’ responses
generally fell into two categories:
o Having the ability to select a mentee from a group of mentees; or
o Having their opinions considered in the assignment of a mentee (i.e. being able to accept or
reject the mentee they were assigned).

Recommendations:


If the intent of the question matches respondents’ interpretations of “input” as described above,
no changes are necessary.

ICF Macro, an ICF International Company

Page 32

ITEM 5
Did your principal work with you to choose your mentee(s)?
 Yes
 No

Findings:


All but one respondent answered “No” to this question. The one who responded “Yes” later
explained that despite her response, it was actually her Assistant Principal that worked with her,
not the principal.



Two teachers reported that usually there is some dialogue between principal and mentor teacher
regarding mentees. The other ten participants indicated that in their experience, principals do
not usually work with mentor teachers to choose their mentees.



About half of the respondents said that it was someone else’s job to select mentees, not their
principal’s. For example, the one respondent who indicated in Item 4 that they had input in
choosing their mentee indicated that her principal was not involved at all in the process; it was
handled by another member of her school’s leadership team.

Recommendations:


In a significant number of schools, the selection of mentees seems to be handled by someone
other than the principal. Therefore, consider revisiting the question to determine whether the
intent is to measure specifically whether respondents worked with the principal to choose their
mentees, or whether they worked with anyone. If the intent is the latter, also consider whether
Item 5 is redundant with Item 4.

ICF Macro, an ICF International Company

Page 33

ITEM 6
Has your principal discussed your mentoring relationship with you this school year?
 Yes  GO TO Item 7.
 No  GO TO Item 8.

Findings:


Only two of twelve teachers responded affirmatively to this item. One of these teachers stated
that her district requires principals to meet with mentors once a month.



When asked to explain the phrase “discuss your mentoring relationship with you,” teachers’
responses fell into two categories:
o Discussing how mentees were progressing; and
o Discussing how the mentors themselves were managing their mentor responsibilities.
Some teachers referenced both categories in their response while other teachers only referenced
one of these categories.



As with Item 5, several participants indicated that while they answered “No” to Item 6, they
actually meet with some other staff member other than the principal.

Recommendations:


Consider clarifying the item as to which of the two types of discussions described above should
be considered. Discussions about the progress of mentees and discussions about the progress
of the mentor relationship itself are very different, and perhaps their frequency should not be
measured through the same question.



As with Item 5, consider revisiting the question to determine whether the intent is to measure
specifically whether respondents had discussions with the principal specifically, or whether
they had these discussions with anyone.

ICF Macro, an ICF International Company

Page 34

ITEM 7
How often does your principal discuss your mentoring relationship with you?
 At least once a week
 Once or twice a month
 A few times a year

Findings:


All teachers indicated that this question was clear.



Of the two teachers who answered this question, one responded “once or twice a month”, and
the other responded “a few times a year”. As noted above, one teacher stated that monthly
meetings between principals and mentors are required by her district.

Recommendations:


No changes are required for this item—although any changes to Item 6 (such as changing the
item to address all administrators rather than the principal specifically) should obviously be
carried through to Item 7 as well.

ICF Macro, an ICF International Company

Page 35

ITEM 8
To what extent do you agree with the following statement?
Outside my school, I have a strong support system (e.g. family, close friends, neighbors).





Strongly agree
Somewhat agree
Somewhat disagree
Strongly disagree

Findings:


Eleven teachers strongly agreed with the statement, while one teacher somewhat agreed. The
teacher that somewhat agreed stated that she would have preferred to have the option to
“agree.”



Three teachers commented that the term “support” was too broad and could refer to personal or
professional support. Two of the three teachers eventually decided the question referenced
professional support, while the other teacher decided the question referenced personal support.



When asked what was meant by the phrase “a strong support system,” most teachers’ responses
generally included statements referring to having close family and friends they can rely on.
However, three actually referenced sources of support within their schools; two teachers
considered communication and cooperation with their colleagues, and another thought of
support she had received from her school system in her role as a mentor.

Recommendations:


If appropriate, clarify the question to emphasize that respondents should only reference nonschool-based support systems.



Revisit the intent of the question, and consider whether there is a more direct or appropriate
way of phrasing the question. For example, if the intent is to measure teachers’ ability to deal
with job-related stress, sources of support within their school may well be important to
consider, and should not be excluded.

ICF Macro, an ICF International Company

Page 36

ITEM 9
During this school year, did you or do you …
 a) Coach a sport?
 b) Sponsor any student groups, clubs, or organizations?
 c) Serve as a department lead or chair?
 d) Serve as a curriculum specialist?
 e) Serve on a school-wide or district-wide committee or task force?
 f) Serve as a consultant or coach in a particular subject?
 g) Serve as a mentor coordinator?
 h) Have a teacher union leadership position?
 i) Present at a conference?
 j) Participate in any other leadership activity? Specify: _________________________

Findings:
Specific Items


Part (d): Only one of the 12 respondents reported that the title of curriculum specialist was
used at their school. One teacher stated, however, that this term was used at the district level
and it was given to the Associate Superintendent. In general teachers had two concepts of the
role of a “curriculum specialist”:
o A curriculum writer or developer; or
o A curriculum content expert or resource person who provides teacher support.
One teacher indicated that all teachers in her school conduct curriculum mapping, which
essentially makes them an expert. She therefore considered herself a curriculum specialist and
checked this item in the list. Another teacher who checked this item stated that she served on a
curriculum committee for one subject.
Titles that teachers thought were similar were:
o Curriculum Committee Member. In one district this is a paid summer position and in
another district this is a rotating responsibility.
o Curriculum and Administration Dean. This is a district level position.
o Curriculum Facilitator. This is a school level position where the Facilitator visits
classrooms and models lessons (one per school).
o Academic Coach
o Associate Superintendent



Part (f): The title of consultant is not used in any school system represented by the teachers
interviewed. One respondent indicated that she serves as a consultant in her school, because
she often provides informal feedback to younger teachers.

ICF Macro, an ICF International Company

Page 37



Part (f): The title of coach is more frequently used. One teacher stated that her school has a
literacy coach, and one teacher stated her school has an academic coach. For all other teachers
the concept of a coach varied. Teachers’ responses are summarized below:
o The department head or chair is the coach as he/she serves as the content area expert (no
one is just a coach) (2 teachers).
o A coach is someone knowledgeable about content, but this position doesn’t formally
exist.
o A tutor--someone who stays after school to work with students.
o We only have a graduation coach to make sure students don’t drop out.
o We have athletic coaches and coaches for after school activities such as a spelling bee
coach.
o Mentors for new teachers can be sometimes thought of as coaches.



Part (g): Four teachers stated that the term mentor coordinator was used in their school and/or
district. Other teachers stated that although someone acted in the role of a mentor coordinator,
they did not have a title; this was just considered an additional responsibility. Whether or not
they used this particular title in their school, all participants seemed to understand what was
meant by this phrase.



Part (h): One teacher whose state did not allow teachers to unionize stated she felt this question
should have a “not applicable” option.



Part (i): A few teachers wondered whether presenting at large faculty meetings should also be
included under this item. They all eventually decided it should not, but one stated that if she
were completing the paper survey she would have probably checked this box as it would make
her “look good.” Two teachers were unsure whether this item included parent-teacher
conferences. One of the two decided it did, and checked the box.



Part (i): At least one participant misread this part of the question to mean that she was present
at a conference (i.e., that she attended a conference).



Part (j): Some participants checked part (j) and specified other leadership activities they had
participated in. In some cases, however, it was unclear that these activities represented the type
of leadership intended by the question; for example, one wrote that she was a member of a
professional organization, and another said he was a tutor in his school.

Overall


In general, respondents seemed to want to check as many boxes as possible, and in some cases
tried to rationalize checking a box (such as the teacher who considered herself a “consultant” to
other teachers.



Some teachers were unsure of whether this question was restricted to school-level leadership
positions or if district and state-wide leadership positions were to be included. Most teachers
decided to include all levels of activity.



When asked what additional activities they felt should be included in the list, teachers
mentioned leadership teams and steering committees (such as a prom planning committee),
although it appears these activities could be included under parts (b) or (e).



One teacher did not understand that she should check all activities in the list that applied to her.

ICF Macro, an ICF International Company

Page 38

Recommendations:


One general problem with this question is that some of the items relate to titles (such as
“curriculum specialist” or “department chair” while other items related to roles (such as a
presenter at a conference, or a sponsor of school groups). The intent could perhaps be clarified
for respondents by separating these two types of questions into two different items.



Modify the stem of the item so that teachers are aware they are to “check all that apply.”



If appropriate, consider modifying the stem of the item to reflect that teachers are to consider
all school, district, or state-wide positions.



Provide a description of the role of a curriculum specialist (part (d)) and consultant/coach (part
(f)) to ensure that teachers interpret this item consistently.



Reword part (i) as “Give a presentation at a professional conference”.

ICF Macro, an ICF International Company

Page 39

Appendix A: Interview Protocol for
First Phase of Interviews

NCES Beginning Teacher Longitudinal Survey
Interview Protocol
I. Introduction
“Thank you for agreeing to be interviewed for this project. My name is ____________, and I work for
ICF Macro, an independent company that has been hired by the U.S. Department of Education to
conduct this study. We will be asking you to help revise and improve a questionnaire called the
Beginning Teacher Longitudinal Survey.
“During this interview I am going to ask you to read and answer a number of questionnaire items, one
at a time. As you go through the questions, I would like you to explain what you are thinking out loud,
so I can get a sense of your thought process as you answer each item. For example, if you are trying
to decide what your answer is, please explain why you are unsure. If you have trouble understanding
a question, or are confused by it, please be sure to explain that to us as well.
“The feedback you provide in this interview will be completely anonymous. In our report to the
Department of Education, we will not connect anyone’s comments with their name. The feedback that
we collect will have a direct impact on the design of next year’s survey, so please open and honest in
your comments.
“Do you have any questions before we begin?”
Before you begin, ask them the screening question for their group just to confirm that they are
qualified for the interview:
All
1.

Are you currently a K-12 classroom teacher?

2.

Have you ever temporarily left the teaching profession?

3.

What is the name of the school you were teaching in before you temporarily left the profession?

4.

For how long were you away from the teaching profession?

5.

What was your last year as a teacher before you temporarily left the profession?

6.

What were you doing during your break from teaching?

7.

In what year did you return to the profession?

8.

Why did you decide to return to teaching?

II. Questionnaire Items
Go through the appropriate questionnaire items with the participant. As the participant answers each
item, record their answer on the sheet. If they hesitate while answering or reading a question at any
point, ask them to explain why.
In each case, allow them to complete 1 series (there are 5 series) of questions before asking any
follow-up or probe questions, or providing any clues as to the purpose or meaning of the question. The
series are as follows:
-

Series 1: Items 1 through 4
Series 2: Items 5 and 6
Series 3: Items 7 through 12
Series 4: Items 13 through 16
Series 5: Item 17

ITEM 1

You were teaching in [school] during the 2007-08 school year. Did you return to that school?
__ Yes  GO TO Item 2.
__ No  GO TO Item 3 below.

Interviewer Notes

Probes
Was there anything about this question that was unclear?

ITEM 2
Have there been any changes in your current school, such as a change in name, location, or
grades offered?
__ Yes  GO TO Item 5 below.
__ No  GO TO Item 5 below.

Interviewer Notes

Probes
Do you think teachers who have returned to their old school could be expected to know if these
changes occurred?

ITEM 3
Are you currently teaching in the SAME STATE as you were during the 2007-08 school year?
__ Yes  GO TO Item 4.
__ No  GO TO Item 4.

Interviewer Notes

Probes
Is there anything about this question that you think could be confusing or misinterpreted?

ITEM 4
Which of the following describes the change from your 2007-08 school to your current
school? (For this question, all charter and Bureau of Indian Education [BIE]-funded schools
are considered public schools.)
*Mark (X) only one box.
__ Moved from a PUBLIC school to another PUBLIC school in the SAME SCHOOL
DISTRICT.
__ Moved from a PUBLIC school to another PUBLIC school in a DIFFERENT SCHOOL
DISTRICT.
__ Moved from a PUBLIC school to a PRIVATE school.

Interviewer Notes

Probes
Is there anything about this question that you think could be confusing or misinterpreted?

ITEM 5
Indicate the level of importance EACH of the following played in your decision to return to
the position of a pre-K-12 teacher.
*Mark (X) one box on each line.
The following scale is used for the items below: 1) Not at all important, 2) Slightly important, 3)
Somewhat important, 4) Very important, 5) Extremely important
If any of the reasons for returning to teaching do not apply to you mark “1” as not at all
important.
I returned to the position of a pre-K-12 teacher:
1
a. Because I had a change in residence or wanted to take a job more
convenient to my home.
b. Because my maternity/paternity leave ended or I no longer needed to
stay at home with my children.
c. Because my health or the health of a loved one no longer required
me to be out of teaching.
d. Because I decided I was not ready to stop teaching.
e. Because I passed the required test(s).
f. Because I was offered the grade level or subject area that I wished to
teach.
g. Because I needed the income to meet my financial obligations (e.g.,
rent, loans, credit card payments).
h. Because my current school or district offered at least partial
forgiveness of my student loans.
i. Because I needed the health benefits.
j. Because I wanted the retirement package.
k. Because I could continue receiving my teacher retirement benefits.
l. Because I was given a housing incentive by my current school.
m. Because I wanted a higher standard of living than my household
income provided.
n. Because I wanted job security.
o. Because I decided to pursue a pre-K-12 teaching position as a career.
p. Because I completed the coursework I was pursuing.
q. Because a part-time teaching assignment became available.
r. Because I liked the school schedule/calendar.
s. Because I obtained a position in a school with desirable
characteristics.
t. Because I was able to maintain my seniority.
Other factors
u. Because of other factors not included in previous items a – t
 please specify:

2

3

4

5

Interviewer Notes

Probes
Do you think there are any important reasons for returning to teaching that are not included in this list?

Question d, o: What do you see as the difference between what items d and o are asking?

Question i: Do you think there are other types of benefits that would be important enough to a teacher
to make him/her return to teaching?

Question j, k: What do you see as the difference between what items j and k are asking?

Question l: Is it clear what is meant by the term “housing incentive?” Please give me a few examples
of what you think a "housing incentive" is?

Question m: What do you understand this statement to be asking?

Question r: What do you understand this statement to be asking? What aspects of the school
schedule/calendar do you think are being referenced? Is the wording awkward or confusing?

Question s: Is the term “desirable characteristics” clear? What characteristics do you think of when you
see this term?

Question t: Explain what you think the term “seniority” means?

ITEM 6
From the items above, which do you consider the one most important reason in your decision
to return to the position of a pre-K-12 teacher?
*Enter the letter from Item 5 above.
|___| Most important

Interview Notes

Probes
Did you think your most important reason for returning to teaching was adequately reflected in the table?

Interviewer – please check if the reason given here is consistent with the answer they gave immediately
before they began this section. If not, please ask why.

ITEM 7
Since December 31, 2008, have you completed a teacher preparation program leading to your
initial teaching certificate?
__ Yes  GO TO Item 8.
__ No  GO TO Item 9 below.

Interview Notes

Probes
What do you understand by the term “teacher preparation program?”

What do you understand the term “initial teaching certificate” to mean? [wait for response]
The intent of this question is to determine if teachers obtained their first teaching certificate since December 31,
2008. Do you think the term “initial teaching certificate” is clear? Is there another term that would be more
appropriate?

ITEM 8
Was the teacher preparation program a part of an alternative certification program?
(An alternative certification program is a program that was designed to expedite the transition of nonteachers to a teaching career, for example, a state, district, or university alternative certification program,
Teach For America, or National Teacher Project.)
__ Yes
__ No

Interviewer Notes

Probes
Did you have any difficulty determining whether your teacher preparation program was an alternative
certification program?

Is the definition of “alternative certification program” sufficiently clear?

ITEM 9
Since December 31, 2008 have you renewed your teaching certificate?
__ Yes
__ No
Interviewer Notes

Probes

Was there anything about this question that was confusing?

ITEM 10
Since December 31, 2008 have you received an endorsement to your certificate?
__ Yes
__ No

Interviewer Notes

Probes
What do you understand by the term “endorsement”? (It should mean an added field or grade level to an
already existing teaching certification)

ITEM 11
Since December 31, 2008 have you completed a master’s degree, educational specialist or
professional diploma, doctoral degree (Ph.D., Ed.D), or professional degree (M.D., J.D.,
D.D.S.)?
__ Yes  GO TO Item 12.
__ No  GO TO Item 13.
Interviewer Notes

Probes
Is there anything about this question that may seem unclear?

ITEM 12
Please specify the field of the master’s degree, educational specialist or professional diploma,
doctoral degree (Ph.D., Ed.D), or professional degree (M.D., J.D., D.D.S.) that you have
completed.
*Record one of the assignment field codes and names listed in Table 1 on page 7
|___|
Interviewer Notes

Probes

Did you have any difficulty identifying your field from Table 1?

ITEM 13
This school year (2008-09) have you been assigned as a mentor to other teachers in your school or
district?
__ Yes  GO TO Item 14.
__ No GO TO Item 17.

Interviewer Notes

Probes

What do you understand the phrase “assigned as a mentor” to mean?

ITEM 14
How many teachers are you mentoring in your school or district this year?
|___|
Interviewer Notes

Probes
Do you think a mentor would find it difficult to report the exact number of teachers he/she mentors?

ITEM 15
Were you given any training related to mentoring by your school or district prior to becoming a
mentor?
__ Yes
__ No
Interviewer Notes

Probes
What do you understand by the term “training”? Do you think this term could be ambiguous?

ITEM 16
To what extent do you feel prepared to be a mentor?
__ Not at all prepared
__ Somewhat prepared
__ Well prepared
__ Very well prepared

Interviewer Notes

Probes

Did you have any difficulty identifying your level of preparation on the scale?

ITEM 17
For your main teaching assignment have you:
*Mark (X) one box on each line.
(Please answer yes or no to the following):
g. Received an in-state teaching certificate in this field
h. Received an in-state teaching certificate for the grades you teach
i. Received a National Board Certification in this field
j. Earned a bachelor’s or higher degree in this field
k. Taken and passed a state exam showing subject competency in this field
l. Taken and passed the HOUSSE (High Objective Uniform State Standard of Evaluation) in
this field

Interview Notes

Probes
What is your main teaching assignment?

As you answered all the questions in this section did you consistently only reference your main
teaching assignment?

Did you have any difficulty distinguishing between what was being asked in part (a) and part (b)?

Please explain your understanding of National Board Certification? Do you think that most teachers
understand what this is?
Note to interviewer: National Board Certification is advanced professional certification based on the
National Board for Professional Teaching Standards (NBPTS). There are 25 different certificates
available. The assessment for National Board Certification consists of a multimedia teaching portfolio
that is used to evaluate pedagogy and student outcomes, and a three-hour assessment center
examination.

Was there anything confusing about part (e)?

Please explain your understanding of HOUSSE? Is this a term with which you think most teachers are
familiar?
Note to interviewer: HOUSSE allows current teachers to demonstrate subject matter competency and
Highly Qualified Teacher (HQT) requirements through a combination of proven teaching experience,
professional development, and knowledge in the subject acquired over time through working in the
field.

Table 1. Major Fields of Study Codes
Elementary Education
101
102

Early childhood or Pre-K, general
Elementary grades, general

Secondary Education
103
104

Middle grades, general
Secondary grades, general

Arts and Music
141
142
143
144
145

Art or arts or crafts
Art history
Dance
Drama or theater
Music

English and Language Arts
151
152
153
154
155
156
157
158
159

Communications
Composition
English
Journalism
Language arts
Linguistics
Literature or literary criticism
Reading
Speech

English as a Second Language
160
161
162

ESL or bilingual education: general
ESL or bilingual education: Spanish
ESL or bilingual education: other languages

Foreign Languages
171
172
173
174
175

French
German
Latin
Spanish
Other foreign language

Health Education
181
182

Mathematics
Computer science

Natural Sciences
211
212
213
214
217
218

110

Biology or life sciences
Chemistry
Earth sciences
Engineering
Physics
Other natural sciences

Special education, any

Other Education
131
132
133
134
135
136

Administration
Counseling and guidance
Educational psychology
Policy studies
School psychology
Other non-subject matter specific education

Subject Matter Specific
Social Sciences
221
222
Studies)
223
224
225
226
227
228
229
230
231
232
233
234
235

Anthropology
Area or ethnic studies (excluding Native American
Criminal justice
Cultural studies
Economics
Geography
Government or civics
History
International studies
Law
Native American studies
Political science
Psychology
Sociology
Other social sciences

Vocational, Career, or Technical Education
241
242
243
244
245
246
247
249
250

253

Health education
Physical education

Mathematics and Computer Science
190
197

General Education
Special Education

254
255
256

Agriculture and natural resources
Business management
Business support
Marketing and distribution
Health occupations
Construction trades, engineering, or science
technologies (including CADD and drafting)
Mechanics and repair
Manufacturing or precision production (electronics,
metalwork, textiles, etc.)
Communications and related technologies (including
design, graphics or printing; not including computer
science)
Personal and public services (including
culinary arts, cosmetology, child care, social work,
protective services, custodial services, and
interior design)
Family and consumer sciences education
Industrial arts or technology education
Other vocational, career, or technical education

Miscellaneous
261
263
264
265
266
267

Architecture
Humanities or liberal studies
Library or information science
Military science or ROTC
Philosophy
Religious studies, theology or divinity

Other
268

Other

Appendix B: Interview Protocol for
Second Phase of Interviews

NCES Beginning Teacher Longitudinal Survey
Interview Protocol
I. Introduction
“Thank you for agreeing to be interviewed for this project. My name is ____________, and I
work for ICF Macro, an independent company that has been hired by the U.S. Department of
Education to conduct this study. We will be asking you to help revise and improve a
questionnaire called the Beginning Teacher Longitudinal Survey.
“During this interview I am going to ask you to read and answer a number of questionnaire
items, one at a time. As you go through the questions, I would like you to explain what you are
thinking out loud, so I can get a sense of your thought process as you answer each item. For
example, if you are trying to decide what your answer is, please explain why you are unsure. If
you have trouble understanding a question, or are confused by it, please be sure to explain that
to us as well.
“The feedback you provide in this interview will be completely anonymous. In our report to the
Department of Education, we will not connect anyone’s comments with their name. The
feedback that we collect will have a direct impact on the design of next year’s survey, so please
open and honest in your comments.
“Do you have any questions before we begin?”
Before you begin, ask them the screening question(s) for their group just to confirm that they are
qualified for the interview:
All
1.

Are you currently a K-12 classroom teacher? (Yes)

All participants must answer “Yes” to Q1 to qualify.
Mentor Group
2.
Are you currently a mentor teacher?
3.

Have you ever been a mentor teacher?

Participant must answer “Yes” to one of these two questions to qualify in this group.
Reduction in Force (RIF) Group
4.
Have you taught at a school that has been affected by a reduction in force?
Participant must answer “Yes” to Q4 to qualify in this group.

II. Questionnaire Items
Go through the appropriate questionnaire items with the participant. As the participant answers
each item, record their answer on the sheet. If they hesitate while answering or reading a
question at any point, ask them to explain why.
In each case, allow them to complete 1 series (there are 4 series) of questions before asking any
follow-up or probe questions, or providing any clues as to the purpose or meaning of the
question. In this document, a series is identified by a color block. For example, all the questions
in Series 1 are blue in this document; all the questions Series 2 are yellow in this document, etc.
After the interviewee completes each series (color block), then go back and ask any follow-ups
that are necessary for each of the individual items.

ITEM 1
Have you had your contract not renewed as part of a reduction in force?
 Yes  GO TO Item 3.


No  GO TO Item 2.

ITEM 2
Have you taught at a school that has been affected by a reduction in force?
 Yes  GO TO Item 3.
 No  GO TO Item 4.

ITEM 3
What was the primary reason for this reduction in force?
 Budget cuts or budget shortfalls
 Reduced pupil enrollment
 School and/or district merger
 Reason unknown
 Other reason – please specify ________________________

Note: Items 1 and 2 are not proposed questions for the survey; they were included to provide
context for Item 3. Therefore, we are less interested in participants’ feedback on those items.
Interviewer Notes

Probes
If participant skips Item 3 because they have never taught at a school where a RIF took place,
still ask him/her the following probes.

Is there anything about Item 3 that could be confusing or unclear?

Based on your understanding, are there any other reasons that reductions in force could take
place? What are they?

How likely do you think it could be that a teacher whose contract was not renewed because of a
reduction in force (i.e., a teacher who answered “Yes” to Item 1) would not know the reason for
the reduction in force at their school?

ITEM 4
To what extent do you agree or disagree with each of the following statements?

a) My principal supports
me in classroom
management issues.
b) My principal supports
me in my interactions
with parents.
c) My principal is readily
available to me.
d) My principal listens to
my concerns.
e) My principal takes time
to visit my classroom.
f) My principal supports
my professional
development.
g) My principal provides
me with useful
feedback about my
teaching.
h) My principal has
respect for teachers.
i) My principal has
respect for students.
j) My principal treats
teachers fairly.
k) My principal
encourages
collaboration among
teachers.
l) My principal supports
student teachers in my
school.
m) My principal is readily
available to student
teachers in my school.

 There are no
student teachers in
my school
 There are no
student teachers in
my school

Strongly
Agree

Somewhat
Agree

Somewhat
Disagree

Strongly
Disagree









































































































Interviewer Notes

Probes
Could any of the parts of Item 4 be confusing or unclear? Are there any items that you feel may
be ambiguous?

Did you have difficulty answering any of these questions for any reason?

Explain your response to item (e). Why did you select the option you did? [Note: We are trying
to get at what people understand by the term “take time to visit my classroom”]

How knowledgeable would you be about items (l) and (m) if you were not actually a student
teacher? To what extent would you know whether your principal supports student teachers, or is
readily available to them?

The purpose of this question is to ask teachers to evaluate their principals based on a variety of
factors. Do you think there are any important aspects of a principal’s job that are not reflected in
this question? Are there any other items that should be added?

ITEM 5
How satisfied are you with the principal at your current school?
 Very satisfied
 Somewhat satisfied
 Somewhat dissatisfied
 Very dissatisfied
Interviewer Notes

Probes
Is there anything about Item 5 that could be confusing or unclear?

Did you have any difficulty answering this question for any reason?

When selecting your answer for this question, what factors did you consider? What were the
primary factors that determine whether or not you were satisfied with your principal?

ITEM 6
Are you currently a mentor teacher?
 Yes  GO TO Item 8.
 No  GO TO Item 7.

ITEM 7
Have you ever been a mentor teacher?
 Yes  GO TO Item 8.
 No  GO TO Item 12.

ITEM 8
Did you have input in choosing your mentee (s)?
 Yes
 No
Note: Items 6 and 7 are not proposed questions for the survey; they were included to provide
context for Item 8. Therefore, we are less interested in participants’ feedback on those items.
Interviewer Notes

Probes
If participant skips Item 8 because they have never been a mentor teacher, still ask him/her the
following probes.
Is there anything about Item 8 that could be confusing or unclear?

What kind of “input” do you think this question is referring to?

Have you ever taught at a school that had mentor teachers? If so, based on your experience do
mentor teachers usually have input in choosing their mentee(s)? If so, what kind of input?

ITEM 9
Did your principal work with you to choose your mentees(s)?
 Yes
 No

Interviewer Notes

Probes
If participant skips Item 9 because they have never been a mentor teacher, still ask him/her the
following probes.
Is there anything about Item 9 that could be confusing or unclear?

In your own words, what do you think Item 9 is asking?

(If participant has taught at a school that had mentor teachers) Based on your experience, do
principals usually work with mentor teachers to choose their mentee(s)? If so, how?

ITEM 10
Has your principal discussed your mentoring relationship with you this school year?
 Yes  GO TO Item 11.
 No  GO TO Item 12.

ITEM 11
How often does your principal discuss your mentoring relationship with you?
 At least once a week
 Once or twice a month
 A few times a year

Interviewer Notes

Probes
If participant skips Items10 and 11 because they have never been a mentor teacher, still ask
him/her the following probes.
Is there anything about Item 10 that could be confusing or unclear?

In your own words, what does the phrase “discuss your mentoring relationship with you” mean?

Is there anything about Item 11 that could be confusing or unclear?

(If participant has taught at a school that had mentor teachers) Based on your experience, do
principals usually discuss mentoring relationships with mentor teachers? If so, what are these
discussions like? How often do they usually take place?

ITEM 12
To what extent do you agree with the following statement?
Outside my school, I have a strong support system (e.g. family, close friends, neighbors).
 Strongly agree
 Somewhat agree
 Somewhat disagree
 Strongly disagree

Interviewer Notes

Probes

Is there anything about Item 12 that could be confusing or unclear?

Did you have any difficulty answering this question for any reason?

In your own words, what do you think is meant by a “strong support system”?

When selecting your answer for this question, what factors did you consider? What were the
primary factors that determine whether or not you think you have a “strong support system”?

ITEM 13
During this school year, did you or do you…
 a) Coach a sport?
 b) Sponsor any student groups, clubs, or organizations?
 c) Serve as a department lead or chair?
 d) Serve as a curriculum specialist?
 e) Serve on a school-wide or district-wide committee or task force?
 f) Serve as a consultant or coach in a particular subject?
 g) Serve as a mentor coordinator?
 h) Have a teacher union leadership position?
 i) Present at a conference?
 j) Participate in any other leadership activity? Specify: ________________________

Interviewer Notes

Probes

Could any of the parts of Item 13 be confusing or unclear?

Are there any other leadership activities that are not included on this list? If so, what?

In your own words, what do you think is meant by Item (d)? Is “curriculum specialist” a term
used in your school? If not, what do you think is the equivalent term?

In your own words, what do you think is meant by Item (f)? Is a subject area “consultant” or
“coach” a term used in your school? If not, what do you think is the equivalent term?

In your own words, what do you think is meant by Item (g)? Is “mentor coordinator” a term
used in your school? If not, what do you think is the equivalent term?