School Principal Interview

National Study of Education Data Systems and Decision Making

School Leader Interview Protocol

Introduction

The following is a suggested introduction. When you begin the interview, make these points, but use your own words for a more personal introduction:

The National Study of Educational Data Systems and Decision Making is a large-scale project sponsored by the US Department of Education to look at the ways that states, districts and schools are using data in their educational decision making. The project studies the prevalence and quality of both the data systems that are in use nationwide and the supports in place for educational decision makers to use them. In particular, we are interested to learn about how teachers are using data from these systems to make decisions about their instruction, and what types of professional development and support teachers are receiving to learn about the systems and their applications.

We want you to know that this project does not evaluate you or your school; we’re looking to learn from your experiences to get a better understanding of how teachers and schools across the country use data systems for teaching and learning.

Before we start, I need to have you review and sign this consent form.

AFTER THEY REVIEW CONSENT FORM: Do you have any questions before we get started?

District and School Context

The purpose of this section is to get background on the roles of the principal and the district in instigating data-driven decision making at the school level.

1. How long have you been at this school?

2. When did you get involved in using educational data systems to inform decisions made at this school by yourself and by teachers? Probe on reasons for using data. [IF MENTIONS SYSTEM OTHER THAN CURRENT ONE] When did you get involved in using [system name]?

[IF NOT COVERED ABOVE OR NOT USING A DISTRICT-SUPPORTED DATA SYSTEM] What role did the district play in getting you involved with data-driven decision making?

1. Did you receive support from any other group or company?

1. [IF DISTRICT INITIATED SYSTEM] What are the district’s long-term goals for using data to support teaching and learning?

   1. What do you see as the driving force for data-driven decision making in the district?

   2. Is there one person who seems to be leading the push for data-driven decision making in this district?

   3. If so, who is that? How do they communicate their vision to school staff?

2. What have been your goals, as a school administrator, for implementing the use of data for instructional decision making in your school?

   1. How do your school goals align with district goals in this area?

   2. How do you communicate the goals of data-driven decision making to your staff?

   3. What are the specific features of your schools’ data-driven decision making activities (e.g., professional development, regular data meetings)?

3. Have your school or district’s goals around data-driven decision making been impacted by NCLB? (IF YES) What changes were made? Did you have a student data system prior to the NCLB accountability requirements?

Data System

The purpose of this section is to learn more about this school’s data system [insert name] and its interoperability with other systems (e.g. district, state).

6. Do school staff enter data into [system name]?

7. What types of student data are routinely entered into [name of data system] at this school?

Check “yes” or “no” for the following types of information:

Type of Information Does the school enter…	Yes	No
a. Student demographic data?
b. Student attendance data?
c. Student standardized test score data?
d. Student grades?
e. Samples of student work
f. Student performance on benchmark tests?
g. Student participation in special programs or receipt of services? (including special education)
h. Student course enrollment histories?
i. Other? (describe):

8. What about teacher-related data? What types (if any) are routinely entered into [system name] at this school?

   Type of Information Does the school enter…	Yes	No
   a. Teacher certification/education data?
   b. Teacher professional development data?
   c. Model lesson plans or instructional resources?
   d. Other? (describe):

9. Who enters the data into the system? Is data entry a requirement for teachers? [IF YES] What data are teachers responsible for entering and how often are they supposed to enter it?

10. Are you confident in the accuracy of the data in the system? What measures are taken to ensure data accuracy?

    1. Have you had problems with discrepancies in data?

    2. Is there a process in place to deal with these kinds of discrepancies?

11. In your view, is it easy for teachers and administrative staff to enter data in [system name]?

[IF NO] How could it be better?

Data Use

In this section we are looking for details about how teachers and administrators are using data in this school.

12. Now I’m going to ask about the types of data available for school use—the types of data school staff have access to and the students for whom you have data. Check all that apply for the following types of information:

Data Type	For Last Year’s Students	For this Year’s Students	For User-defined Groups	For Individual Students
a. Student demographic data?
b. Student attendance data?
c. Student standardized test score data?
d. Student grades?
e. Student performance on benchmark tests?
f. Student participation in special programs or receipt of services? (including special education)?
g. Student course enrollment histories?
i. Other? (describe):

13. What proportion of the teachers in this school would you say look at data at least three times a year to assist in planning, conducting, or evaluating instruction?

14. How often does the typical teacher in your school access data from the system for each of these purposes (which features account for the most use)?

    Activity	Multiple times a year	At least once a year	Rarely or Never
    a. Looking at student test scores for subgroups.
    b. Tracking student promotion rates by subgroup.
    c. Tracking other measures of student progress.
    d. Determine student placement in courses or special programs.
    e. Inform parents about student progress.
    f. Inform curriculum changes.
    g. Evaluate promising classroom practices.
    h. Decide what professional development to take
    i. Other:

15. Do teachers at this school typically look at student data from other sources for these purposes? \

[IF YES] Please describe the data they look at and how the data is utilized.

16. Do teachers in this school typically look at student data as part of their own individual planning or in groups? If they do typically work in groups, what types of groups? Probe for the following:

    1. Work with colleagues in departments or grades?

    2. District-level work for school staff at individual schools

    3. District-level work for staff from multiple schools

    4. Other?

17. What do you do to “bring more teachers on board” with using data for their instructional decision making? Are there other things the district does to encourage this practice among teachers?

18. How often do you access data from the system for each of these purposes?

Activity	Multiple times a year	At least once a year	Rarely or Never
a. Looking at student test scores for subgroups.
b. Tracking student promotion rates by subgroup.
c. Tracking other measures of student progress.
d. Determine student placement in courses or special programs.
e. Inform parents about student progress.
f. Inform curriculum changes.
g. Evaluate promising classroom practices.
h. Decide what professional development to offer
i. Evaluate teacher performance
j. Other:

Support for Data-Driven Decision Making

The purpose of this section is to understand the level of support available to this school, and perceptions on level and adequacy of district support. We are also looking for information on professional development that has taken place, and what additional needs teachers have.

19. What if any resources (funds, staff, equipment) has the district allocated to your school to meet the increasing need for data collection, entry and reporting?

    1. Have these resources and support been adequate? [IF NO] What else does your school need?

    2. Any support from outside the district?

20. Do you know of staff the district has dedicated to supporting data-driven decision making in schools?

    1. Technical staff who handle questions and problems with [name of district data system]?

    2. Instructional support for teachers to learn how to use data from [name of data system] in their instructional practice.

    3. How would you rate the quality of the support these staff provide?

21. What kinds of professional development on data-driven decision making or training to use [name of data system] have your teachers participated in?

    1. Who was the provider?

    2. What was the focus (e.g., state assessment data, subject-specific topics)?

    3. How much professional development came from (a) the district, (b) the state, and (c) other sources.

    4. What proportion of professional development focused on learning to use the system (technical)?

    5. How much professional development focused on using data for instructional decision making.

22. Do you feel that the amount of professional development for your teachers on using data in their teaching has been too much, not enough, or just about right?

    1. How would you characterize the quality of the teacher professional development on data use your teachers have received?

    2. Who makes the decision about what and how much professional development in data-driven decision making your teachers get?

23. Do you feel that teachers in your school have any unmet needs in regard to professional development in the use of data for instructional purposes? If so, what are they?

24. How much and what kind of professional development on data-driven decision making is provided to your non-teaching staff involved in data entry, including during the summer and any on-line offerings?

25. What have been the biggest challenges you have faced as you have worked to implement data-driven decision making at this school? [Probe on past and current challenges] What factors have facilitated the implementation? [Probe on factors at the district and school level]

Effects of Data Driven Decision Making at the School

26. Do you believe that use of [system name] has changed teaching and learning at your school? [IF YES] Please describe.

27. Do you believe that students in your school have benefited in any way from your increased use of data-driven decision making? [IF YES] Please describe. [IF NO] Why not? What will it take before students do benefit?

28. Do you believe that faculty have benefited in any way? [IF YES] How and to what do you attribute this? [IF NO] Why not?

29. How has the [name of system] affected administrative functions?

30. Do you see costs or cost savings associated with use of the system? How do they balance out (net cost or net savings)? Can you describe and estimate the size of the cost/savings?

31. Are there any other things you would like us to know about your use of data for instructional decision making in this school that we have not already covered?

Closing

32) What advice would you have for other schools who want to use data to improve instruction?

33) Are there any other things you would like us to know about your work in the school with DDDM that we have not already covered?

Before you leave, be sure to ask for copies of any documents such as student handbooks, curriculum frameworks, standards, teacher professional development materials, and school improvement and accountability materials.

National Study of Education Data Systems and Decision Making

Teacher Interview Protocol

Introduction

The following is a suggested introduction. When you begin the interview, make these points, but use your own words for a more personal introduction:

The National Study of Educational Data Systems and Decision Making is a large-scale project sponsored by the US Department of Education to look at the ways that states, districts and schools are using data in their educational decision making. The project looks at the prevalence and quality of both the data systems that are in use nationwide and the supports in place for educational decision makers to use them. In particular, we are interested to learn about how teachers are using data from these systems to make decisions about their instruction, and what types of professional development and support teachers are receiving to learn about the systems and their applications.

We want you to know that this project does not evaluate you or your school; we’re looking to learn from your experiences to get a better understanding of how teachers and schools across the country use data systems for teaching and learning.

Before we start, I need to have you review and sign this consent form.

AFTER THEY REVIEW CONSENT FORM: Do you have any questions before we get started?

Background

1. Please tell me a little bit about your role here at this school.

1. What subjects and grade levels do you teach?

2. Do you have any additional roles at the school, especially with respect to curriculum or the use of data?

Data Use for Instructional Decision Making

The purpose of this section is to learn more about your use of this school’s data system [insert name]

2. How often do you access the system [insert name]?

3. What situations have prompted you to use the system? NOTE: If the teacher has access to a computer during the interview, ask them to open the data system and show you the reports and types of data they typically access for any situations they say led them to use data.

   1. Have you used data from the system in planning your curriculum for the year?

[IF YES] What did you conclude from the data? How often have you accessed data for this purpose?

[IF NO] Why not?

b. Have you used data from the system in forming groups of students within you class?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose?

2. Have you used data from the system to develop individualized learning plans for some or all of your students?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose this year?

3. Have you used data to make decisions about how long to spend on specific instructional objectives?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose this year?

4. Have you used data collected during the school year to see whether your students are on track to meet grade-level standards?

[IF YES] Can you describe your process for doing this? How often have you accessed data for this purpose?

5. Have you used data from other sources or other data systems for any of these purposes?

[IF YES] Please describe and explain why you used this other source instead of (or in addition to) the [name] data system.

4. Do you typically look at student data for these purposes on your own or as part of a group? [IF AS PART OF A GROUP] Please describe.

   1. Are different types of data looked at in groups vs. individually? Why these differences?

   2. Which approach is most useful?

[IF THE TEACHER ALREADY PROVIDED SPECIFIC EXAMPLES IN RESPONSE ABOVE QUESTIONS SKIP TO QUESTION 7. IF NO SPECIFIC EXAMPLES HAVE BEEN PROVIDED, ASK QUESTION 6]

5. Think of a specific situation in which you used data to make a decision about your instructional practice. We’d like to hear about your decision and your thinking about data’s impact on your instruction.

   1. Who did the decision concern?

   2. What data did you use and what did it tell you?

   3. What was the result of the decision?

   4. How comfortable did you feel making decisions based on the data?

6. Overall, what do you feel has been the impact of your data use on students’ educational experiences?

7. Have you had questions or needed help using the system or interpreting data from the system?

[IF YES] Did someone help you? If yes, please describe. Who can you turn to for help in your school or district?

8. How could [data system name] be more useful to you as a teacher?

   1. What other kinds of student information would you like to be able to access?

   2. In what ways could the system be easier to use?

Student Data Scenario Discussion

In this final section of the interview we will be asking teachers to look at and interpret mock-up scenarios based on hypothetical student data. These mock-ups will provide an opportunity to assess teachers’ understanding of common data interpretation tasks, and to understand what types of decisions, if any, they might make on the basis of their interpretations.

The scenarios will be developed by TWG members with data system expertise and assessment experts at a workshop to be held in Menlo Park during the summer of 2006. The scenario will ask the teacher to look at graphs or tables of student data and describe what the data tell them and the instructional inferences they would draw. Data concepts to be tested may include:

• Variance

• Measurement error

• The difference between group and individual performance

• Estimating reliability related to the number of items

We will be developing these scenarios in consultation with TWG members and experts in data-driven decision making. The timeline for creation of the scenarios is….

National Study of Education Data Systems and Decision Making

Other Data Use Meeting Protocol

Observer(s):

Date of observation:

Length of observation:

Details about the Meeting

Name of meeting:

Meeting focus:

Length of the meeting:

Frequency this type of meeting is held (e.g. once a month, 2-3 times a year):

Participants (number and position(s):

Meeting leader (indicate school role):

Meeting Notes

Take notes on the meeting activities and discussion. Where possible, capture any particularly revealing quotes and the position of the person speaking (e.g. teacher, technology coordinator). As soon after the meeting as possible, answer the questions below, based on your notes and memory:

• What is the purpose or focus of the meeting?

• Does the meeting appear to be one of a series of similar meetings, a one-time event, or “staged” for the site visit?

• What types of data are discussed?

• Where did the data come from and which student groups does it pertain to?

• What, if any, questions do teachers express concerning how to access data?

• What, if any, questions do teachers have about data interpretation?

• What, if any, actions are proposed based on data?

• What, if any, insights does the group gain as a result of looking at data together?

• Do teachers question the relevance, validity, or reliability of the data? [IF SO] Please describe.

• Do teachers describe their instructional challenge in terms of certain demographic groups (e.g., boys or ELL or African American students) . . .

  • At the beginning of the meeting?

  • After looking at data?

[IF THERE IS A CHANGE IN THIS RESPECT, PLEASE DESCRIBE. INDICATE WHETHER POTENTIAL APPROACHES TO IDENITIFIED PROBLEMS ARE PROPOSED FOR STUDENTS ON THE BASIS OF DEMOGRAPHIC CHARACTERISTICS, PRIOR EDUCATIONAL EXPERIENCES, OR CURRENT ACADEMIC PERFORMANCE]

Artifacts Collected

List here any artifacts or information collected from the meeting, for example:

• Agenda

• Activities

• Professional development materials

• Printouts of student data (with names or other identifying information removed)

D-13