Case Studies

Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems

Att_1875-NEW (3992) rev App E Case Study District Interview Materials 102009 OMB changes accepted

Case Studies

OMB: 1875-0254

Document [doc]
Download: doc | pdf



Appendix E to OMB Clearance Package

Case Study District Interview Materials



District Case Study Notification Email



Date: [October 2009]

To: [District Title III Coordinator]

CC: [District Superintendent]

Subject: National Evaluation of Title III Implementation

Attachments: Title III District Brochure, Title III District Survey Instructions



The National Evaluation

of Title III Implementation





Dear [District Title III Coordinator],



We are writing to provide you with the information needed for your district to participate in the National Evaluation of Title III Implementation. Your participation, along with that of districts across the country, will ensure that local perspectives on Title III and English language acquisition and enhancement are represented.



Funded by the U.S. Department of Education, this study is designed to provide Congress, the U.S. Department of Education, other policymakers, and educators with important information about states’ and districts’ implementation of Title III provisions, as of 2009-10. The results of this study will help to inform the next reauthorization of the Elementary and Secondary Education Act (ESEA). For further information about the study and its data collection procedures, please consult the attached study information document.



Your district has been selected to participate in two key components of the study. First, you are invited to participate in the study’s on-line survey of Title III-funded districts and consortia. Please find specific instructions for completing the survey, which is already available on-line, in the attached Title III Survey Instructions document. Second, your district has been selected for the case study component of the study. Our visit to your district, and several others, will provide an in-depth picture of Title III implementation and the critical local perspective in this important national evaluation. Case study director Laura Golden will contact you shortly to explain the case studies in detail and make arrangements for visiting your district.



Should you have any questions about the survey, please contact Dr. Paul Hopstock at [email address] or [telephone number]. If you would like to inquire about the case study visits, please contact Laura Golden at [email address] or [telephone number].



Thank you very much in advance for your cooperation. We truly appreciate your willingness to provide the time and expertise needed for the success of this important study.





Sincerely,





Jennifer O’Day, Ph.D. James Taylor, Ph.D.

Principal Investigator Project Director

National Evaluation of Title III Implementation National Evaluation of Title III Implementation



The National Evaluation of Title III Implementation is funded by the Policy and Program Studies Service (PPSS) of the US Department of Education and conducted by the American Institutes for Research (AIR) with its partners the Windwalker Corporation and edCount, LLC.  The valid OMB control number of this information collection is XXXX-XXXX.

District Informational Brochure

The National Evaluation

of Title III Implementation



District Information Document



What is the National Evaluation of Title III Implementation?

The National Evaluation of Title III Implementation, sponsored by the U.S. Department of Education’s Policy and Program Studies Service, is the first Federal study to take an in-depth look at the implementation of Title III at the state and local levels. This study is unique in that it goes beyond examining states’ implementation of Title III provisions to take a closer look at how these provisions play out at the local level to shape the education of English Language Learners (ELLs).

To this end, during the fall of 2009, the study is complementing its state-level research activities – which include interviews with all 50 states’ Title III directors, reviews of state policy documents, and analyses of several states’ student achievement data – with the following district-level data collections:

  1. A nationally-representative survey of Title III-funded districts and consortia

  2. In-depth case studies of several Title III-funded districts, nested within 5 states



Why is this study important?

This study is designed to provide Congress, the U.S. Department of Education, other policymakers, and educators with an up-to-date, in-depth look at how states and districts across the nation are implementing Title III provisions as of 2009–10. The study will inform the next reauthorization of the Elementary and Secondary Schools Act (ESEA) and provide important information about English Language Proficiency (ELP) standards, ELP assessments, annual measurable achievement objectives (AMAOs), and activities designed to improve outcomes for English language learners.

Findings from the study will be released in a series of three evaluation briefs and two reports, which will be publicly available on the U.S. Department of Education’s Web site. Executive summaries of the study reports will also be sent to participating state and district Title III officials.

Is participation in the study required?

District administrators are required to participate in the study as a condition of receiving Title III funds. Individuals’ participation in the principal, teacher, and parent liaison focus groups is voluntary.

How were districts selected?

Districts that received Title III funds during the 2008-09 school year were randomly selected for participation in the survey. Twelve of these districts were selected for participation in the case studies based on numerous factors, including their size, locale, and ELL student population.

What are the benefits for participating?

Your participation will contribute to a greater understanding of how states and local agencies are implementing Title III policy and programs for ELL students.

Your participation will ensure that policymakers, researchers, and practitioners have access to important information about the implementation of Title III from the perspective of local school districts and educators.

Participating states and districts will receive executive summaries of the study reports.

Case study districts will also receive:

  • a dedicated study site coordinator who will serve as the single point of contact for the district;

  • small tokens of our appreciation for district focus group participants; and

  • a web-based briefing on the study results.



What will the survey require?

The Web-based survey can be accessed anytime online and should take about 45 minutes to complete. It can largely be completed without reference to district or Title III records.

What will the case studies involve?

The districts participating in the case studies will be visited for two days by a professional, experienced team of researchers who will conduct:

Interviews with:

  • District ELL or Title III Coordinator

  • District Curriculum and Instruction Director

  • District Assessment Director

Focus Groups (4-8 participants) with:

  • Elementary and secondary school principals/resource staff

  • Elementary and secondary school teachers

  • Parent liaisons and parents



Each interview or focus group should take approximately 60 minutes. No special preparation is needed. Questions will focus on participants’ daily work experiences and will be easily answered. Our study site coordinator will work to schedule our two-day visit at a convenient time and place for the district and will attempt to schedule our focus groups at a time and place adjacent to existing staff and parent meetings outside of school hours to reduce disruptions to the school day and the need for travel.

What kinds of questions will be asked?

The survey and case study site visits will feature questions about how districts:

  • use state ELP standards;

  • assess ELLs for identification, placement, and academic progress;

  • monitor student participation and progress in language instruction programs;

  • respond to Title III accountability designations;

  • ensure teacher quality and continued professional learning regarding ELL instruction; and

  • assist schools in differentiating instruction for diverse types of ELLs.



Will district-provided information be confidential?

You and your district’s or school’s identities will be protected; immediately after we receive your responses, your name and the district’s or school’s name will be dissociated from the data. Your responses will be used to summarize findings in an aggregate manner (across groups of subgrantees), or will be used to provide examples of program implementation in a manner that does not associate responses with a specific site or individual. The study will identify states in its reports.

What will be done to ensure the quality of the research?

The research team will assume responsibilities designed to protect the participants and to ensure that the data collection activities are of the highest quality and lowest burden possible. These responsibilities include:

  • Assuring the confidentiality of participants involved in data collection activities through an approved data security plan;

  • Undergoing thorough IRB (institutional/ethical) review related to data collection activities;

  • Obtaining Office of Management and Budget clearance for the data collection. The valid OMB control number of this information collection is XXXX-XXXX;

  • Piloting surveys and protocols with state and district respondents in order to refine and improve the content and procedures of the data collection instruments used in the full study;

  • Periodically presenting the study design and results for review by an expert technical advisory panel that includes a teacher of ELLs, a district Title III administrator, a state Title III director and several nationally-known experts in methodology and English language acquisition and enhancement.



Who is conducting the study?

The National Evaluation of Title III Implementation is sponsored by the U.S. Department of Education’s Policy and Program Studies Service and is being conducted by:

  • The American Institutes for Research (AIR) (www.air.org), leads the study with Dr. Jennifer O’Day serving as principal investigator. AIR is a not-for-profit, non-partisan organization in Washington, DC with a 50-year history of conducting applied research and providing technical assistance in the areas of education, health, workforce and international development.

  • The Windwalker Corporation (www.windwalker.com), is a small business located in Virginia that is experienced in conducting on-line surveys and surveys on topics related to English language learners.

  • edCount, LLC. (www.edcount.com), is a small business in Washington, DC that specializes in standards, assessments, and accountability policy with a staff that includes several former ESL educators and Title III administrators who are experienced in case study research methods.



For more information, please contact:



The National Evaluation of Title III Implementation


Informed Consent

Purpose

The Policy and Program Studies Service (PPSS), Office of the Under Secretary, U.S. Department of Education (ED) requests clearance for the data collection for the Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems. The purpose of the study is to provide an up-to-date, in-depth picture of implementation of No Child Left Behind’s (NCLB) Title III provisions across the nation as of 2009-10. To this end, the evaluation will employ multiple lenses through which to view the patterns and complexities of implementation at both the state and district levels.

To assist with the evaluation, we are asking district staff to participate in interviews. Depending on your role at the district level, you will be interviewed about topics such as assessment of English learners, Title III accountability, and improvement strategies to support English learners. The interviews are designed to last no more than one hour.

Risks and Discomfort

There are few anticipated or known risks in participating in this study.

Benefits

Your participation in the evaluation will contribute to an understanding of how local agencies are implementing Title III policy and programs for English language learners.

Confidentiality

We will treat the information that you supply in a confidential manner. Only selected research staff will have access to data. We will NOT present results in any way that would permit them to be identified with you or any other specific individual. No personally identifiable information, such as name or school/district affiliation, will be disclosed to anyone outside the project, except as required by law.

More Information

If you would like more information about this study, you may contact the Project Director, James Taylor, at the American Institutes for Research at (202) 403-5000 or at jtaylor@air.org. For questions regarding your rights as a subject participating in this research, please IRB at IRBChair@air.org, toll free at 1-800-634-0797.

Informed Consent

I have read the above information. I have asked questions and received answers. I consent to participate in the study.


Signature: _____________________________________ Date:____________________________________

Print Name:_______________________________________ Position:__________________________________

District/School Name:_____________________________________

Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems

District Assessment Director Interview Protocol

State:

District:

Respondent’s name and title:

Date of Interview:

Interviewer:

Note-taker:

Introduction

Thanks again for taking the time to speak with me this morning/afternoon. Before we start, I’d like to provide a little background on our work, and answer any questions you might have for me.

As you probably know, I work for an independent non-profit research organization called the American Institutes for Research, and we are conducting an evaluation of Title III implementation for the US Department of Education


I want to assure you that all information obtained today will be kept confidential and will only be used for the purposes of this study. We will not use your name, or your district’s name, and will not attribute any quotes. We also will not share what you and I discuss with other people in this district. Our study will identify the states that we visit, but will not disclose the names of the districts in each state.


I’d like to ask you to sign a consent form before we begin. It outlines some of the issues I’ve just mentioned with regard to anonymity and confidentiality. Please take a minute to read it and let me know if you have any questions.


If you don’t mind, I would like to record this interview simply for note-taking purposes. No one outside of our research team would hear the recording, it would just be for my own reference. If you would like me to turn off the recorder at any point, just let me know. Would that be OK?


I have three main areas that I would like to discuss with you today. I’ll be talking to you about issues related to assessment of English learners, Title III accountability, and improvement strategies to support English learners.


Do you have any questions before we begin?


Note to interviewer: Throughout the interview, possible probes are set aside following each general question. Asking these questions may help to move the interview pace or may prompt a less talkative respondent. However, do not feel you need to ask each of these probes; it is likely that the respondent will cover many of these issues when responding to your initial inquiry. Keep the tone conversational and comfortable.









District Assessment Director

This interview will take place with a district official who is knowledgeable about ELL assessment and implementation of Title III accountability – could be assessment director and/or Title III coordinator. This protocol focuses on assessment, accountability, communication with stakeholders, etc. This interview is expected to take approximately one hour.

Background

  1. I’d like to start by talking a bit about your background and your role in the district. Can you briefly describe your background, especially as it relates to assessment and English language learners? Can you tell me about your primary responsibilities? (4.1)



Probe, if necessary:

  • How did you come to be involved in school administration?

  • How many years have you been involved with this school district?

  • Have you ever been in a district with a sizeable ELL population?

  • Probe for capacity to fulfill this role, training received, support provided, constraints, how many others share similar responsibilities, what other responsibilities s/he has,



District Policies regarding ELLs

Next, I am interested in learning more about some of your district policies that pertain to students who are English language learners.

  1. How are students initially identified as ELL? And how is re-designation determined? (2.1.3)



Probe for:

  • Measures used, such as state ELP test, district assessments, teacher recommendation, and other measures. (2.1.3) (3.2.3.5) (3.2.3.6)



  1. With regard to placement in LIEP programs – how is a student’s placement determined for participation in programs and supports available for ELLs in the district? How is exit from them determined? (2.1.3) (3.2.3.5) (3.2.3.6)



Probe for:

  • About how many ELL students are participating in LIEP programs in your district? How long do they tend stay in various program types? (for example, in the case of a bilingual program, do they stay in it? or transition out to English only?)

  • How do ELL students transition from one program type to another?

  • Perceived effective of placement procedures


  1. How are ELLs identified for participation in special education? What services do they receive? (2.1.3)



Assessment of ELL students

  1. I know that your state uses ________ as the test of English language proficiency. How does your district use the information you get from that test? (2.1.3) (2.1.4) (2.1.5)

Probe for:

  • Kinds of analyses the district conducts with the results and how used



  1. In addition to this test, are there any other measures your district uses for assessing the progress of ELLs? What are they? How does your district use the information you get from these measures? (2.7.1)

Probe for:

  • How frequently are these other assessments administered? Do teachers develop their own measures of English proficiency?

  • Are there instructional tools that are used to measure progress more frequently than the annual state test?

  • How is data collected, shared, and used to drive instruction?

Listen for:

  • Use of state ELP standards and content standards in development of formal and informal assessments (1.1.6)




Title III Accountability

As you may be aware, under NCLB, there has been a new emphasis on accountability for English language learners, and I’m interested to talk with you about how Title III accountability has played out in your district.

  1. I’d like to ask you specifically how your ELL students are doing with regard to targets associated with NCLB. First, has your district met AYP for the ELL subgroup? Have there been any specific steps you’ve taken to address your AYP status? (5.1) (3.2.3.1) (3.2.3.2)

Listen for:

  • Any changes in ELL practices over time associated with AYP status



  1. Next, let’s talk about your targets for English learners under Title III – that is, the Annual Measurable Achievement Objectives, or AMAOs. Can you tell me about the three AMAO targets your district needs to meet this year? (3.1.1.2) (3.1.1.3)

Probe for:

  • How are each of the AMAO targets defined

  • Do these targets seem appropriate for your district – that is, are they too rigorous, or within reach?

  • Do you feel that stakeholders in your district have adequate understanding of the Title III AMAOs? (3.2.1)

Listen for:

  • Awareness and level of understanding of the targets

  • Distinctions between AMAO 1, 2, and 3





  1. How is your district doing in terms of meeting AMAO targets? Which targets have you met or missed, and for how long? (5.1.1)





  1. If the district has missed AMAOs for at least two years: I’d like to know more about what happens if you don’t meet your AMAO targets. So after you missed AMAOs for [2 or 4] years, what was the first consequence – did you work with the state to develop an improvement plan? (3.2.3.1)

Probe for:

  • Who determines these consequences (that is, which stakeholders are involved at the district level, how much does the state intervene?)

  • How did your Title III improvement efforts overlap with Title I consequences?



  1. If the district has missed AMAOs for at least two years: What actions, specifically, did your district implement as a result of missing Title III AMAOs? (2.1.5) (3.2.3.1) (3.2.3.2)

Probe for:

  • Specific actions that have been taken in this district related to accountability (i.e., which of the measures have actually been implemented and what evidence we might see of that)




  1. If you compare Title III accountability to Title I, which do you think is more meaningful (or salient) for stakeholders in your district? In other words, which system of accountability do you think stakeholders pay more attention to and why? Which do you think is more likely to have consequences for English learners? [Note, this may come up spontaneously in response to the previous two questions] (3.2.3.3)

Probe for:

  • How do the consequences associated with missing AYP (under Title I) relate to the consequences associated with missing the additional AMAOs (under Title III)

  • What is added by having the additional AMAO targets, awareness of AMAO targets and how much attention is paid to them


  1. If the district has missed AMAOs for at least two years: What assistance do you receive associated with missing AMAO targets? (4.3.3)

Probe for:

  • Assistance provided by the state and who receives


  1. For districts participating in a consortium: Given that you are working in a consortium of districts, how are accountability consequences determined across participating districts? Do you work together on consequences, or do each of you develop your own improvement plan? (3.2.3.1) (3.2.3.2)



Communication with stakeholders

  1. I know that both states and districts have responsibility for reporting about student performance under Title I. With regard to communication about Title III accountability results, who takes the lead on that? What is the process for communication? (3.2.2.1) (3.2.1.2)

  • When do districts and schools learn how they did on Title III tests? When do you receive AMAO results from the state?

  • Do teachers and principals generally receive the results of students’ ELP tests? Do they learn how their school did on AMAOs under Title III?


  1. How are parents informed of their school’s performance? When do you provide this information to parents? (3.2.4.1) (3.2.4.2)

Probe for:

  • How needs of ELL parents are accommodated, such as notification in home language

  • Any differences at elementary and secondary levels



Concluding Questions

  1. Taking into account everything we’ve discussed, what are your thoughts on the progress of ELL students in your district? (5.3)



  1. Wrapping up, could you describe any challenges associated with Title III implementation that we haven’t already discussed? I’m particularly interested in your perspective on assessment issues as they relate to English learners. (2.1) (2.5) (2.6) (2.7)



Also probe for:

  • Unique contextual issues

  • Capacity challenges/strengths (LEA level)

  • Instructional challenges/strengths



  1. Are there any highlights associated with ELL assessment that you did not previously mention? Anything I should have asked you, but didn’t? (2.1) (2.5) (2.6) (2.7)





Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems

Director of Curriculum and Instruction Interview Protocol

State:

District:

Respondent’s name and title:

Date of Interview:

Interviewer:

Note-taker:

Introduction

Thanks again for taking the time to speak with me this morning/afternoon. Before we start, I’d like to provide a little background on our work, and answer any questions you might have for me.


As you probably know, I work for an independent non-profit research organization called the American Institutes for Research, and we are conducting an evaluation of Title III implementation for the US Department of Education


I want to assure you that all information obtained today will be kept confidential and will only be used for the purposes of this study. We will not use your name, or your district’s name, and will not attribute any quotes. We also will not share what you and I discuss with other people in this district. Our study will identify the states that we visit, but will not disclose the names of the districts in each state.


I’d like to ask you to sign a consent form before we begin. It outlines some of the issues I’ve just mentioned with regard to anonymity and confidentiality. Please take a minute to read it and let me know if you have any questions.


If you don’t mind, I would like to record this interview simply for note-taking purposes. No one outside of our research team would hear the recording, it would just be for my own reference. If you would like me to turn off the recorder at any point, just let me know. Would that be OK?


During our conversation, I’ll primarily be asking you about your vision for instruction in the district, instructional practices, professional learning, and utilization of resources, with an orientation toward how those areas relate to the needs of English language learners.


Do you have any questions before we begin?


Note to interviewer: Throughout the interview, possible probes are set aside following each general question. Asking these questions may help to move the interview pace or may prompt a less talkative respondent. However, do not feel you need to ask each of these probes; it is likely that the respondent will cover many of these issues when responding to your initial inquiry. Keep the tone conversational and comfortable.









District Director of Curriculum and Instruction

This interview will take place with a district official who is knowledgeable about content area instruction in the district, such as director of curriculum and instruction, chief academic officer, deputy superintendent, and in a smaller district, it might be the superintendent. This protocol will allow us to get a general sense of the district’s vision for instruction, how district programs are set up, how resources are utilized, integration of content and language instruction, support for parents, and professional learning. The use of this protocol will have to be tailored toward the role of the individual being interviewed, as well as toward the context of the particular district in which this interview is taking place. This interview is expected to take approximately one hour.



Background

  1. I’d like to start by talking a bit about your background and your role in the district. Can you tell me about your primary responsibilities? (4.1)



Probe, if necessary:

  • How many years have you been involved with this school district?

  • How did you come to be involved in school administration?

  • To what extent does your role include consideration of ELL issues?

  • Have you ever been in a district with a sizeable ELL population? What background do you bring in the use of ELL practices?


  1. Next, can you tell me a bit about the district context? How would you describe your district to a visitor? (6.1)

  • Background on district, including SES levels

  • Are there unique contextual factors we should know about – whether related to leadership, teaching force, recent reform issues, or ELL challenges?


  1. In particular, how would you describe the ELL student population in your district? (6.2, 6.3, 6.4, 6.5, 6.6, 6.9)

  • Size and nature of ELL population (including language backgrounds and levels, levels of prior education), significant subpopulations of ELLs and their characteristics

  • Ways in which ELLs are distributed across the district (i.e., do they tend to be grouped in certain areas and schools? or spread throughout the district?)

  • Description of any unique contextual factors that are noteworthy with regard to ELL practices, such as participation in a consortia of districts for Title III purposes








District instructional context

  1. What are some of the main priorities and goals that guide instruction in this district? Are there specific priorities and goals for ELL students? (1.1.7)

Listen for:

  • Use of a mandated/ intended curriculum for some subjects (such as Open Court)

  • How goals/priorities play out at elementary and secondary levels

  • How goals/priorities play out with regard to ELL students

  • Use of state standards (content and ELP)


  1. I’d like to know more about the curriculum used in your district’s schools. Is there a district-wide curriculum in place for reading or mathematics? How or why was it selected? In what ways is it effective (or not) for the ELL students in your district? (1.1.6) (1.1.8) (1.1.9)

Listen for:

  • Extent to which ELL needs were considered in selection of curriculum

  • Role of state in curriculum selection, extent to which the district has autonomy

  • Differences at the elementary and secondary levels

  • Use of state standards in decision making – including ways in which state standards influence text selection, curriculum development, formal and informal assessments, etc.



  1. What do you see as the primary strengths that ELL students bring to the instructional environment in this district? What do you see as the primary needs of the ELL students in this district? (3.2.3)

Listen for:

  • The extent to which ELL needs are perceived as different from those of the general population.

  • The extent to which the interviewee articulates distinct needs among the ELL population

  • The extent to which ELL needs are addressed within the general population.



  1. What are some of the key strategies for addressing ELL needs in your district? (1.1.8) (3.2.3.5) (3.2.3.7) (3.2.3.8)

Probe as necessary:

  • Role of respondent in carrying out ELL strategies and practices

  • How the specific needs of ELLs who are struggling are addressed (e.g., Response to Intervention or RtI)

  • Ways in which instruction is differentiated for needs of individual ELLs in content classes (probe for academic language)

  • Use of native language in classes with ELL students

  • Ways in which ELD instruction is incorporated in content area instruction

  • How this plays out at elementary and secondary levels

  • Specific needs of significant subpopulations of ELLs that may have been identified and how those are addressed.

  • Use of state standards (content and ELP)



  1. How are your district’s ELL students doing academically? How do you know? How do you use the information you get? (5.3, 2.1, 2.7, 4.5.7)

Probe for:

  • How the information is used to shape professional development and drive instruction

  • Ways in which assessment data is used to inform decisions regarding curriculum and instruction for ELLs

  • Ways in which various types of assessment data are used, such as ELP assessments, content area assessments, district assessments, periodic assessments, etc.

  • Any changes over time in types of data tracked and how used





Teacher quality

  1. Next, I’d like to ask you to describe the teaching force in this district. To what extent do you feel teachers are prepared to meet the needs of students in this district? To what extent do you feel teachers have the skills and knowledge needed to teach ELL students? (4.5)



Listen for:

  • Capacity of content teachers and ELL teachers

  • Any changes in teacher capacity over time

  • Differences at the elementary and secondary levels



  1. How are those teachers who are best able to address the needs of ELL students distributed across the schools in this district? Are there any policies or practices for assigning those teachers to particular schools? (4.5.4) (4.5.5)

Listen for:

  • Extent to which stronger teachers are in the schools with the largest numbers of ELLs







  1. How does your district go about recruiting and retaining teachers who are knowledgeable both in their content area and in English language instruction? Are there any challenges you face in recruiting and retaining the teachers you need? (4.5.2)

Listen for:

  • Differences at the elementary and secondary levels

  • Extent to which ELL qualifications are a factor in recruiting content area teachers and which ELL qualifications are considered

  • Challenges associated with recruiting teachers with appropriate skills for teaching ELL students and how addressed

  • Challenges associated with recruiting and retaining teachers, such as competition from a neighboring district, etc.



  1. How do teachers in your district demonstrate their fluency in English?   Do any teachers in your district provide instruction in languages other than English?  If so, how do teachers demonstrate fluency in languages other than English? (4.5.1)

Listen for:

  • Awareness of Title III or state teacher fluency requirements        

  • Challenges associated with implementing teacher fluency requirements



Professional learning

  1. Now I’d like to ask you about the professional learning that teachers engage in. (If we are able to talk to a PD person in the district, these questions should be asked of that person in depth.) What are the primary types of professional learning available to teachers in your district? Are there any opportunities specifically related to instruction for ELL students? What are they? What is the district’s role in providing these opportunities? (1.1.9) (4.3.3) (4.5.7)

Probe for:

  • A complete description of each type: What is the focus (and how it relates to the priorities expressed earlier)? Who provides? Intensity? Perceived effectiveness? What has been the impact on instruction? What has been the impact on academic achievement? Etc.?

  • Who determines the content, and who determines which teachers should attend

  • Differences at the elementary and secondary levels

  • Differences in what is provided for ELD teachers vs. content area teachers

  • Is any PD mandatory? Are there incentives to participate in PD?


Listen for:

  • Use of state standards

  • Any assistance that is provided by state



  1. What kinds of professional learning do principals and administrators engage in regarding instruction for ELL students? (1.1.9) (4.3.3)

Listen for:

  • Support provided by district, any assistance provided by state

  • Use of state standards



  1. As a district leader, where do you turn for assistance on ELL issues or to build your own professional knowledge in this area? (4.3) (4.1.4)



Resources

  1. Next, I’d like to get a sense of the kinds of resources you draw on in addressing the needs of the ELL students in your district. How does your district use its Title III funds? What other financial resources do you access to support ELL practices? What other kinds of resources do you use? (4.1.1)

Probe for:

  • Use of Title I funds

  • How money is used to support academic achievement as well as language proficiency and its impact

  • Any challenges encountered with respect to “supplement not supplant" provisions in Title I or Title III?



  1. Are there resources (financial, human, or other) you feel you need in order to fully address ELL needs in this district, but are lacking? If so, what are they? (4.1)



Parent support

  1. Are there any district-wide programs or supports available for parents? Can you describe these? (3.2.3.10)



Probe for:

  • Which parents are targeted for these programs, who provides the support, the focus, etc.

    • Elementary parents

    • Secondary parents

  • In what ways do they benefit parents of ELLs in particular?



Concluding Questions

  1. Are there any gaps in what you see that ELLs need and what you are able to provide in the district? What challenges and constraints do you face in providing what’s needed? (4.1)

Listen for:

  • Any changes over time in how the district has been able to address ELL needs?



  1. Wrapping up, are there any highlights that you did not previously mention? Anything I should have asked you, but didn’t?

Evaluation of State and Local Implementation of Title III Standards, Assessments, and Accountability Systems

District ELL Coordinator Interview Protocol

State:

District:

Respondent’s name and title:

Date of Interview:

Interviewer:

Note-taker:

Introduction

Thanks again for taking the time to speak with me this morning/afternoon. Before we start, I’d like to provide a little background on our work, and answer any questions you might have for me.


As you probably know, I work for an independent non-profit research organization called the American Institutes for Research, and we are conducting an evaluation of Title III implementation for the US Department of Education


I want to assure you that all information obtained today will be kept confidential and will only be used for the purposes of this study. We will not use your name, or your district’s name, and will not attribute any quotes. We also will not share what you and I discuss with other people in this district. Our study will identify the states that we visit, but will not disclose the names of the districts in each state.


I’d like to ask you to sign a consent form before we begin. It outlines some of the issues I’ve just mentioned with regard to anonymity and confidentiality. Please take a minute to read it and let me know if you have any questions.


If you don’t mind, I would like to record this interview simply for note-taking purposes. No one outside of our research team would hear the recording, it would just be for my own reference. If you would like me to turn off the recorder at any point, just let me know. Would that be OK?


I have several main areas that I would like to discuss with you today. I’ll be talking to you about issues related to the instruction, curriculum, and assessment of English language learners, as well as teacher quality and professional development.


Do you have any questions before we begin?


Note to interviewer: Throughout the interview, possible probes are set aside following each general question. Asking these questions may help to move the interview pace or may prompt a less talkative respondent. However, do not feel you need to ask each of these probes; it is likely that the respondent will cover many of these issues when responding to your initial inquiry. Keep the tone conversational and comfortable.









ELL Coordinator

This interview will take place with a district official who is knowledgeable about ELL instruction in the district – could be an ELL Coordinator or possibly Title III coordinator. (If we talk to a PD person and/or an HR person, those sections of this protocol should be used.) This protocol focuses on instruction and programming for ELL students, professional learning, recruitment and retention of qualified ELL teachers, etc.

This interview is expected to take approximately 90 minutes. To the extent possible, this interview should be conducted after the other two district interviews so that wherever possible these questions can be skipped or scaled back if sufficient information has been gathered from other interviews.

Background

  1. I’d like to start by talking a bit about your background, especially as it relates to the instruction and assessment of English language learners. Can you briefly describe your background and your role in the district? (4.1)



Probe, if necessary:

  • How many years have you been involved with this school district?

  • Have you ever been in a district with a sizeable ELL population? Probe for capacity to fulfill this role, training received, support provided, constraints, how many others share similar responsibilities, what other responsibilities s/he has, etc.

  • How did you come to be involved in school administration?



  1. Next, can you tell me a bit about the district context? How would you describe the ELL student population in your district? How would you describe the rest of the student population? (6.1, 6.2, 6.3, 6.4, 6.5, 6.6, 6.7, 6.8, 6.9)



  • Background on district, including size and nature of ELL population

  • Description of any unique contextual factors that are noteworthy with regard to ELL practices, such as participation in a consortia of districts for Title III purposes

  • Size and nature of ELL population (including language backgrounds and levels, levels of prior education), significant subpopulations of ELLs and their characteristics

  • Ways in which ELLs are distributed across the district (i.e., do they tend to be grouped in certain areas and schools? or spread throughout the district?)



Instruction of ELLs

  1. What do you see as the primary strengths that ELL students bring to the instructional environment in this district? What do you see as the primary needs of ELL students in this district? How are those needs addressed? (3.2.3.7) (3.2.3.8) (3.2.3.9)



  1. Does the district have specific priorities and principles that guide the work with ELL students in the district? If so, what are they? If not, what does guide this work? (1.1.7) (3.2.3.4) (3.2.3.5)



Probe for:

  • How does the ELL instruction fit into the larger vision for instruction in the district?

  • How do the priorities/goals play out at elementary and secondary levels?

  • What is the role of the native language in instruction for ELLs?



Listen for:

  • Use of state standards (content and ELP)



  1. What are the main instructional programs and supports in place for ELLs in the district? Please describe any specialized classes, programs and/or instructional supports available for ELLs. (This could include bilingual program, ESL classes, sheltered content classes, newcomer classes, individual tutors, bilingual aides, academic counselors, etc.) (1.1.8) (3.2.3.5) (3.2.3.9)



Probe for:

  • Describe each individually: What is the focus? Who provides the instruction? Which students participate? How long have these programs been in place? What is their perceived effectiveness? How do you know if they are effective?

  • Use of native language in classes with ELL students

  • Differences in programming at the elementary and secondary levels

  • For elementary schools, probe for whether there is a designated ELD time during the school day and a description of what takes place during that time.

  • Programs that target specific subpopulations of ELLs (such as newcomers)

  • Specific interventions that may be available to ELLs who are struggling beyond what is in the classroom (such as, Response to Intervention or RtI)



Listen for:

  • Use of state standards (content and ELP)



  1. How are the needs of individual ELLs met in the context of content area classes? In other words, how is the content made accessible for ELLs? Particularly for ELLs who are struggling? (3.2.3.8)

Probe for:

  • Ways in which instruction is individualized for needs of individual ELLs in content classes (3.2.3.8)

  • Use of academic language in instruction

  • Ways in which ELD instruction is incorporated in content area instruction (3.2.3.7)

Listen for:

  • Use of state standards (content and ELP) (1.1.8)



  1. If not previously mentioned: To what extent do you think that teachers are generally familiar with the state ELP standards? How useful do you think the standards are for teachers and how are they used? (1.1.6) (1.1.8) (1.1.9)

Listen for:

  • Extent to which (and ways in which) ELL instruction incorporates/addresses state ELP standards and content standards

    • How teachers learned about the ELP standards and how they access them. (For example, are they sitting in a binder on the shelf? On the internet? Etc.?)

    • Academic achievement





Placement of ELLs into LIEP programs



  1. If not already answered by Assessment Director: How are students initially identified as ELL? And how is re-designation determined? (2.1.3) (3.2.3.5) (3.2.3.6)



Probe for:

  • Measures used, such as state ELP test, district assessments, teacher recommendation, and other measures.



  1. If not already discussed in Question 5 above: With regard to the ELL programs and support you just mentioned, how is a student’s placement into those programs and supports determined? How is exit from them determined? (2.1.3) (3.2.3.5) (3.2.3.6)



Probe for:

  • Number/percentage of students participating in ELL programs and supports

  • How long do ELLs tend stay in various program types? (For example, in the case of a bilingual program, do they stay in it? or transition out to English only?)

  • How do ELL students transition from one program type to another?

  • Perceived effective of placement procedures





  1. Do you face any challenges or constraints in appropriately placing and transitioning ELLs into programs? If so, what issues do you face? How are those issues resolved? (2.1.3) (2.1.2) (2.2.1)



Probe for:

  • Challenges faced at the elementary level, such as transition from bilingual instruction to English only programs, etc.

  • Challenges faced at the secondary level, such as graduation requirements, access to core curriculum and college prep courses, etc.



Curriculum and materials

Next, I’d like to know more about the curriculum, texts and materials used for ELL instruction (in mainstream and ELD classes). (3.2.3.5) (3.2.3.9)

  1. What are the district expectations regarding the use of curriculum, texts and materials in the instruction of English language learners (in mainstream and ELD classes)? Is there a district-wide curriculum for ELD instruction? If so, is it mandated? How was it selected? (1.1.6) (1.1.8) (1.1.9) (3.2.3.5) (3.2.3.9)

Probe for:

  • Factors considered in the selection of curriculum and materials for ELL instruction


Listen for:

  • Use of state standards (content and ELP) in decision making – including ways in which state standards influence text selection, curriculum development, formal and informal assessments, etc.


Assessment

  1. If not already answered by Assessment Director: Aside from the state ELP test, how is the progress of ELL students assessed? Are there additional measures your district has adopted? (2.7.1)

Probe, if necessary:

  • How frequently are these measures administered? Do teachers develop their own measures of English proficiency?

  • Are there instructional tools that are used to measure progress more frequently than the annual state test?

  • What does the district do with the data collected through these assessments?

Listen for:

  • Use of state ELP standards and content standards in development of formal and informal assessments (1.1.6)

  • How data is used to drive instruction



ELLs with special needs

  1. If not already answered by Assessment Director: How are ELLs identified for participation in special education? What services do they receive? (2.1.3)



Title III Accountability

[Note: The questions below will also be asked of the Assessment Director and can be scaled back according to what has been learned from that interview.]

Under NCLB, there has been a new emphasis on accountability for English language learners, and I’m interested to talk with you about how Title III access and accountability have played out in your district.

  1. I’d like to ask you specifically how your ELL students are doing with regard to targets associated with NCLB. First, has your district met AYP for the ELL subgroup? Have there been any specific steps you’ve taken to address your AYP status? (3.2.3.1) (3.2.3.2) (5.1.1)

Listen for:

  • Any changes in ELL practices over time associated with AYP status

  • Have ELL special education students received equal services



  1. Next, let’s talk about your targets for English learners under Title III – that is, the Annual Measurable Achievement Objectives, or AMAOs. Can you tell me about the three AMAO targets your district needs to meet this year? (3.1.1.2) (3.1.1.3) (3.2.1.1)

Probe for:

  • How are each of the AMAO targets defined

  • Do these targets seem appropriate for your district – that is, are they too rigorous, or within reach?

  • Do you feel that stakeholders in your district have adequate understanding of the Title III AMAOs?

Listen for:

  • Awareness and level of understanding of the targets

  • Distinctions between AMAO 1, 2, and 3



  1. How is your district doing in terms of meeting AMAO targets? Which targets have you met or missed, and for how long? (5.1.1)



  1. If targets have been missed: Has your district experienced any consequences as a result of missing AMAO targets? If so, what are they? (5.1.1)

Resources

  1. What resources can you draw on – whether financial resources, colleagues with specialized expertise, regional assistance centers, etc. – to provide support for Title III priorities? (4.1.2) (4.3)


Probe for:

  • Kinds of resources district has available for this work – both Title III and others – and how they use these resources.  (Think of resources broadly)

  • How district uses Title III money

  • Any challenges encountered with respect to “supplement not supplant" provisions in Title I or Title III?



Teacher Quality

Next, I’d like to know more about the teachers in your district, the extent to which you believe they are prepared to teach ELLs, the ways in which they are distributed in schools across the district, and how your district goes about recruiting and retaining the teachers you need.

  1. First, to what extent do you feel teachers have the skills and knowledge needed to successfully teach the ELL students in this district? Where are they strong? In what areas do they need to improve? (4.5.4) (4.5.5)



Probe for:

  • Extent to which district has a pool of qualified teachers

  • Any differences at the elementary and secondary levels

  • Any differences between capacity of content teachers and ELL teachers

  • Any changes in teacher capacity over time



  1. How do you ensure that ELL students are taught by teachers who are knowledgeable about both content and ESL instruction? How are those teachers who are best able to address the needs of ELL students distributed across schools in the district? (4.5.2) (4.5.4)



Probe for:

  • Distribution of HQTs in classes with the most ELLs

  • How teachers are assigned to classes with ELL students (and to LIEP programs)

  • Are there incentives (or disincentives) for qualified teachers to teach certain classes?

  • Any policies or practices for assigning teachers to schools (particularly schools with large numbers of ELLs)

  • Extent to which stronger teachers are in schools with largest numbers of ELLs



  1. How does your district go about recruiting and retaining teachers who are knowledgeable in the content area and English language instruction? (if it is possible to talk with an HR person in the district, these questions should be asked of that person in depth) (4.5.2)



Probe for:

  • Extent to which the district pays attention to ELL qualifications in recruiting teachers

  • Challenges associated with recruiting teachers with the needed skills and how addressed



  1. How do teachers in your district demonstrate their fluency in English?   Do any teachers in your district provide instruction in languages other than English?  If so, how do teachers demonstrate fluency in languages other than English? (4.5.1)

Listen for:

  • Awareness of Title III or state teacher fluency requirements        

  • Challenges associated with implementing teacher fluency requirements



Professional learning



  1. Now I’d like to ask you about the professional learning that teachers engage in. (If we are able to talk to a PD person in the district, these questions should be asked of that person in depth.) What are the main types of professional learning available for teachers in your district, specifically focused on issues related to ELL students? What is the district’s role in providing these opportunities? (4.5.7)

Probe for:

  • A description of PL related to ELLs: What is the focus (and how it relates to the priorities expressed earlier)? Who provides? Intensity? Perceived effectiveness? What has been the impact on instruction? What has been the impact on academic achievement? Etc.?

  • Who determines the content and which teachers should attend

  • Differences at the elementary and secondary levels

  • Differences in what is provided for ELD teachers vs. content area teachers

  • Is any PD mandatory? Are there incentives to participate in PD?


Listen for:

  • Use of state standards (1.1.9)

  • Any assistance that is provided by state (4.3.3)



  1. What kinds of professional learning do principals and administrators engage in regarding ELL instruction in the content areas? (1.1.9) (4.3.3)

Listen for:

  • Support provided by district, any assistance provided by state

  • Use of state standards



  1. As a district leader, where do you turn for assistance on ELL issues or to build your own professional knowledge in this area? (4.3) (4.1.4)





Parent support

  1. Are there any district-wide programs or supports targeted for parents of ELL students? Please describe. (3.2.3.10)



Probe for:

  • Which parents are targeted for these programs, who provides the support, the focus, etc.

    • Elementary

    • Secondary

  • To what extent do ELL parents access them? In what ways do they benefit parents of ELLs?

  • If there are other programs or supports for parents, how do ELL parents benefit from those?



Concluding Questions

  1. Taking into account everything we’ve discussed, what are your thoughts on the progress of ELL students in your district?



  1. Wrapping up, could you describe any challenges associated with Title III implementation that we haven’t already discussed? (4.1, 4.2, 4.3, 4.5)



Also probe for:

  • Unique contextual issues

  • Capacity challenges/strengths (LEA level)

  • Instructional challenges/strengths



  1. Are there any highlights associated with the instruction of ELL students that you did not previously mention? Anything I should have asked you, but didn’t? (4.1, 4.2, 4.3, 4.5)


E-0


File Typeapplication/msword
File TitleEVALUATION OF STATE AND LOCAL
AuthorInformation Technology Group
Last Modified By#Administrator
File Modified2009-10-23
File Created2009-10-23
© 2024 OMB.report | Privacy Policy