Response to Public Comments

Both the Alliance for Excellent Education and Knowledge Works provided insightful comments and thoughtful questions on the National High School Reform Study. Both suggested including survey items about the extent to which high schools are offering personalized learning opportunities for students. Many of the suggested additions from the two organizations overlap. The table below presents the organizations’ comments in the first two columns, and the third column presents the Department’s responses, specifically where related items may be found in the survey, where time or budget constraints prevent their inclusion, and what may be added, given temporal and financial constraints. Knowledge Works Suggested Construct / Item	Alliance for Excellent Education Suggested Construct / Item	Corresponding Question / Response
Partnerships with local business leaders to identify and analyze local labor market trends		This type of partnership is a key element in identifying the relevance of CTE course offerings in schools, but the National High School Reform Study (NHSRS) is focused on strategies to prevent at-risk students from dropping out of high school.
Aligned college-and-career ready standards to transferable skills or competencies		The NHSRS focuses on the implementation of strategies to engage at-risk students in school and prevent such students from dropping out of high school. Alignment of standards to transferable skills or competencies, while critically important to engaging these students, falls slightly outside the study’s focus. In every study, boundaries must be drawn, often regretfully, but time and resource constraints compel tough choices.
Provided all students with clear learning outcomes to enhance understanding of what is expected for success		This is difficult to capture in a survey to school administrators, since the definition of clear should depend upon students’ perceptions, and students are not surveyed in this study.
Repurpose professional development resources to more effectively support educators in the development of transferable skills or competencies and learning objectives aligned to college- and career-ready standards		This is difficult to capture in a survey to school administrators, since the definition of effective should be explored through the analysis of empirical evidence and depends upon teachers’ perceptions of the professional development. Teachers are not surveyed in this study.
Development and effective implementation of personalized learning plans based on students’ current knowledge and skills, college-and-career ready standards and competencies, academic interests, optimal learning approaches Personalized learning plans?	Does the school provide a personalized sequence of content and skill development informed by the student’s academic interests that is designed to enable the student to achieve his or her individual goals and ensure he or she can graduate on time and ready for college and a career?	This topic seems valuable and critical to include. Both organizations offer the overarching suggestion that we should consider an emphasis on personalized learning. Thus, we added two new questions on this topic: 102, 103. The new questions address these specific comments but also reflect more general concerns of the policy research community and of the field. Questions about the effective implementation of personalized learning plans fall outside the scope of a school administrator survey. We must prioritize gathering consistent information across the strategies and reserve such compelling follow-up questions for subsequent studies.
Development of strategies to differentiate instruction based on each student’s academic needs	Does the school use data to inform and differentiate instruction to meet individual students’ academic needs?	Including such items in a survey of school administrators invites social desirability bias; teachers supposedly should differentiate instruction, but few teachers actually do so appropriately and effectively. School personnel likely will claim that they use data to differentiate instruction but in reality, do they implement this differentiated instruction? Thus inclusion of such a question increases the risk of overestimation due to method of data collection.
Improve teachers’ and principals’ practices and effectiveness by using feedback provided by evaluation systems		These two approaches require a more thoughtful, in-depth method of data collection than a survey of school administrators. Teachers should be asked if they find feedback from evaluation systems useful and how. Collaboration with community partners should be explored more comprehensively and deeply – how often, in what manner, to what end. Again, prioritizing what constructs to include in the limited-scope, limited-time NHSRS meant we needed to exclude some constructs, including these two thoughtful suggestions.
Change practices, policies, and rules to foster collaboration for teachers and community partners
Develop plan to expand learning options through partnerships with community organizations, including opportunities for students to earn credits that count toward the state’s high school graduation requirements	To what extent, if any, were community partnerships developed to enhance increased learning time?	Of the several technical experts in the field and in the research community whom we consulted to develop the NHSRS survey instrument, two suggested including questions on school partnerships with community organizations and businesses, because such partnerships represent an important component in student access to social services. Thus, we have included items on this topic (see Questions 23 through 28). We asked those experts if asking additional questions about how these partnerships related to learning opportunities warranted sacrificing other items. They suggested that the other items merited inclusion more.
Do students select enrollment in interest-based pathway?		The NHSRS survey instrument inquires about smaller learning communities, which may be based on or around student interests. However, we do not ask about student selection of pathways.
Students have opportunity to demonstrate mastery of standards in multiple, comparable ways based on their academic interests and preferred learning approaches		This suggestion taps into a valuable approach for students who may not thrive in traditional assessments to demonstrate their skills. As much as this is important and useful, it did not make the cut for inclusion in the NHSRS study.
Earn credit based on demonstrated mastery of standards and competencies, not amount of time spent on topic	Do students have opportunities for competency-based advancement? If so, what are they? (1.4) To what extent, if any, is technology being utilized to support the implementation of competency-based systems of advancement? (3.3)	This topic seems valuable and critical to include. Both organizations suggest that we should consider an emphasis on competency-based advancement. Thus we added two new questions on this topic: 104, 105. The new questions address not only these specific comments but also general concerns of the policy research community and of the field.
Educators held accountable for ensuring all students on track to graduate on time and be proficient in college and career-ready standards and competencies		It is unclear how this reflects individualized student learning pace. This seems to reflect accountability practices in the district and school, not differential learning paces. The survey section on use of early warning systems is a way to learn about the extent to which educators are tracking data to identify issues that can prevent students from graduating.
Students who master college- and career-ready standards faster have access to more rigorous learning options, e.g. dual enrollment courses or participation in an early college high school program	(1) How many Advanced Placement courses are offered at the high school? (2) How many dual-enrollment courses are offered at the high school? (3) Does the high school partner with an institution of higher education to implement an early-college high school? (4) What are the rates of enrollment, completion, and where applicable, credits earned, by subgroup, for the following courses: a. Advanced Placement b. International Baccalaureate c. Early college d. Dual enrollment	Question 53 asks what form of advanced coursework is offered to students, including Advanced Placement, International Baccalaureate, early-college programs, and dual enrollment. Question 45 does ask approximately what percentage of students is offered accelerated academic programs. In addition, other ED-sponsored studies capture information about AP courses, which allows NHSRS to focus on other elements and possibly link to these data when constructing the public release dataset. Due to time and space constraints, we do not ask about rates of completion and credits earned by subgroup. This in-depth look may be best reserved for follow-up studies or other data sources.
Organize the school schedule, calendar, or instructional time differently to maximize student learning gains and personalize the learning environment	Has the high school implemented increased learning time? If so, how? Is the high school using flexible scheduling and/or block scheduling to support the implementation of improvement strategies?	Questions 94 through 99 ask about school schedule, school calendar, instructional time, and flexible scheduling to support the implementation of improvement strategies.
Maximize the time that educators have to collaborate on professional development activities	To what extent, if any, are teachers provided with professional development to enhance their ability to personalize instruction?	Please see earlier comment about the complexity of professional development questions in this survey milieu.
Use of ongoing, embedded formative assessment to evaluate student performance in real-time Results of formative assessments used to personalize daily instruction and learning objectives for students	To what extent, if any, are teachers provided with professional development to enhance their ability to develop and utilize data from multiple forms of assessment to personalize instruction, such as interim, formative, and performance assessments?	Use of formative assessments to tailor instruction is good instructional practice for all students, but does not necessarily represent a dropout prevention strategy, which is the focus of this survey. In addition, requiring teachers to use formative assessments does not ensure that they actually utilize the data or know how to tailor instruction to meet the needs of all students
Support for blended learning opportunities that combine online and teacher-based instruction		Questions 37, 38, 48, 49, 61, and 68 all ask about blended learning approaches.
Educator access to technology-based content, resources, and tools that personalize instruction and training to use them effectively	To what extent, if any, is training provided to educators and students on the effective use of technology? (3.4)	These questions all tap into the use and effectiveness of technology by teachers for improving their instructional effectiveness and by students for enhancing their educational experience, applying their lessons to the real world, and connecting to the community. Such items reflect the dire need for funding to support a technology-focused survey. ED’s Office of Educational Technology no doubt appreciates the value placed on technology as an approach to engage at-risk students to stay in high school. Yet, this falls outside the scope of the NHSRS.
Student access to digital tools and software used by educators to deliver high-quality curricula, instruction, assessments
Digital tools to enhance curriculum by connecting students with local community and business partners
Sufficient school facilities to maximize technology access and effectiveness
Interoperable data systems that include human resources data, student information data, budget data, and instructional improvement system data	To what extent, if any, has the school undertaken a comprehensive planning process in order to determine how technology can be most effectively utilized to enhance teaching and learning?	Unfortunately, the NHSRS does not seek in-depth information about how schools’ multiple data systems relate to each other or how these data systems may be connected to a more comprehensive planning process about harnessing technology to enhance teaching and learning. The NHSRS does seek information about the interoperability and coordination of strategies, specifically school services provided to address the needs of students at risk of educational failure may be coordinated with one or more other strategies, or they may be isolated. Coordinated services are designed to work together in an integrated system to match students to appropriate resources. Isolated services operate independently, with little communication with other programs or services.
Using information technology systems that allow parents and students to export their information in an open data format
Ensuring the use of interoperable data systems that increase alignment between in-school and out-of-school learning experiences
	Does the school implement strategies to develop caring, consistent relationships between students and adults that communicate high expectations for student learning and behavior?	Asking the school administrator to comment on or characterize strategies to develop caring relationships that communicate high expectations sounds promising in theory, but in practice may elicit social desirability bias. In addition, are there sufficient numbers of schools across the country promoting such strategies to warrant inclusion in an already tightly-constrained survey? Instead, the NHSRS includes items on developing a positive school climate (specifically Questions 31 through 33).
	Does the school coordinate or provide students with integrated support services to meet the non-academic needs that may serve as barriers to students’ academic success?	The NHSRS does seek information in Question 102 about the interoperability and coordination of strategies, specifically school services provided to address the needs of students at risk of educational failure may be coordinated with one or more other strategies, or they may be isolated. Non-academic needs are covered by Questions 23 through 30, and Question 102 should tap into whether the school coordinates academic and non-academic support services. Coordinated services are designed to work together in an integrated system to match students to appropriate resources. Isolated services operate independently, with little communication with other programs or services.
	To what extent, if any, are teachers provided with professional development to enhance their ability to develop competencies in students including higher-order analytical skills, critical thinking, evaluation of evidence, problem-solving, and communication skills?	These questions about schools’ providing opportunities for professional development and teacher collaboration may be more useful -- and elicit more accurate information -- if they were instead asked of teachers about their participation in professional development and professional communities. The provision of such opportunities is helpful, only to the extent that professional development is attended, is appropriate, is research-based and beneficial, is conducive to faithful implementation in classrooms by teachers, and is monitored and/or guided by informed instructional leadership. Given these conditions for effective professional development – and for active engagement in professional learning communities – it is more valuable to reserve space on a school administrator survey for items that are not better asked of teachers.
	To what extent, if any, are educators provided with professional development that is differentiated based on the demonstrated needs of the individual educator?
	To what extent, if any, are educators provided with opportunities to participate in professional learning communities?
	To what extent, if any, do teachers have opportunities to co-teach or participate in cross-curricular projects?
	To what extent, if any, is technology being utilized to support over-aged and under-credited students?	We consider such students at-risk of dropping out of high school, and based on this consideration, the entire survey in essence asks about supports for these students.
	Does the high school provide multiple courses that integrate core academic courses and career and technical education (CTE) courses? Does the high school offer multiple college and career pathways that include a comprehensive array of core academic courses, CTE, work-based learning, and student support services?	Integration of academic courses and CTE courses were rated highly by the NHSRS experts when this topic was included in the original question list and is deemed by the What Works Clearinghouse as a promising strategy to engage at-risk youth in school. Thus we added two new questions about this issue: 106, 107
	Does the high school have a partnership with at least one institution of higher education to provide a. a data feedback loop to reduce the need for remediation b. dual-enrollment or early college programs c. high school to postsecondary bridge programs	The NHSRS dataset may be able to merge EDFacts data on School Improvement Grants, specifically on: dual enrollment, advance coursework, increased learning time. On the NHSRS, Question 53 asks what form of advanced coursework is offered to students, including Advanced Placement, International Baccalaureate, early-college programs, and dual enrollment.
	Does the high school provide a. counseling and support for college planning, including the application and financial aid process b. strategies for communicating high expectations for students c. middle to high school bridge programs	Several questions on the NHSRS seek information about counseling and support and about transition programs. Question 24 inquires about supports and social services available through school referrals, including guidance on academic and career needs. We have not included questions on college planning or strategies for communicating high expectations for students. Resources to facilitate college planning and financial aid are covered in-depth in the High School Longitudinal Study of 2009 through the National Center for Education Statistics. We could include a question on communicating high expectations for students if such an item is considered a sufficiently high priority for the survey. The question as it stands now needs to be revised before inclusion on the survey. Communicated by whom? For all students? What defines high expectations? The NHSRS team did discuss including this topic but could not develop questions that would plausibly avoid strong social desirability bias, particularly from school administrators. Questions 89 through 93 on the NHSRS instrument focus on middle to high school bridge programs.
	Is the high school utilizing an early warning indicator system to identify students at risk of not graduating? If so, does this include data on: a. course performance b. attendance c. behavior/discipline Are these data at the following levels to assess staff performance and target professional development: a. classroom b. grade c. school	Questions 81 through 88 focus on early warning indicator systems used by high schools. Question 84 solicits information on what data points are included in these early warning systems, specifically course performance, attendance, and behavior/discipline. Question 86 inquires about who at various levels of the school system can access data from early warning systems. Due to resource constraints, the instrument does not include questions about the use of these data in staff evaluations or professional development.
	To what extent, if any, is the high school implementing specific efforts at the ninth-grade level to support students with a successful transition into high school?	Questions 89 through 93 on the NHSRS instrument focus on middle to high school transition programs. Question 90 includes several specific items on transition strategies targeted to ninth-graders.
	What strategies are being implemented to provide academic and non-academic support?	Questions 74 through 80 and Questions 23 through 30 tap into non-academic supports and resources, such as mental health services, immigrant services, and social services. Questions 54 through 61 ask about academic support classes. Questions 62 through 73 inquire about academic tutoring, a specific form of academic support.
	Is the high school implementing a. a response to intervention; b. restorative justice programs; c. multi-tiered systems of support; and/or d. social-emotional curriculum.	This topic falls outside the scope of the NHSRS, though such strategies seem potentially valuable and are meriting much attention from other offices at ED currently. The section on positive school climate programs addresses elements in restorative justice programs. The definition for the section on student support teams refers to a multi-tiered system of supports that Response-To-Intervention (RTI) promotes. We originally included RTI within the (e.g.) but removed it in response to feedback. Thus we expect other policy and research efforts to include data collection on these topics.
	Is the high school utilizing student and staff surveys to assess school climate, working conditions, and their impact on student-teacher relationships?	This is a very interesting question. What is the estimated predominance of this approach in high schools? The NHSRS is attempting to solicit information on the most prominent reform strategies to maximize variability and analytic utility.
	Does the school have a zero-tolerance discipline policy? If so, please explain.	The inclusion of the item to elicit an explanation of if/how a zero- tolerance discipline policy is enforced by schools suggests that these questions would be better suited for a qualitative study or a longer survey that allows for open response. The survey section on positive school climate programs was included to address attempts to move away from zero-tolerance disciplinary policies that can increase the chances of students dropping out of school.
	Does the high school have a positive school climate policy agenda? Did the high school reexamine the applicable code of conduct through, for example, the use of student, parent/caregiver, and staff surveys, to assess current policies.	Questions 31 through 33 of the survey inquire about the high school’s positive school climate policy. These questions were vetted by practicing school administrators and technical experts in research and in school districts.
	Over the past five years, have there been any changes in the number of support staff, such as guidance counselors, social workers, and appropriately trained paraprofessionals and assistants available to support student achievement? Please explain.	This question, while meritorious substantively, presents challenges to the timeframe for the questionnaire. It is best asked as an open-response question, and the survey already strains against time limits. In addition, the study attempts to set a baseline of strategies, generally, and capturing change in the last five years expands the scope past which this one study can reach.
	To what extent, if any, are teachers and administrators provided with professional development regarding culturally responsive practice? With professional development regarding social and emotional learning competencies?	In prioritizing the constructs to include on the NHSRS instrument, professional development was deemed important but less so than other strategies, especially considering methodological concerns and complexities about professional development questions. Questions about professional development should include not only whether such professional development is offered but also dosage, format, perceived effectiveness, alignment with curriculum, etc. One question begets several others, squeezing out other more important questions to the NHSRS.
	Is the high school incorporating strategies for assessing staff attitudes and providing access to training and support that can reduce deficit-oriented approaches (e.g., focusing on what students can do rather than what they cannot do) that often result in low student expectations? If so, explain.	The inclusion of the item to elicit an explanation of how such attitude assessments are utilized by schools suggests that these questions would be better suited for a qualitative study or a longer survey that allows for open response.
	ED’s High School Reform Study provides an opportunity to establish a baseline on Title I funding for high schools in order to determine the impact of community eligibility on high schools.	Examining Title I funding for these high schools expands the scope of work beyond the NHSRS’ reach. Understanding Title I funding deserves its own study, not a supplement to a study about specific high school reform strategies. Because NHSRS will have data on the participating high schools, including their identification codes, we can explore linking Title I data from EDFacts to the NHSRS dataset.

The final suggestion from the Alliance for Excellent Education calls for making the High School Reform Study’s dataset available, along with key academic outcome data and demographic data. Indeed, from the very start of the NHSRS, we have intended to release the data along with such demographic data, even merging EDFacts data wherever possible. These EDFacts sources include data on advanced coursework, dual enrollment opportunities, school calendar and schedule, student membership, graduation rates, and priority or focus school status. These data incorporate much of what the Alliance specifically suggested as critical information to collect: (1) four-year adjusted cohort graduation rates; (2) extended-year adjusted cohort graduation rates; (3) achievement on state assessments; (4) the school’s letter grade or other ranking, if the high school is in a state that uses an accountability system that incorporates letter grades, a “star” rating system, or other form of an index to report on school performance; and (5) status as a “priority” or “focus” school under ESEA flexibility.

We hope both organizations understand that we had to make tough choices given the length of the survey and that we relied on the feedback from our national experts and practitioners to identify the most salient dropout prevention strategies to include. There is always a tradeoff in achieving the desired response rate and including more items.  We too would like to find out more information on some of the topics that both organizations raised, but we hope that the NHSRS survey will provide a baseline regarding the array of strategies that high schools are currently using to prevent at-risk students from dropping out of high school—a resource that is currently unavailable. Perhaps future surveys can utilize this baseline information to learn more about particular strategies or about some of the strategies we were not able to include. We appreciate the time and effort both organizations took to provide feedback and hope they will utilize the information that the survey generates. We welcome an opportunity to discuss any of these points further.

9