1875-NEW TSLP OMB Package_Supporting Statement Part A

This document requests forms clearance approval from the Office of Management and Budget (OMB) for the collection of data for the Implementation Study of the Turnaround School Leaders Program (TSLP). In particular, we are requesting approval for (1) project coordinator, project training provider, and school district partner surveys; (2) interviews and focus groups with project staff and aspiring and current leaders; and (3) collection of extant data, including early outcomes data and other relevant project-specific data. This is an initial OMB submission for contract ED-PEP-11-O-0088/TO28 to Westat and its research partner, Policy Studies Associates.

A1. Circumstances Necessitating the Data Collections

Overview of the Program

Leaders who are trained and committed to lead turnaround efforts in the nation’s lowest-performing schools play an important role in improving student outcomes.¹ Yet, interviews with external partners engaged in leadership pipeline development, as well as monitoring of performance of state education agencies (SEAs) and local education agencies (LEAs) under the School Improvement Grant (SIG) program,² indicate that many LEAs do not have the capacity or resources to recruit or develop school leaders who are able to undertake successful turnaround efforts. The Department launched the TSLP to support school turnaround by investing in partnerships between LEAs and others (including SEAs, institutions of higher education, or public or private nonprofit or for-profit organizations) to build the quality and supply of leaders with the skills and knowledge to turn around low-performing schools that participate in or are eligible for the SIG program.

Overview of the Study

Over recent years, the Department has conducted important studies and reviews of school turnaround. While most of these have at least touched on the question of leadership, none has focused exclusively on the role of school leadership generally. This study stands at the nexus between studies of school turnaround and school leadership. This study seeks to generate information to help policymakers and practitioners who struggle with the challenges of developing leaders to turn around low-performing schools and to add to the field’s general body of knowledge about turnaround leadership. The study will include case studies of seven Cohort 1 TSLP projects, including each grantee’s partners; an analysis of extant data, including grantee applications, early outcomes data, and other relevant project-specific data; and surveys of all local coordinators of Cohort 1 TSLP projects, one representative from each of the training partner organizations of each Cohort 1 TSLP project, and school district partners.

Study Questions

The study questions (Exhibit 1) reflect the appropriate focus for an implementation study of a new federal program. In short, the questions ask for a description of how TSLP projects develop the leadership pipelines for turnaround schools (Study Question 1), the roles project partners play (Study Question 2), the ways in which grantees have adapted their project plans in response to changing circumstances (Study Question 3), and what early outcome data show about TSLP projects (Study Question 4). Exhibit 1 below lists the detailed study questions that will guide the development of data collection instruments and the organization of the final report.

Exhibit 1. Detailed Study Questions

1. How do TSLP projects identify, develop, and support leaders for low-performing schools?

a. What strategies to identify, develop, and support leaders have grantees adopted? How do grantees recruit and select candidates for participation? At the time of selection, to what extent are participants currently serving or aspiring to become leaders of low-performing schools?

b. What characteristics are projects seeking to develop in current and aspiring school leaders?

c. What types of preparation or professional development are grantees offering through their projects?

d. How are mentoring and coaching activities structured?

e. What incentives are being used to help retain school leaders in placement schools?

2. What role do project partners play in implementing projects and helping grantees to achieve project goals?

a. How is responsibility for achieving project goals distributed across partners?

b. How are the roles that LEAs play in the partnership distinct from non-LEA roles?

c. Who are the key personnel in grantee and partner organizations who are involved in implementing the TSLP-funded project? What roles do they play in the project’s administration?

d. How closely do grantees work with their project partners?

e. What challenges and success did grantees experience with developing and maintaining partnerships?

3. How have grantees modified projects to adapt to challenges or meet the demands of changing circumstances?

a. What challenges do grantees face in implementing their original plans? What are common reasons that grantees give for modifying the project plan?

b. How do grantees make decisions about adjustments to the project? How are data and research evidence incorporated into these decisions?

c. How have grantee theories of action evolved?

d. How have the turnaround school leaders grants contributed to developing a sustainable, long-term pipeline of effective leaders for turnaround schools?

4. How are grantees measuring the success of their TSLP projects, and do early outcome data show promising results?

a. Are grantees using any locally developed metrics, in addition to the Government Performance and Results Act (GPRA) indicators, to measure their success? What challenges (e.g., logistical, data quality) do grantees face in collecting data on their projects?

b. Who is responsible for collecting and analyzing data on performance indicators?

c. Over what period of time will grantees collect and analyze data to measure the project’s success?

d. What are the early outcomes for TSLP projects, based on both GPRA measures and any locally developed metrics?

5. How have the turnaround school leaders grants contributed to developing a sustainable, long-term pipeline of effective leaders for turnaround schools?

a. What methods of recruiting, selecting, training, placing, and supporting turnaround leaders that were developed or implemented by the grantees will be sustained after the end of grant funding?

b. What factors contribute to sustaining innovations in preparing and placing turnaround leaders?

c. Have grantee districts developed long-term pipelines for leaders for turnaround schools, and how has the grant supported that development?

A2. Purposes and Uses of the Data

To better understand the issues mentioned in A1, this study will examine the implementation of the TSLP and provide information on how grantees (1) identify, develop, and support leaders and aspiring leaders of low-performing schools; (2) adjust their project plans; (3) use data to examine progress; and (4) work with project partners to meet goals. The ultimate purpose of the study is to glean specific lessons learned for turnaround leadership development (for the field), program improvement (for program staff), and program design (for policymakers).

A3. Use of Technology to Reduce Burden

The data collection plan has been designed to maximize efficiency, accuracy, and convenience for respondents and to minimize their burden. The study will use technology to reduce burden by using a web-based survey to collect data from survey respondents. Burden will be further reduced with the use of skip patterns, where appropriate. Notification of participation and login credentials will be sent via email, whenever possible. As alternatives, respondents will be offered the opportunity to complete the survey through telephone follow-up calls or use of a hard-copy version. All of these formats allow respondents to complete the survey at their convenience and accommodate individual preferences.

A4. Efforts to Identify Duplication

The data collection plan uses existing data to the extent possible, e.g., relying on application documents and APR data. Information that is requested through the study’s surveys, interviews, and focus groups is not readily available elsewhere.

A5. Methods to Minimize Burden on Small Entities

The burden will be reduced on the small business entities (e.g., training partners) by using a short web-based survey. It will take approximately 30 minutes for a respondent to complete the web-based survey.

A6. Consequences of Not Collecting the Data

The data collection plan described in this submission is necessary for the Department to conduct an in-depth study of methods grantees are using to train leaders for turnaround schools. While several studies have suggested promising ways to train turnaround leaders, the Department and practitioners still need specific, practice-based information on how to design and implement programs. Both surveys and qualitative data collection (interviews with program staff and focus groups) are important to gain an understanding of the context and the way the program partners have worked together, as contrasted with the prevalent model of leadership preparation in which higher education agencies provide general training unconnected to local practice.

Not collecting the data planned for this study would reduce the benefit of the federal investment in these programs by preventing dissemination of important lessons learned. Moreover, one important question the study will address is whether the activities undertaken while the funding was available will continue after those funds disappear. This is especially important because although the SIG (and consequently TSLP) was eliminated under the Every Student Succeeds Act, low-performing schools will still need turnaround leaders, and states and districts will still need to understand how to establish effective pipelines to recruit, train, and retain these leaders.

A7. Special Circumstances

None of the special circumstances listed applies to this data collection.

A8. Federal Register Comments and Persons Consulted Outside the Agency

Federal Register Announcement. The 60 day Federal Register notice was published in Volume 81, June 3, 2016, page 35756. To date no public comments have been received.

Consultation outside the Agency. The Department created a technical working group (TWG) to provide expert guidance on the study. The TWG will meet at two key points during the course of the study. During the first convening, which was held in April 2016, the TWG provided feedback on the study design and data collection instruments and protocols. The purpose of the second convening of the TWG will be to provide feedback on a draft of the final report. The TWG members represent a variety of perspectives on and experience with leadership in turnaround schools (Exhibit 2). The TWG consists of five members: a current district administrator who works with turnaround leaders; an academic who studies school leadership; a current principal ambassador fellow with expertise in turnaround school leadership; a current external assistance provider to turnaround schools, districts, and states; and a current state board of education member, who also has expertise in rural education.

Exhibit 2: Technical Working Group Members and Affiliations

Name	Affiliation
Nancy Brightwell	Beacon Shepherd & Community Superintendent, Charlotte-Mecklenburg Schools
Charles Payne	Professor, University of Chicago
Chris Pearson	Campus Principal Ambassador Fellow, U.S. Department of Education
Sam Redding	Executive Director, Academic Development Institute
William White	Member, West Virginia Board of Education

A9. Payments or Gifts

No payment or gifts will be provided to respondents.

A10. Assurances of Confidentiality

Researchers will adhere to federal rules regarding the protection of human subjects in research. The research team has a duty to protect all information, but particularly sensitive information. The following provisions will apply on this project:

As part of the data collection training, all members of the research team will be trained on data confidentiality. Specifically, researchers will be trained on how to store data without individual names and how to discuss survey and interview data only within a team context for analysis purposes.
As part of obtaining consent for surveys and interviews, each respondent will be apprised that his/her participation in the project is voluntary, that he/she may cease participation at any time during the survey or interview, and that “responses to this data collection will be used only for research purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific individual. We will not provide information that identifies you to anyone outside the study team, except as required by law.” In interviews, researchers will provide this information orally as well as in writing in the consent form. All respondents will be asked to sign a consent form (see Appendices L and M).
The voluntary nature of respondent participation, the confidentiality provisions, and consent forms are subject to and overseen by Westat’s respective Institutional Review Board for human subjects research.
All electronic data will be stored on secure servers. Access to the server is password protected, with required changes at regular intervals and strong password elements. Each user’s access is limited and determined by the network administrator. All surveys and interviews will be assigned a randomly generated unique ID. No data will contain respondent names.
Westat standard practice is to shred documents and destroy electronic data once the data are no longer required, typically within three years to allow for any questions that may arise after the final report’s publication.

The final report will name grantees and grantees’ partner organizations. It will not identify individual survey respondents or interview or focus group participants by name. Given the small number of transcripts, the differing roles of most interviewees, and the content of the interviews, it is likely that an informed reader will have little difficulty in identifying, for example, the project coordinator or district superintendent.

The study team will provide transcripts of interviews and focus groups to the Department personnel who are responsible for overseeing this study; however, all names and other explicit identifiers will have been removed prior to delivery.

A11. Justification of Sensitive Questions

The survey, interview and focus group protocols do not include sensitive questions.

A12. Estimates of Hour Burden

The data collection plan has been designed to maximize efficiency, accuracy, and convenience for respondents and to minimize their burden. The study calls for surveys of all (12) Cohort 1 project coordinators, representatives from grantees’ training partner organizations (under the circumstances that the project coordinator is not part of the training organization) and representatives from school district partners. The study also includes site visits to seven sites where interviews will be conducted with the project coordinator, three LEA staff members (the assistant superintendent, the principal’s supervisor, and turnaround school professional staff), and two representatives from each partner organization. In addition, two separate focus groups will be held during each of the visits (one with aspiring turnaround leaders and one with current turnaround leaders). Each focus group will include five people.

Exhibit 3 shows the estimated burden for each of the data sources. Specific assumptions follow:

The project coordinator and district partner surveys will take approximately one hour to complete and the training partner survey will take approximately 30 minutes to complete.
The grantee project coordinator interview will take approximately one and a half hours to complete.
Interviews with key informants will take one hour each.
The focus group aspiring leaders and interviews with current and placed leaders should take one hour each.
Collection of extant data will take one hour each.

Exhibit 3. Annual burden estimates, by data source

Data source	Respondents	Estimated number of responses	Estimated annual burden per response (in hours)	Total estimated annual burden (in hours)	Average hourly wage rate	Total estimated annual burden (in dollars)^b
Surveys	Project coordinator	12	1	12	$48.23	$578.76
Surveys	One representative from each training partner organization	13	0.5	6.5	$48.23	$313.50
Surveys	One representative from each school district partner	32	1	32	$48.23	$1,543.36
Interviews	Project coordinator	7	1.5	10.5	$48.23	$506.42
Interviews	Three staff (assistant superintendent, principal’s supervisor, turnaround school professional) from each school district partner	30	1	30	$74.80	$2,244.00
Interviews	Other partners	9	1	9	$54.03	$486.27
Focus groups	Aspiring leaders (5 per site)	35	1	35	$46.68	$1,633.80
Focus groups	Placed leaders (5 per site)	35	1	35	$46.68	$1,633.80
Extant data collection	Cohort 1 grantee project coordinator	12	1	12	$48.23	$578.76
Total		185		182		$9,518.67
Annualized Basis		62		61		$3,172.89

The estimated average hourly burden is based on the personnel national average salaries from the Center for Benefit-Cost Studies of Education (CBCSE) Database of Educational Resource Prices for Project Coordinators ($90,670), Assistant Superintendents ($139,463), Principals and Turnaround professionals ($90,954) and partner representatives ($101,578), and Aspiring leaders and current leaders ($87,760).

A13. Estimate of Cost Burden to Respondents

There are no additional respondent costs associated with this data collection other than the hour burden estimated in item A12.

A14. Estimate of Annual Cost to the Federal Government

The estimated annualized cost of the study to the federal government is $245,742. This estimate is based on the total contract cost of $614,355, amortized over a 30-month performance period. It includes costs already invoiced, plus budgeted future costs that will be charged to the government for the study redesign, data collection, analysis, and reporting.

A15. Program Changes or Adjustments

Not applicable. This is a new request.

A16. Plans for Tabulation and Publications of Results

The core analyses of this study will focus on what can be learned across TSLP grantees. The survey data from all 12 Cohort 1 grantees will provide an overall picture of the activities and outcomes funded by the TSLP as well as an important reference point for the case study data. The study will also include cross-case analyses of the site visit data.

The study team will develop a data analysis roadmap by isolating which combination of data sources will be able to address each study question. We will upload all de-identified data from all sources (including transcripts, survey responses, and extant data) into qualitative research software (QRS) and then code all data using the conceptual constructs as a guide. By analyzing data in a single QRS database, we will be able to efficiently integrate and synthesize information across sources, as all information will be tagged with the same set of codes. It will also facilitate assessing inter-rater reliability across coders. As a result, when investigating and analyzing the data associated with a specific construct, we will be able to immediately cull information from all relevant sources. This will allow us to ensure that we are effectively integrating findings by providing immediately available evidence from surveys, interviews, focus groups, and extant data in one single report at the conclusion of the study. We will document our work carefully to facilitate quality control and review.

The study team will rely primarily on descriptive statistics in our analyses of the survey and outcome data (cross-tabulations to be specific) to provide an overview of the data and to explore relationships among variables.

The study’s final report will focus on telling the key stories across grantees, using survey, case study, and extant data. We will organize the report around the Study Questions.

The study will occur in three phases: study design (including external review), data collection, and data analysis and reporting. The study design phase includes developing the data collection strategy and data collection instruments and occurs from fall 2015 through summer 2016. External review includes TWG, OMB, and IRB review of the study design and will occur during spring and summer 2016. Data collection will begin in fall 2016 end in spring 2017 and data analysis and reporting will take place from January to March 2018. Exhibit 4 provides an overview of the timeline for the project.

Exhibit 4: Timeline of activities and deliverables

Activity	Start Date	End Date
Study Design	October 2015	September 2016
Data Collection	October 2016	May 2017
Data Analysis and Reporting	January 2017	March 2018
Data Analysis	January 2017	June 2017
Second TWG meeting	January 2018
Publish final report	March 2018

A17. Approval to Not Display OMB Expiration Date

All data collection instruments will include the OMB expiration date.

A18. Explanation of Exceptions

No exceptions are required.

1 Mendels, Pamela. "Principals in the Pipeline: Districts Construct a Framework to Develop School Leadership." Journal of Staff Development 33.3 (2012): 48-52 and Herman, Rebecca, et al. Turning Around Chronically Low-Performing Schools. IES Practice Guide. NCEE 2008-4020. National Center for Education Evaluation and Regional Assistance (2008).

2 Authorized under the Elementary and Secondary Education Act of 1965 (ESEA), as amended, Title I, Part A, Section 1003(g), 20 U.S.C. 6303(g).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Juanita Lucas-McLean
File Modified	0000-00-00
File Created	2021-01-23