TIF 1875-0256 Surveys_4249_Supporting Statement A_9 9 10 final

This document has been prepared to support the clearance of a revision to the previously approved collection for the data collection instruments for the Evaluation of the Teacher Incentive Fund (TIF) Program (OMB Control Number 1875-0256). The Policy and Program Studies Service (PPSS) of the U.S Department of Education (ED) is conducting this evaluation. PPSS has contracted with SRI International and their subcontractors Berkeley Policy Associates and The Urban Institute to conduct the evaluation. In the introduction to the supporting statement, we provide a description of the TIF program and a description of the evaluation questions and study design. The previously approved collection and this revision focus on the implementation study.

The implementation study of TIF includes telephone interviews and site visits (previously approved) as well as a survey of teachers and principals. This study will help explain the relationships among project characteristics and system supports, and project outcomes. The previously approved submission included the telephone interviews and site visits with TIF grantees. This revision is a request for surveys of principals and teachers in all TIF sites and was not included in the original submission as at that time it was an option that had not been exercised. At this time, the telephone interviews have been completed and will not be discussed here. The two site visits are also included in the discussion and were approved in the original submission.

In addition to the implementation study, the TIF evaluation design includes a feasibility study. The feasibility report is nearing completion and will be presented to OMB for discussion in the coming weeks. If we are approved to conduct an outcomes analysis based on what we learned in the feasibility report, we will submit any appropriate additional revision of the original request. Exhibit 1 summarizes the data collection activities of this evaluation, their completion status, and their approval status.

Exhibit A1

TIF Data Collection Activities

Data Collection Activity	Completion Status	Approval Status
Telephone Interviews	Completed	Approved and completed; not discussed in this addition
Site Visits	Scheduled fall 2010, 2011	Approved; described in this review
Principal and Teacher Survey	Scheduled January 2011	Subject of this OMB revision
Optional Outcomes Study	Feasibility study completed	Feasibility report will be submitted to OMB separately; not discussed in this addition

The Teacher Incentive Fund Grants Program

The TIF program, first funded in fiscal year (FY) 2006 with an appropriation of $99.0 million, was designed to spur an increase in the number and quality of educator performance pay compensation systems. The Teacher Incentive Fund is authorized in PL 109-149, the Appropriations Act of 2006. Discretionary grants are made on a competitive basis by the TIF program office, which is located in the Office of Elementary and Secondary Education (OESE) (See Appendix A).

Local education agencies (LEAs) and State education agencies (SEAs), either alone or in partnership with non-profit organizations, were eligible to apply for competitive grants in FY 2006 to develop and implement innovative performance-based compensation systems for public school teachers and principals in high need schools (including charter schools). Grantees were required by the 2006 appropriations legislation to “develop and implement performance-based teacher and principal compensation systems in high-need schools.” These systems were to “consider gains in student academic achievement as well as classroom evaluations conducted multiple times during each school year among other factors and provide educators with incentives to take on additional responsibilities and leadership roles.” High-need schools were defined as those with more than 30 percent of their enrollments from low-income families (based on free and reduced lunch eligibility or other State-approved poverty measures). The competitive priorities of the 2006 grant competition also stressed a commitment to sustaining the projects and recruiting and retaining qualified teachers in hard-to-staff schools and subject areas.

TIF has funded 34 grantees across the Nation, primarily LEAs but also include State agencies, individual schools, and non-profits (such as charter schools or charter school networks). Grantees vary in the number of eligible educators, from fewer than 100 to more than 10,000, and in the demographic composition of participating schools —although they generally serve schools with high concentrations of minority students and high proportions of low-income students (in order to be eligible for a TIF grant the school must have met the high-need threshold based on poverty measures). One grantee withdrew from the program in 2009 and will not be included in our site visit and survey efforts (though it did participate in phone interviews).

In addition to the TIF program office at ED, TIF grantees are supported by the Center for Educator Compensation Reform (CECR). CECR is an ED-funded organization that raises national awareness about alternative and effective strategies for educator compensation reform and provides technical assistance to TIF grantees on a variety of issues including sustainability, implementation, and evaluation. CECR also hosts an annual grantee conference.

Teacher Incentive Fund Evaluation Implementation Study

The evaluation of the TIF program consists of: (1) an implementation study of the local TIF projects from Cohorts 1 and 2 (funded in 2007 and nearing the end of their performance periods); (2) a study of the feasibility of conducting a rigorous assessment of the relationship between the range of TIF project designs and locales and the outcomes of interest (strengthening the education workforce and increasing student learning); and (3) depending on the results from the feasibility study, an analysis of the program’s outcomes. The full outcomes analysis is proposed as an option, which will be exercised only if the projects have indeed been implemented and appropriate data is available to conduct the analyses. The implementation study is the focus of this revision.

The implementation study will describe: 1) the implementation of performance pay in each TIF grantee; 2) the similarities and differences in implemented performance pay programs across grantees. Data collection activities began with telephone interviews with key stakeholders in the 34 original TIF sites, these have been completed and will not be discussed in this revision. One of the 34 grantees withdrew from the program in fall 2009 and will not be included in subsequent data collection. Two rounds of more in-depth case studies in a sample of sites will be conducted in fall 2010 and fall 2011. These activities were the focus of the original OMB package. This revision reflects the addition of a survey of teachers and principals which will provide data representative of these populations among the 33 TIF grantees. These surveys will provide quantifiable implementation data and provide TIF participants’ and nonparticipants’ knowledge, beliefs, attitudes, and motivations critical to the success of compensation reform. The surveys will be distributed to a random sample of eligible principals and teachers and would represent a full range of program knowledge and experiences in each grantee program. This revision request will focus on the activities remaining in the implementation study including the survey activities as well as a description of the previously approved site visits.

While we propose to collect data on the implementation of TIF, we are aware that another study of TIF will also be undertaken in the coming year. This new study, sponsored by the Institute for Education Sciences (IES) addresses a different research question than the PPSS study that is the focus of this request. The main objective of the IES evaluation is to estimate the impact of differentiated performance-based incentive pay on student achievement and the mobility and retention of effective educators. The evaluation design is an experiment in which researchers will randomly assign schools within a district to either a treatment or control group.

The implementation portion of our study (including the surveys that are the focus of this submission) is not designed to estimate the impact of the TIF program or of any of its component parts. However, our study will collect data on the implementation of the first two rounds of TIF grants, including data on differentiated, performance-based incentive pay as well as on other key program features—such as the role of the principal, teacher evaluation, career ladders, etc. If feasible, we will also investigate student and educator outcomes in the next-most-rigorous study design to randomized trials—namely, regression-discontinuity and interrupted time series. Regardless of whether we conduct an outcomes analysis, however, it is important that the program office and policy makers understand the nature of compensation systems and associated supports (e.g., the educator evaluation systems required by TIF) undertaken by the first two rounds of TIF grantees. Below we describe the evaluation questions that will guide our data collection on implementation.

Implementation Evaluation Questions and Data Sources

The implementation evaluation questions for this study include the following:

What are the main characteristics or components of local TIF performance pay plans in terms of strategies (rewarding educators who increase student learning, attracting effective educators to specific schools or content areas, and/or rewarding educator knowledge and skills), targets (principals and academic teachers; principals only, schoolwide and/or classroom), and sizes of awards (in absolute terms as well as percentage of teacher salaries and expenditures)?
To what extent are grantees implementing performance-pay systems as planned?
What system supports (planning and buy-in, clear communication, project and funding stability, adequate data systems, and alignment with other human resource policies) and broader contextual factors impede or enhance implementation of performance pay systems?
What evidence exists that the performance pay systems are being established in the local grantee sites? In terms of stakeholder satisfaction as well as the percentage of a district’s personnel budget that is used for performance-pay, what are the prospects for sustainability beyond the life of the grant?
What is the feasibility of assessing the outcomes (both educator and student-level) of incentive systems?
Do certain kinds of TIF systems appear to warrant further investigation, including research to improve the design and implementation of any promising projects at the local level?

Implementation Study Data Collection Activities

The evaluation employs multiple methods, gathering both qualitative and quantitative data. Wherever possible, we rely on secondary data sources to reduce burden on project, state, district, and school personnel, and any new data collection will be planned with other data collection efforts in mind to avoid overlap.

Principal and Teacher Surveys

Although there has been a large increase in research on performance pay, most of the recent research has been confined to individual performance pay initiatives. Our evaluation of the TIF program is one of the few studies that examines a large number of initiatives. While the telephone interviews (completed) and site visits will provide detailed information on selected grantees, teacher and principal surveys are the only implementation data scheduled to be collected from all grantees after the second year of the evaluation.

In order to study TIF in all its variations, we plan to include all grantees (regardless of the nature of their performance pay program), and survey as many as 8,000 principals and teachers to look at issues such as teacher understanding and motivation around the program. Because we will include all grantees, we will be able to generalize findings to a broad array of program types.

The surveys will be distributed to a random sample of eligible principals and teachers in all TIF sites. (For a more detailed discussion of sampling, please see Supporting Statement B). The data gathered from these respondents will represent the full range of program knowledge and experiences in each grantee program. The surveys are not designed to estimate the impact of the TIF programs, but rather to provide quantifiable implementation data. These surveys will allow us to capture participants’ and non-participants’ knowledge, beliefs, attitudes, and motivations critical to the success of compensation reform. To insure that the resulting estimates are representative of the TIF project as a whole and can facilitate comparisons across a range of potential subgroups we will survey 526 principals and 6,483 teachers from all 33 grantees currently participating in TIF.

These surveys will support comparisons across individual programs, enabling the identification of the opportunities and barriers common to all TIF grantees and performance pay initiatives in general. Moreover, the survey will be the first comprehensive examination of teachers’ and principals’ participation in a national performance pay effort. Telephone interviews and case studies can identify key issues, and surveys can confirm or clarify how those issues play out across a range of settings.

The surveys will tell us about implementation challenges common to grantees operating in a variety of contexts. A series of questions will capture the contextual factors that define the school and district climate so that we may better understand how the local context can impede or enhance implementation of performance pay systems. The surveys will also ask teachers and principals about their perceptions of the fairness of the compensation system to identify characteristics of the system that promote positive and negative educator views. In addition, the surveys will provide data about how well teachers and principals understand what they have to do to earn an award, what supports they receive, and what they think they do differently as a result of the performance pay system. The surveys will also ask a few questions about respondents’ professional experience (e.g., years in the profession, years worked in the school/district).

Equally important, the surveys will gather information about teacher motivation to participate in performance pay initiatives to determine whether there are particular program features or contextual factors that influence teacher participation and responsiveness to potential rewards. While there is overlap in the topics of interest for the teacher and principal surveys, the focus of the questions will vary. The principal survey will focus more on leadership and include questions about principals’ perceptions of the role of performance pay in school-wide improvements as well as improvements in their own practices and the practices of teachers in their schools.

The teacher survey will be designed for use with all eligible teachers, both participants and non-participants, in the TIF program who teach in grantee schools and districts. We plan to use skip patterns when appropriate to enable us to learn why these teachers chose not to participate and to streamline the process of completing the surveys for non-participating teachers.

We will aim to obtain an equal probability sample of participating teachers within each grantee controlling the number of teachers that would need to be sampled from any individual school. We first select the desired number of schools with probability proportionate to the school’s number of TIF-eligible teachers (imposing minimum and maximum numbers) and then choose an equal number of teachers among the sampled schools. The teacher survey will include more than 8,000 teachers to be sampled from approximately 640 schools representing all grantees (except for two that use TIF solely for principal performance pay). Assuming an 80 percent response rate, this would yield 6,483 completed teacher surveys. The large teacher sample provides several advantages, including a sample representative of the full diversity of grantee efforts as well as the precise measurement of national and grantee-level estimates (where grantee size permits).

For the principal survey, a random sample of participating principals from all grantees will be chosen. Because most grantees have a relatively small number of schools (and, therefore, principals), it is not possible to obtain grantee-level estimates of principals’ responses with any degree of precision for most grantees. Additionally, to insure that the resulting estimates are representative of the TIF project as a whole and can facilitate comparisons across a range of potential subgroups, the sample will include at least 15 schools for all grantees (or all schools in the grantee, if that is fewer than 15 schools); and will not sample more than 60 schools from any grantee.

Sampling methods are described in detail in Supporting Statement B.

Surveys will be provided in two formats: paper and online. The paper survey will be distributed to respondents at their schools, accompanied by instructions on how to complete the survey online if they prefer. We anticipate that the principal survey will take respondents 20-25 minutes to complete and the teacher surveys 15-20 minutes. The storage of electronic and hard copy survey materials will be secure according to guidelines described in the Assurances of Confidentiality section of this document.

Case Study Site Visits

We intend to conduct two sets of site visits to collect in-depth information on grantee practices. The first set of site visits will be conducted in year two of the evaluation (fall 2010) and the second set of site visits will be conducted in the fourth year (fall 2011). Twelve sites will be selected for each set of site visits and each visit will include three days of interviews by a two-person team of researchers to determine progress in implementation. Some of the same sites may be visited in both years of the site visits based on the criteria for inclusion outlined in Supporting Statement B. We will arrange in-person interviews to be conducted during these site visits with representatives from the local TIF project office, principals, a cross-subject and cross-grade sample of teacher participants, and other stakeholders as necessary to get a broader picture of grant activities.

The purpose of the site visits is to: (1) verify data gathered through the telephone interviews (completed) and program documents, and (2) to complement the interviews with information about stakeholder satisfaction and potential for sustainability that can best be gathered through direct observation and in-person discussion. Researchers will learn about the format and content of the funded activities, the types of activities teachers are participating in, the work teachers are expected to perform, and the objectives and goals teachers are expected to reach. In particular, site visits will be crucial in the event that subsequent outcome analyses reveal a significant influence of TIF on outcomes such as retention of effective teachers in high-need schools. By discussing stakeholder satisfaction and observing school climate in the TIF environment, we will gain tools for potentially explaining the correlation of TIF projects and improvements with teacher outcomes.

The grantees selected for the site visits will be selected so as to represent the diversity of TIF grantees’ approaches to developing and implementing performance-based compensation systems. Depending on availability, at least four of the sites will be implementing projects where 50 percent or more of the TIF grant activities award is allocated to performance pay for improved student achievement; four sites primarily implementing broader forms of differentiated compensation (e.g., increased pay for teaching in specific schools or subjects); and four sites implementing the most comprehensive systems (combining awards based on improved student achievement with various compensated teacher professional development activities, e.g., new teacher career ladder projects; Teacher Advancement Program/TAP.

Sites will be selected using methods described in detail in Supporting Statement B.

Protocols for the site visits are semi-structured to allow for the study team to appropriately customize the questions for each respondent. They have been developed to maximize understanding of the unique issues and reform strategies in each site and minimize any redundancy. Depending on the precise roles and responsibilities of each respondent and the data already available from each grantee, interviewers will adjust the protocols to ask only those questions which are appropriate to each respondent and for which researchers do not already have verified data.

Site visitors will review information regarding the grantee before going into the field. Site visitors will receive a manual containing all materials relevant to case study data collection (e.g., lists of types of respondents, selection criteria for respondents, protocols, consent forms, and structured debriefing forms). The training will help team members develop a common understanding of the conceptual framework driving the evaluation, the purposes of the data collection, the protocol questions, and the analyses in which the case study data will be used. Before going into the field, site visitors will also review results of the telephone interviews and extant documents, including grantee applications, annual reports, research reports, and background materials on the district(s) and schools to be visited. The protocols for these site visits are included with this OMB package.

We will make digital recordings of the site visit interviews for completeness and consistency, and will communicate with interviewees as needed to obtain complete data and address inconsistencies in responses if they occur. The interviews will then be coded using both the evaluation questions and themes detected across grantees and question categories to create site summaries. These summaries will be used for cross-site analysis. The storage of electronic and hard copy interview materials will be secure according to guidelines described in the Assurances of Confidentiality section of this document.

Planning and Accounting for Grantee Variation

Based on the uniqueness of each TIF grantee as well as the dynamics of the districts or States in which they operate, our study methodology and data collection instruments have been designed to maximize the quantity and quality of relevant information to be collected from each individual while minimizing the burden on their time.

As mentioned above, the 33 TIF grantees vary widely along a number of attributes—including the recipient (State, local education agency, or nonprofit), the geography and size (whole States, single schools, urban areas, rural areas) that have implications for the design of the study protocols. Our flexible protocols—partnered with highly trained interviewers—will allow for interviews that are tailored and appropriate for a given grantee’s experiences and project structure. In the development of the principal and teacher surveys we designed questions, instructions, and skip patterns to allow for instruments that are clear and concise and applicable for the variability across grantees.

For the site visits, both the number and nature of key informants interviewed and the positions they hold will also vary somewhat across projects based on the specifics of each grantee. For example, in a TIF site where the grantee is a State entity and the project includes multiple districts and many schools, there will likely be roles filled by a number of individuals in the project office. In other sites, such as in the case of a small network of charter schools, a single person may represent the project office. Likewise, the individual representing the grantee leadership may be a district superintendent if the grantee is a district or a principal if the grantee is a school. The breadth of involvement will also vary by site making the flexible protocols essential to the smooth running of the interviews. For example, the technical assistance providers may be integrally involved in the projects of some grantees while playing a smaller role in others, local unions have been more or less involved in specific sites, the scope and stage of evaluations vary, and in some sites there may have been little or no media coverage of the program. Protocols contain all of the questions that might be asked of any respondent with each role. Based on a given individual’s responsibilities, the interviewer will ask only the questions that apply to a particular individual.

Key informants at each site and for each role will be identified with assistance from the TIF program office, along with the grantee’s project leadership (most often the project director).

Before data collection begins, interviewers will review extant documents that have been submitted to the TIF program office, including grantee applications, annual reports, research reports, and background materials on the grantees. Interviewer training will focus on providing the interviewers the necessary background and information to customize each interview to account for variation in grantee characteristics as well as prepare interviewers to respond to and pursue unanticipated or interesting findings.

When taken together, the three data collection activities for the implementation study not only complement each other, they solidify findings by triangulating the data. Understanding performance pay is particularly important at a time when the issue is near the top of policy discussions at the federal, state, and district levels.

Instruments to be Cleared through this Submission

We are requesting clearance for the survey data collection instruments which are included with this submission. Also included are the previously approved protocols for the site visits. Exhibit A2 provides a summary of the data collection instruments for the implementation study.

Exhibit A2

Instruments for Clearance

IC Title	Collection Activity	Approval Status	Timeline
IC 16 TIF Principal Survey	Principal and Teacher Surveys	Revision submission	Winter 2011
IC 17 TIF Teacher Survey	Principal and Teacher Surveys	Revision submission	Winter 2011
IC 18 Survey Endorsement letter from ED	Principal and Teacher Surveys	Revision Submission	Winter 2011
IC 19 TIF Principal Survey Letter	Principal and Teacher Surveys	Revision Submission	Winter 2011
IC 20 TIF Teacher Survey Letter	Principal and Teacher Surveys	Revision Submission	Winter 2011
IC 1 Letter of Data Collection Notification	Telephone Interviews	Approved: original submission	Complete
IC 2 Award Structure and Payout Form	Telephone Interviews	Approved: original submission	Complete
IC 3 Telephone Interview Protocol: TIF Project Staff	Telephone Interviews	Approved: original submission	Complete
IC 4 Telephone Interview Protocol: Grantee-level Senior Administrative Staff	Telephone Interviews	Approved: original submission	Complete
IC 5 Telephone Interview Protocol: Educators in TIF Sites	Telephone Interviews	Approved: original submission	Complete
IC 6 Telephone Interview Protocol: Stakeholders in TIF Sites	Telephone Interviews	Approved: original submission	Complete
IC 7 Site Visit 1 Protocol: TIF Project Staff	Site Visit 1 Protocols	Approved: original submission	Fall 2010
IC 8 Site Visit 1 Protocol: Grantee-level Senior Administrative Staff	Site Visit 1 Protocols	Approved: original submission	Fall 2010
IC 9 Site Visit 1 Protocol: Educators in TIF Sites	Site Visit 1 Protocols	Approved: original submission	Fall 2010
IC 10 Site Visit 1 Protocol: Stakeholders in TIF Sites	Site Visit 1 Protocols	Approved: original submission	Fall 2010
IC 11 Site Visit 2 Protocol: TIF Project Staff	Site Visit 2 Protocols	Approved: original submission	Fall 2011
IC 12 Site Visit 2 Protocol: Grantee-level Senior Administrative Staff	Site Visit 2 Protocols	Approved: original submission	Fall 2011
IC 13 Site Visit 2 Protocol: Educators in TIF Sites	Site Visit 2 Protocols	Approved: original submission	Fall 2011
IC 14 Site Visit 2 Protocol: Stakeholders in TIF Sites	Site Visit 2 Protocols	Approved: original submission	Fall 2011
IC 15 Consent Form	Telephone Interviews, Site Visit 1, Site Visit 2	Approved: original submission	Complete, Fall 2010, Fall 2011

II. SUPPORTING Statement For Paperwork Reduction Act Submission

A. Justification for the Evaluation of the Teacher Incentive Fund (TIF) Program

1. Necessity of Information Collection

This evaluation is the first systematic study of the TIF grants program. Because TIF represents a substantial investment in teacher performance pay programs, this evaluation is crucial for (1) establishing whether the program is working as intended by Congress and (2) identifying which elements of TIF projects are most effective.

This is a revision to the previously approved collection for the data collection instruments for the Evaluation of the Teacher Incentive Fund (TIF) Program (OMB Control Number 1875-0256).

2. Use of Information

The results of this data collection will be used to inform a variety of stakeholders regarding the nature and outcomes of the TIF program. More specifically, the information will be used to:

Describe the main characteristics or components of local TIF performance plans;
Ascertain grantee implementation and prospects for project sustainability beyond the life of the grant;
Describe system supports and broader contextual factors that impede or enhance implementation of performance pay systems;
Inform Congress and other policy-makers about federally-supported educator performance pay;
Inform future reauthorizations of the TIF program.

The audience for this evaluation includes the ED, Congress, education policy-makers, K-12 administrators and teachers, and researchers.

3. Use of Information Technology

We will use a variety of advanced information technologies to maximize the efficiency and completeness of the information gathered for this evaluation and to minimize the burden the evaluation places on respondents at the district and school levels. Respondents for the principal and teacher surveys will be given the option to complete either a paper and pencil or online version. During the survey data collection period, an e-mail address and phone number will be available for respondents to contact with questions or requests for assistance. The e-mail address will be printed on all data collection instruments, along with the name and phone number of a member of the data collection team. A response rate tracking system will be set up in advance to provide regular updates on response rates. For the site visits, technology will be used only by project staff in recording, analyzing, and reporting.

4. Efforts to Identify Duplication

At each phase of the evaluation, efforts will be made to minimize and reduce the burden on respondents. Wherever possible, we rely on secondary data sources to reduce burden on project, State, district, and school personnel. These data collection activities will be designed to avoid redundancy and to build on the work of other researchers. We are familiar with ongoing research on pay-for-performance, as well as the specific evaluations of the TIF grantees, and propose to complement—rather than duplicate—existing data activities.

To minimize respondent burden we will build on existing evaluations of TIF grantees. We will gather existing evaluation reports in order to understand exactly what kind of data they are collecting and from whom. We will also work local evaluators to see if they can schedule their local data collection activities at a different time than the national evaluation surveys.

We believe that the sponsorship by ED of the grantees and local grantee evaluations offers a special opportunity to avoid redundancy and reduce burden. As a condition of the program, all TIF Grantees are conducting local evaluations the results of which are reported to the TIF program office along with annual performance reviews. These data will be accessed by researchers prior to conducting original data collection. Because our evaluation includes both the documents and interviews with local evaluators, we have the opportunity to build on, rather than replicate, this existing work. We will also work closely with researchers from the Center for Educator Compensation Reform (CECR)—the technical assistance provider for the grantees. We will review reports produced by CECR. We will also interview CECR technical assistors as part of data collection for each grantee.

5. Methods to Minimize Burden on Small Entities

TIF grantees are generally local school districts, but also include State agencies, charter schools, and school networks. The evaluation strategy—outlined in other parts of this document—is to minimize the burden on respondents in evaluation activities by collecting as much relevant information as possible prior to conducting interviews and site visits, to refine protocols in advance of interviews to avoid redundancy, and streamline the data collection for each respondent.

6. Consequences if Information is not Collected or is Collected Less Frequently

Failure to collect this information will prevent Congress and ED from obtaining evaluation data necessary to assess the impact of the TIF program on educator recruitment and retention and student achievement. Failure to collect this data would also impede progress in determining the components of effective educator compensation reform initiatives.

7. Special Circumstances

Not applicable.

8. Federal Register Comments and Persons Consulted Outside the Agency

The 60 day Federal Register notice was published in Volume 75, page 17135 on April 5, 2010.

Federal Register Comments. No public comments have been received.

Consultation Outside the Agency and with Respondent Representatives.

We have assembled a Technical Working Group (TWG) of individuals with expertise in implementation of teacher and principal incentive systems, including performance pay, teacher profession development, administrator (principal) compensation reform, and/or a strong quantitative research background and expertise in regression discontinuity and other quasi-experimental designs/methodology. The TWG is advising the evaluation on all matters related to study design, sample selection, instrumentation, data collection, and data analysis. The members of the TWG, and their titles and professional affiliations, are:

Teacher professional development

Suzanne Wilson

Director of the Center for the Scholarship of Teaching and Chair of the Department of Teacher Education

Michigan State University

Jennifer King Rice

Associate Professor, Department of Education Policy and Leadership

University of Maryland

Performance pay

Patricia King

Director of School Improvement

Minnesota Department of Education

Matthew G. Springer

Research Assistant Professor of Public Policy and Education

Vanderbilt University

Administrator Compensation Reform

Dan D. Goldhaber

Research Professor

University of Washington

Ellen B. Goldring

Professor of Educational Policy and Leadership

Vanderbilt University

Regression discontinuity or other quasi-experimental designs/methodology

Peter Schochet

Senior Fellow

Mathematica Policy Research, Princeton, NJ

Susanna Loeb

Director of the Institute for Research on Education Policy and Practice

Stanford University

In addition to the Technical Working Group, we consulted with TIF program officers from ED.

9. Respondent Gifts

We will not make any monetary gifts or payments to survey respondents in data collection activities. School site liaisons for the teacher survey will be compensated (through a subcontractor) as is discussed in Supporting Statement B.

10. Assurances of Confidentiality

We have established a set of standards and procedures to safeguard the privacy of participants and the security of data as they are collected, processed, stored, and reported. As the lead in the data collection for the evaluation, the SRI research team will adhere to the Multiple Projects Assurance with the Office of Protection from Research Risks (OPRR) maintained by SRI. SRI's Assurance number is M-1088. SRI’s Human Subjects Committee is its official Institutional Review Board (IRB) charged with responsibility for the review and approval of all research involving human subjects. SRI clears all data collection protocols through its internal Human Subjects Committee as a safeguard to protect the rights of our research subjects. The National Evaluation of the Teacher Incentive Fund has already gained approval from SRI's Human Subjects Committee.
SRI project staff have extensive experience collecting information and maintaining confidentiality, security, and integrity of interview and survey data. In accordance with the SRI institutional policies, privacy and data protection procedures will be in place. These standards and procedures for survey data are summarized below.

Responses to this data collection will be used to summarize findings in an aggregate manner (across groups of grantees), or will be used to provide examples of program implementation in a manner that does not associate responses with a specific site or individual. In the report, pseudonyms will be used for each grantee.
The study team may refer to the generic title of an individual (e.g., "project director," or "high school teacher") but neither the grantee name nor individual name will be used. The contractor will not provide information that associates responses or findings with a subject to anyone outside the study team. We will aggregate data across respondent types (e.g., project directors, teachers) for most of the study analyses. In no case will we disaggregate data to such a degree that it would be possible to identify individual grantees or other entities.
We will acknowledge the cooperation of participating institutions in the final report, but will not identify them in the text of any report.
We will inform participants of the purposes of the data collection and the potential uses of the data collected. The risks and benefits of participating in this study, which are expected to be nominal, will be explained in the text of the survey introduction and by interviewers for the site visit. Informant’s consent will be actively requested and documented during site visits.
We will educate project team members on respondents’ confidentiality assurances and how to handle sensitive materials and data. We will caution all persons assigned to the study not to discuss confidential data.
During training, we will reemphasize the need for interviewers and other data collection personnel to protect the privacy of respondents. We will caution such personnel not to discuss interview data with others outside the evaluation, and emphasize that they must restrict discussions within the evaluation team to the essential needs of the data collection activity.
Each principal survey will be accompanied with a return envelope so as to allow the respondent to seal it, once it has been completed. All respondents will mail their postpaid surveys individually to SRI. For the teacher survey, school-site liaisons will collect the surveys from respondents in sealed envelopes to return in a package for the whole school. If the survey is taken online, security measures are in place to safeguard respondent identities and to ensure the integrity of the data.
We will disassociate names and addresses from the data as they are entered into the database and will be used for data collection purposes only. As we gather information on individuals or sites, we will assign each a unique identification number, which we will use for raw data, print-out listings that display the data, and analysis files. We will also use the unique identification number for data linkage. The researchers will not provide information that associates responses or findings with a subject or district to anyone outside the study team.
All paper surveys will be stored in secure areas accessible only to authorized staff members.
Access to the survey database will be limited to authorized project members only; no others will be authorized such access. Multilevel user codes will be used, and entry passwords will be changed frequently.
All identifiable data (e.g., tracking data) will be shredded as soon as the need for this hard copy no longer exits.
We will store all electronic recordings of interviews, interview notes, and other project-related documents in secure areas that are accessible only to authorized staff members.
We will shred all interview protocols, forms, and other hard-copy documents containing identifiable data as soon as the need for this hard copy no longer exists.
We will duplicate all basic computer files on computer-based backup servers to allow for file restoration in the event of unrecoverable loss of the original data. We will store these backup files under secure conditions in an area separate from the location of the original data.

11. Questions of a Sensitive Nature

There are no questions of a sensitive nature included in data collection instruments or procedures. Participation in the study is voluntary and all data collection activities will be conducted as described above to assure respondent confidentiality.

12. Estimate of Information Collection Burden

Respondents will not incur any equipment, postage, or travel costs. Exhibit A3 displays estimates of the total respondent burden in hours and dollars for the principal and teacher surveys and the site visits. The telephone interviews are complete and are not included with this package. The annual reporting and record keeping burden over the three years of the study is with 2,627 respondents and 1,009 hours. Of the 1,009 burden hours for this revision, 280 burden hours are being carried over from the currently approved OMB submission. The school liaisons are hired by the contractor to act as assistants with such tasks as distribution and pick-up of the surveys as well as providing follow-up assistance with nonrespondents. Since they are not respondents, the time spent in these administrative activities is not considered as burden in this collection.

For this revision, principal and teacher estimates include the time associated with completing the survey about the TIF program. Project directors maintain school rosters as part of their regular project administration which will be requested from each of the 33 TIF sites. A request for school rosters will be sent to project directors at each of the grantees. Based on our conversations with the TIF program office, we understand that grantees have this type of information as part of their regular reporting and the format in which we are requesting it should be relatively easy for them to provide.

Two sets of site visits will occur during the course of the study. In the first set of site visits, to be conducted in 2010, 12 grantees will be selected. Within each approximately 30 to 40 respondents will be interviewed per grantee for a total of approximately 420 participants.

In the second set of site visits, to be conducted in 2011, 12 grantees will be selected and within each approximately 30 to 40 respondents will be interviewed for a total of approximately 420 participants. Some of the same sites may be visited in both years site visits based on the criteria for inclusion outlined in Supporting Statement B. It would be possible to visit anywhere from 12 grantees (if the same sites are selected in both samples) to 24 grantees over the two rounds of site visit data collection.

Some participants (for example, project directors) will have participated in the preceding telephone interviews. During the site visits, interviews will be conducted with a variety of key informants including: TIF project staff (the project director at each site as well as other staff, technical assistance providers and evaluators), educators (approximately 15 teachers and 3 principals), 5 senior administrative staff (e.g., personnel directors, data management experts, superintendents of instruction), and 7 stakeholders in TIF sites (e.g. representatives of partner organizations, representatives of parent organizations, teacher association officials, and local education reporters). All numbers are approximate as respondents will be selected as needed to fully understand the unique experience of each grantee.

Exhibit A3

Respondent Burden

	Number of Respondents		Hours per Response	Total Hours	Average Hourly Wage	Total Cost
Principal and Teacher Surveys—2011
Project Directors	33	1		33	$40	$1,320
Teachers	6,483*	0.3		1944.9	$25	$48,623
Principals	526*	0.4		210.4	$40	$8,416
Site Visit 1—2010
TIF Project Staff*	60		1	60	$40	$2,400
Educators in TIF Sites	216		1	216	$40	$8,640
Grantee-level Senior Administrative Staff	60		1	60	$60	$3,600
Stakeholders in TIF Sites	84		1	84	$40	$3,360
Site Visit 2—2011
TIF Project Staff*	60		1	60	$40	$2,400
Educators in TIF Sites	216		1	216	$40	$8,640
Grantee-level Senior Administrative Staff	60		1	60	$60	$3,600
Stakeholders in TIF Sites	84		1	84	$40	$3,360
Total	7,882			3,028		$94,539
Average Annual Burden	2,627			1,009

Note: For this revision request the annual average number of respondents is 2,627 respondents and 1,009 average burden hours. For the previously approved request (including the completed telephone interviews the annual average number of respondents is 393 and the annual average burden hours is 393.

*Sample sizes of 657 (principal) and 8,104 (teacher) will be selected to account for anticipated non-response. See Section B regarding respondent universe.

13. Estimate of Total Annual Cost Burden

There are no additional respondent costs associated with this data collection other than the hour burden estimated in item 12.

14. Estimates of Annualized Costs

The estimated annual cost to the federal government for the entire TIF evaluation, as specified in the contract, is displayed in Exhibit A4. The study began in October 2008. The final report will be due in July 2013 (Month 57 of the program).

Exhibit A4

Annual Cost

	Total Evaluation	Implementation Data Collection	Current Request	Previously Approved
	Total Evaluation	Implementation Data Collection	Principal and Teacher Surveys	Site Visits
Year One: FY 2009	$833,073	$87,298	$54,881	$32,417
Year Two: FY 2010	$1,070,980	$698,887	$416,324	$282,563
Year Three: FY 2011	$615,193	$358,233	$324,954	$33,279
Year Four: FY 2012	$924,573	$313,865	--	$313,865
Year Five: FY 2013	$170,169	--	--	--
Total	$3,613,988	$1,458,283
Average annual cost over three years of data collection		$486,094

We estimate that the average annual cost to the federal government for the activities covered under this OMB revision is $486,094. These activities will occur in FY 2009 thorough FY 2012 and will include collecting and analyzing case study data and principal and teacher surveys, and reporting on results.

15. Change in Annual Reporting Burden

The annual reporting and record keeping burden for the previously approved OMB submission over the three years of the study was 393 hours with 393 respondents. The telephone interviews are complete and are not included with this revision. The annual reporting and record keeping burden over the three years of the study is 1,009 hours with 2,627 respondents. This represents an increase of 616 annual hours.

16. Project Time Schedule

We will conduct the case study tasks according to the schedule shown in Exhibit A3.

Exhibit A3

Schedule of Data Collection Tasks and Deliverables

Data Collection Activity	Data Collection Tasks and Deliverables	Time Period
Current Request
Principal and Teacher Surveys	Survey Sampling	December 2010
	Survey Administration	January – May 2011
	Survey Analysis	August 2011
Previously Approved
Case Study I	Conduct site visits for Case Study I	September – November 2010
	Draft Cross-Case Analysis Memo I	November 2010
	Final Cross-Case Analysis Memo I	January 2011
Case Study II	Conduct site visits for Case Study II	September – December 2011
	Draft Cross-Case Analysis Memo II	February 2012
	Final Cross-Case Analysis Memo II	April 2012

17. OMB Expiration Date

We will inform respondents about the OMB expiration date when they are notified about the study.

18. Exceptions to Certification Statement

No exceptions are requested.

Appendix A1. Legislation Authorizing the Teacher Incentive Fund Program

The Teacher Incentive Fund is authorized in P.L. 109-149 -- the Departments of Labor, Health and Human Services, and Education, and Related Agencies Appropriations Act, 2006, Title V, Part D.

Provided further, That $260,111,000 shall be available to carry out part D of title V of the ESEA, of which $100,000,000 of the funds for subpart 1 shall be for competitive grants to local educational agencies, including charter schools that are local educational agencies, or States, or partnerships of (1) a local educational agency, a State, or both and (2) at least one non-profit organization to develop and implement performance-based teacher and principal compensation systems in high-need schools:
Provided further, That such performance-based compensation systems must consider gains in student academic achievement as well as classroom evaluations conducted multiple times during each school year amount other factors and provide educators with incentives to take on additional responsibilities and leadership roles:
Provided further, That five percent of such funds for competitive grants shall become available on October 1, 2005 for technical assistance, training, peer review of applications, program outreach, and evaluation activities and that 95 percent shall become available on July 1, 2006 and remain available through September 30, 2007 for competitive grants.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	REQUEST FOR CLEARANCE OF DATA COLLECTION INSTRUMENTS FOR THE EVALUATION OF THE TEACHING AMERICAN HISTORY GRANTS PROGRAM
Author	Policy Division
File Modified	0000-00-00
File Created	2021-02-02