RELMA_PLC_OMB_PartA

OMB Package Part A: Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative

September 2013

Table of Contents

Part A. Justification 1

1. Circumstances Necessitating the Collection of Information 1

a. Statement of Need to Evaluate District-Wide Implementation of a Professional Learning Community Initiative 1

b. Research Questions 2

c. Study Overview 3

2. Purposes and Uses of Data 6

3. Use of Technology to Reduce Burden 6

4. Efforts to Avoid Duplication 7

5. Methods to Minimize Burden on Small Entities 7

6. Consequences of Not Collecting Data 7

7. Special Circumstances 7

8. Federal Register Announcement and Consultation 8

9. Consultations Outside the Agency 8

10. Payments or Gifts 8

11. Assurances of Confidentiality 8

12. Justification for Sensitive Questions Hours 10

13. Estimates of Burden Hours 10

14. Estimates of Cost Burden to Respondents 11

15. Annualized Cost to the Federal Government 11

16. Reasons for Program Changes or Adjustments 11

17. Plans for Tabulation and Publication of Results 11

a. Tabulation Plans 12

b. Publication Plans 14

18. Approval Not to Display the Expiration Date for OMB Approval 14

19. Exception to the Certification Statement 14

References 15

Part A. Justification

This submission is a request for approval of data collection activities that will be used to support the Mid-Atlantic Regional Educational Laboratory (REL MA) project, Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative. The study is being funded by the Institute of Education Sciences (IES) of the U.S. Department of Education (ED), and is being implemented by ICF International and its subcontractor, Rutgers University’s Center for Effective School Practices.

This study aims to address the need for systematic information about district-wide implementation of professional learning communities as a critical element in improving teacher quality and instruction, thereby contributing to increased student achievement. The study will survey online a population of teacher participants in school-based professional learning communities and interview principals face to face about the context and their perceptions of the initiative, pre- and post-implementation. Data collection from teachers will focus on what the professional learning communities do, how they operate, and to what extent they produce the outcomes expected of them as framed by six conceptual attributes of professional learning communities and five specific tasks. Data collection from principals will focus on contextual information about school culture and conditions such as resources that support implementation. Teachers and principals will also provide their reflections on the challenges of implementing professional learning communities and their suggestions for improvement. The analysis will enable comparisons among professional learning communities within and across schools, and between teachers’ pre-implementation expectations and post-implementation experiences. Study findings are expected to inform both theory and practice related to implementation of professional learning communities.

This submission requests approval to recruit principals and teachers for the study and conduct in-person interviews with principals and online surveys with teachers in the study district.

• Circumstances Necessitating the Collection of Information

• Statement of Need to Evaluate District-Wide Implementation of a Professional Learning Community Initiative

The specific legislation authorizing this data collection is specified in Part D, Section 174 (20 U.S.C. 9564) of the Education Sciences and Reform Act (ESRA) of 2002. Part D, Section 174 provides for ED to enter into five-year contracts with entities to establish a networked system of ten regional laboratories that serve the needs of each region of the United States. The Regional Educational Laboratories (RELs) are to carry out a range of activities to serve the needs of each region in the United States, including applied research, and development, dissemination, training, and technical assistance activities that focus on how to use data and analysis. The primary mission of the RELs is to help states and districts systematically use data and analysis to answer important issues of policy and practice with the goal of improving student outcomes. Accordingly, this project was conceived in response to a specific request made of the REL MA from a superintendent of a large school district in Pennsylvania (West Chester Area School District, abbreviated as WCASD) to assist in the evaluation of the implementation of professional learning communities in the district during the 2013-2014 school year.

Practitioners in the education field have widely embraced professional learning communities (PLCs) because they are aligned, at least in theory, with several features of effective professional development, such as active learning and teacher collaboration (for a review, see Stoll, Bolam, McMahon, Wallace, & Thomas, 2006). These collaborative teams are believed to be effective because they give teachers the opportunity to be exposed to new ideas and practices, and improve their pedagogy via a process of critical reflection (Hord, 1997; Wood, 2007). Operationally speaking, PLCs are teams of educators (most commonly, teachers) that meet regularly, either in-person or virtually, often but not always during scheduled school time, to develop lesson plans, examine student work, monitor student progress, assess the effectiveness of instruction, and identify needs for professional learning. These teams operate with a commitment to the norms of continuous improvement and experimentation, and engage their members in improving their daily work to advance the achievement of districts’ and schools’ goals for student learning (National Staff Development Council, 2001). Ideally, PLCs should be characterized by a set of dimensions or attributes such as shared beliefs, values, and vision; shared and supportive leadership; supportive structural conditions; supportive relational conditions; collective learning; and peer sharing (DuFour, 2005; Hord, 1997). Further, it is generally acknowledged that strong PLCs are not only those in which new knowledge regarding content and pedagogy is acquired, but also those in which existing assumptions about teaching and learning are challenged and critiqued (Little, 2003).

The increasing interest of education stakeholders in PLCs also appears to reflect a belief in the effectiveness of these learning communities. Recognizing the value of building systems for peer-to-peer professional learning among teachers and other education leaders, the U.S. Department of Education is investing in and supporting PLCs as part of several key initiatives, including Investing in Innovation (i3), the Office of Special Education Programs’ technical assistance centers, Promise Neighborhoods, and Race to the Top. Thus, there is strong support among school administrators and policymakers alike for expanding this practice (Talbert, 2010).

Unfortunately, studies that evaluate the effects of PLCs on teacher professional learning and/or student learning outcomes are limited in number. A recent review (Vescio et al., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ self-reports. However, the researchers found cumulative evidence that is consistent with the hypothesized effects of PLCs. They concluded that participation in PLCs influences teaching practice as teachers move toward student-centered instruction, and that when teachers participate in a learning community, students benefit as well, as indicated by improved achievement scores over time. Similarly, a recent meta-analysis found that PLCs have a small but statistically significant positive effect on student achievement (Lomos, Hofman, & Bosker, 2011). Thus, while the majority of studies that evaluated the efficacy of PLCs have important methodological limitations, the limited evidence from programs that were more systematically evaluated seems to support the connection between teachers’ conscious efforts to improve instructional practice through group/peer study and changes in classroom practice and student achievement (Vescio et al., 2008). We expect that in addition to serving WCASD-specific information needs, this study will help close the important theoretical and empirical gap in the current literature by addressing the question of how and under which conditions (e.g., leadership style, group norms, available resources, and work arrangements) PLCs get in a position to achieve key desired attributes of collaborative work and meet standards of productivity that have the potential to enhance student learning and achievement.

• Research Questions

The primary research questions to be addressed in the study are listed below. The questions were formulated with direct input from the leadership of WCASD to ensure that the study meets their specific needs. However, these questions are also informed by the existing gaps in the current literature regarding the “how and under what conditions” focus discussed in the previous section.

• Research Question 1: How do PLC teams implement the study district’s PLC tasks? That is, what collaborative teamwork routines do they develop to achieve the key attributes of PLCs and the key tasks listed in the study district’s PLC materials?

• Research Question 2: How do the study district’s teachers and principals evaluate their experience with the program? Is their actual PLC experience the same or different from what they expected it to be before the implementation of the program? How do teachers’ evaluation of the program and of the implementation compare with those of principals?

• Research Question 3: What specific artifacts (e.g., essential learning targets, standardized common assessments, and systematic interventions) are produced by each PLC team in the district?

c. Study Overview

In response to a specific request from the superintendent, REL MA will evaluate the implementation of WCASD’s district-wide PLC initiative during the 2013-2014 school year. As a result of its planning process, WCASD produced the WCASD PLC Protocol that explains what PLCs are, how and why they will be integrated into the district, and the specific sequence of five tasks that each PLC team should accomplish over time, regardless of grade level and content area. The protocol was shared with all educational staff in the district but does not specify how PLC teams should be established or what work routines they should adopt. Rather, this decision is left up to the principal and teachers of each school in order to accommodate the unique set of circumstances and resources that exist within each school and to leverage more effectively the expertise and experience of current teachers and administrators.

Because PLC teams in the district will not follow a uniform, fully structured implementation design, the proposed study has two main goals: to describe the process of implementing PLCs in WCASD and to assess and compare the progress that these PLC teams make on specific tasks related to collaborative professional learning and instruction during the first year of implementation. Specifically, the study will involve collecting survey data from WCASD’s teachers and interview data from WCASD’s principals to: describe and compare (a) the work routines of teachers in PLC teams within and across schools in the district and (b) the specific outcomes (or products) of their collaborative work. REL MA will also ask teachers and principals to evaluate both their experience with specific aspects of the implementation – such as team work, productivity, and professional development – and lessons they learned from that experience. This information will permit WCASD to identify evidence-based best practices that can support the ongoing implementation of the district’s PLC program and its impact. This information will also permit WCASD to identify actual and/or perceived barriers to implementation during the first full year of implementation that the district may want to address in subsequent years of the implementation.

To be efficient, REL MA proposes to use the same survey data collected from individual teachers, at the aggregate, to assess and compare the progress that PLC teams in the district make on achieving the stated goals of the program during the first full year of implementation. First, the data collected from individual teachers who are members of the same PLC team will be aggregated to represent measures of PLC team-level variables of interest such as communication and leadership style in the group. Next, the aggregated data (with PLC teams being the units of analysis) will be analyzed in relation to a logic model (see Figure 1) which REL MA developed with input from the district based on a scheme proposed by Nelson and colleagues (Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012) that inquires about the extent to which core components of the program have been implemented. The model is designed to benchmark the “PLC status” (as beginning, evolving, or mature: see Grossman et al., 2000) of each PLC team in the district based on a measurement scale that is theoretically derived from the six key desired attributes of effective PLCs (i.e., shared mission, collaborative culture, collective inquiry, action orientation, commitment to continuous improvement, and results orientation) as identified in the extant literature on the topic (see DuFour et al., 2010; Hord, 1997) and as also referenced in WCASD’s PLC protocol. The short-, medium-, and long-term outcomes listed in the logic model specify the characteristics PLC teams should exhibit as they evolve over time from a beginning status in the short term to a mature status in the long term. The model also specifies the five sequential tasks that PLC teams in the district were asked to complete. These are: (a) formulation of essential learning targets, (b) development of common formative assessments that measure proficiency on essential learning targets, (c) modifications of the core instructional program to allow differentiated instruction by degree of student proficiency on one or more learning targets, (d) development of supplemental interventions for students who do not demonstrate proficiency on one or more learning targets through the core instructional program, and (e) formation of child study teams to coordinate more intensive and more frequent individualized interventions for students who do not meet proficiency on one or more learning targets through the core instructional program and supplemental interventions. In all, using the logic model as an analytical tool will allow the study team to answer the research questions while providing WCASD with valuable insights about the progress of the PLC initiative district-wide.

Data for the study will be collected through a post-implementation online survey of all teachers (N=930) in all of WCASD’s 16 schools, and through semi-structured interviews with all principals in the district (N=16) post-implementation. REL MA will design and administer the post-implementation survey of teachers and will also design and collect the post-implementation interview data from principals. All WCASD’s teachers and principals will be involved in data collection activities because PLCs reside in individual schools and depend to a large extent on school-level supporting conditions. Teachers are the members of PLCs; it is their expectations and perceptions we wish especially to discern. This study will enable a fine-grained assessment of their experiences post-implementation as well as how their experience compares to that of other team members (to this end we will use a unique identifier to link individual teachers to PLC teams). Principals are included because they are on-the-scene observers and gatekeepers who control access to critical resources for implementation, and because they are key stakeholders who are deeply invested in the outcomes of teacher quality and student achievement. Importantly, because PLCs in WCASD will be given complete freedom to organize, it is very likely that teams’ composition and work routines (and therefore the PLC experience of individual teachers and principals) will vary within and among schools in the district. Therefore, it is not feasible to limit data collection to a random sample of teachers and administrators in light of the goal of providing adequate representation of their experiences with implementation in the absence of a well-defined sampling frame. The specific data collection plan with the timeline for all data collection activities is included in Table 3.

Figure 1: Logic Model for Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative

Inputs		Outputs			Outcomes
		Activities	Participation		Short-term	Medium-term	Long-term
Principals facilitate the process of determining grade- and subject-specific PLC teams within each building, such that all members on any given PLC team share learning outcomes (that is, teachers must teach the same academic standards in order to be on the same PLC team). Each PLC team is provided with a copy of the district-wide PLC protocols. Each PLC team meets for approximately 45–60 consecutive minutes weekly. PLC teams have access to expertise from external sources, such as program supervisors, colleagues from other schools, and university professors.		PLC teams organize independently to achieve key attributes of PLCs (shared mission, collaborative culture, collective inquiry, action orientation, commitment to continuous improvement, and results orientation) that are described in the PLC protocols. Task 1: Collaborate to determine 3–5 essential learning targets for each unit taught as part of the core instructional program. Task 2: Design two versions of a common formative assessment for each unit taught to assess students’ proficiency in the essential learning targets. Task 3: Modify core instructional program based on assessment results to allow differentiated instruction. Task 4: Develop supplemental interventions for nonproficient students. Task 5: Form child study teams to coordinate more intensive and more frequent interventions for struggling students.	All WCASD teachers in their respective PLCs All WCASD teachers of a given subject and their program supervisor PLC teams working independently to create the assessments and then reaching consensus with other subject-specific PLC teams PLC teams PLC teams PLC teams (potentially, multiple teams involved)		Regular and active participation in PLC activities Positive evaluation of the PLC experience Positive beliefs about the professional development (PD) value of PLCs Positive beliefs about the impact of PLCs on students Teachers positively evaluate and agree with the essential learning targets. Teachers positively evaluate the quality of the common formative assessment created by the team. Teachers analyze student assessment on formative common assessment to identify need in differentiated instruction. Teachers analyze student assessment on formative common assessment to identify nonproficient students. Teachers analyze student assessment on formative common assessment to identify struggling students.	Continued regular and active participation in PLC activities Personal commitment to collaborative work Personal commitment to PD and learning collaboratively Translation of (or intention to translate) knowledge acquired to actual classroom practice Teachers use essential learning targets to guide instruction activities. All teachers make standard and consistent use of the common formative assessment. Teachers assign students to receive differentiated instruction based on analysis. Teachers assign students to receive supplemental interventions based on analysis. Teachers form child study teams to design individualized interventions for struggling students.		Sustainability of PLCs in WCASD Shared professional culture that values collaboration and learning in WCASD WCASD instrumental in diffusing the practice to other educational entities in the region and beyond As a matter of routine, teachers revise and update essential learning targets as necessary. As a matter of routine, teachers revise and update the common formative assessment as necessary. As a matter of routine, teachers review student progress and reconsider differentiated instruction assignment. As a matter of routine, teachers review student progress and reassess need for intervention (as well as the intervention itself). As a matter of routine, teachers review student progress and reassess need for intervention (as well as the intervention itself).

• Purposes and Uses of Data

Rutgers University’s Center for Effective School Practices will analyze all project data for the REL MA Evaluation of a District-Wide Implementation of a Professional Learning Community Initiative under contract number ED-IES-12-C-0006.

The primary purpose of the study is to evaluate the implementation of the district-wide PLC initiative in the WCASD in response to a specific request from the superintendent. The proposed study has two main goals: to describe the process of implementing PLCs and to assess and compare the progress that PLC teams in the district make on specific tasks related to collaborative professional learning and instruction during the first year of implementation against the desired progress as per the program’s logic model.

Findings from this study will be used by WCASD to formulate an operational definition of each attribute of PLCs and assess PLC teams’ progress toward achieving each one of these attributes, as well as progress toward completing the tasks identified in the district’s PLC protocol. The insights generated by the study will also assist the district’s leadership in considering programmatic modifications that can enhance the productivity and sustainability of PLCs and provide baseline data for continuing efforts to monitor their functioning. The close collaboration between the district and study team is also expected to help build the district’s internal capacity to collect, manage, and use program data more efficiently based on the instruments and methodology used in the study.

It is also reasonable to expect that study findings will inform similar efforts by other districts that are members of the Professional Learning Research Alliance (including the West Amwell Township School District, the South Hunterdon School District, the Monroe Township School District), as well as other stakeholders across the mid-Atlantic region which are not as far along as WCASD in planning and implementing PLCs or which are implementing but lack evidence-based information for adjusting their PLC implementation. The data will also inform efforts by state departments of education (starting with the Departments of Education in New Jersey and Pennsylvania, which are also members of the Professional Learning Research Alliance) to support the planning and implementation of PLCs statewide. REL MA has already prepared a review of the literature on online and hybrid PLCs in response to strong interest from research alliance members.

Finally, it is likely that the findings of this study will be used by other federal agencies and a national audience of practitioners and scholars who are interested in PLCs. The description of how and under which conditions (e.g., leadership style, group norms, available resources, and work arrangements) will shed light on positioning PLCs to achieve key desired attributes of collaborative work, as well as to meet standards of productivity that have the potential of enhancing student learning and achievement, and will narrow a conceptual and theoretical gap in the field.

• Use of Technology to Reduce Burden

Teacher data will be collected online through Survey Monkey. All teachers at WCASD have email and internet access through their district-supported accounts. Teachers will use these accounts to receive email communication about the study, access the surveys, and submit their electronic responses.

Data from the semi-structured interviews of principals cannot be collected through such methods as web surveys. Computer-assisted telephone interviews are not optimal for collecting these data. The proposed face-to-face interviews will be necessary to allow in-depth, conversational exchanges with respondents, and to obtain answers to both open-ended and detailed questions. However, if repeated attempts at scheduling these two-hour interviews during regular work hours fail, the REL MA study team will consider using telephone and/or computer-assisted telephone interviews (e.g., via Skype), providing that these can be audio-recorded for the purpose of transcribing the interviews.

• Efforts to Avoid Duplication

Prior to designing this study, REL MA produced a literature review on online and hybrid PLCs. REL MA has continued to search for relevant literature and to augment its literature base in the course of designing the study and gaining requisite approvals from OMB and the IRB at Rutgers University. Previous reviews of the literature on PLCs (e.g., Hord, 1997; Stoll & Louis, 2007; Wood, 2007) are based on findings from case studies. A recent review (Vescio et al., 2008) identified a mere 11 studies that examined the effectiveness of PLCs and observed that few studies have moved beyond teachers’ self-reports. REL MA’s ongoing review of the literature reveals that no in-depth study has been conducted of a single district’s implementation of PLCs that details the experience of teachers with qualitative and quantitative measures of attitudes and productivity, compares pre-implementation expectations with post-implementation experience, and provides in-depth analysis of differences among PLCs in similar contexts. Thus, the type of PLCs implementation evaluation proposed for this study has not been conducted to date.

• Methods to Minimize Burden on Small Entities

The study will utilize validated survey instruments and efficient interview protocols that focus on the data of most interest to the district and other audiences. WCASD will enable teachers to respond to the survey during periods that are already dedicated to school-wide and district-wide work. In terms of principal interviews, principals will be offered flexible scheduling to avoid potential conflicts or disruptions to their routine work.

• Consequences of Not Collecting Data

This is one-time collection. WCASD has no means other than the proposed data collection to track and assess the implementation of a major initiative that has the potential to deliver effective professional development for all teachers in the district and improve learning outcomes for all students in all of the district’s schools. The rich data collected through this project will generate valuable insights about the process of implementing PLCs in the district (including potential and perceived barriers to implementation) and will provide the baseline for assessing the impact of this program both in the short and long run. There has been no previous comprehensive study of PLCs in WCASD so there is no alternative source for the information to be collected. Moreover, the implementation of PLCs in WCASD presents a rare research opportunity (a natural experiment of a sort) to study in depth the dynamic nature of PLCs and the implementation process, thus making a significant contribution to current theory and evidence-based practice regarding PLCs and their impact on teachers’ professional development and students’ learning outcomes.

• Special Circumstances

There are no special circumstances associated with this data collection.

• Federal Register Announcement and Consultation

• Federal Register Announcement and Consultation

A 60-day notice to solicit public comments was published on October 31, 2013. There are no public comments received to date.

• Consultations Outside the Agency

The study team has worked with IES to identify the five members listed below to serve as the Technical Working Group (TWG) for this study.

Robert Boruch	University of Pennsylvania	P: 215-898 0409
Laura Hamilton	Senior Behavioral Scientist, RAND	P: 412-683-2300, x4403
Chris Hulleman	James Madison University	P: 540-568-2516
Andrew Porter	University of Pennsylvania	P: 215-898-7014
Christopher Rhoads	University of Connecticut	P: 860-486-3321

These experts on research methods, data analysis, implementation, and teaching and learning have provided input on the study’s design. Based on their input, the original emphasis on conducting process and outcomes evaluation of WCASD’s PLC initiative has been replaced with a focus on the process of implementation and the degree to which core elements of the program are implemented. This approach has informed the logic model of the evaluation as well as the overall research methodology which combines quantitative and qualitative data collection.

• Payments or Gifts

The study will not provide respondents with incentives for participation and we will not give payments or gifts to schools or to the district for providing access and study data. The superintendent of WCASD requested this study. The district will administer the pre-implementation online survey as well as the recruitment of all study participants overall. The superintendent has enlisted the support of principals in making time available for teachers to participate during the regular school hours; we plan to interview principals during regular school hours as well, unless they request otherwise. .

• Assurances of Confidentiality

The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant regulations and requirements, including the following:

• The Privacy Act of 1974, P.L. 93-579 (5 U.S.C. 552a).

• The Family Educational and Rights and Privacy Act (FERPA) (20 U.S.C. 1232g; 34 CFR Part 99).

• The Education Sciences Reform Act, P.L. 107-279 (20 U.S.C. 9573).

Information collected for this study comes under the confidentiality and data protection requirements of the Institute of Education Sciences (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Responses to this data collection will be used only for statistical purposes. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific district, school, or individual. We will not provide information that identifies you, your district or your school to anyone outside the study team, except as permitted by law.

REL MA (specifically, Rutgers University’s Center for Effective School Practices, a subcontractor to ICF International) will protect the confidentiality of information for the study and will use it for research purposes only. The project director, Dr. Blitz, will ensure that individually identifiable information about study members remains confidential. When reporting the results, data will be presented in aggregate form only so that individuals and schools will not be identified.

The following safeguards, which are routinely employed by Rutgers University’s Center for Effective School Practices to carry out confidentiality assurances, will be applied consistently during the study:

• A list of study team members will be forwarded to NCEE.

• All employees sign a confidentiality pledge (Appendix A), which describes both the importance of and the employee’s obligation to discretion. All study team members who will have access to study participants or data will sign the pledge and will obtain appropriate clearance.

• The plan for maintaining confidentiality also includes that all personnel with access to individual identifiers provide notarized nondisclosure affidavits and undergo training regarding (1) the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; (2) controlled and protected access to computer files under the control of a single database manager; (3) built-in safeguards concerning status monitoring and receipt control systems; and (4) a secured and operator-controlled, in-house computing facility.

• Procedures will be followed to revoke clearance in a timely fashion from members who leave the study team.

• REL MA staff have obtained clearance through NCEE’s security clearance office.

• Respondents’ personally identifiable information is maintained on separate forms and files, which are linked by sample identification number only.

• Access to hard copy documents is strictly limited. Documents are stored in locked files and cabinets, and discarded materials are shredded.

• Access to computer data files is protected by secure user names and passwords, which are available to specific users only.

REL MA will also include appropriate statements about protecting confidentiality in the requests for data collection from participants and other materials generated by the study team. Those documents include:

• Post-Implementation Interviews of Principals (see Part B, Appendix D)

• Post-Implementation Online Survey for Teachers (see Part B, Appendix E)

The following statements will appear in these documents to assure respondents about preserving confidentiality:

• “Responses to the data collection activities will be used for research purposes only. The reports prepared for the study will summarize findings across the sample and will not associate responses with a specific school or individual. We will not provide information that identifies you or your school to anyone outside the study team, except as required by law”.

• “The contractor follows the confidentiality and data protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). The contractor will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information on respondents will be linked to their institution but not to any individually identifiable information. No individually identifiable information will be maintained by the study team”.

• Justification for Sensitive Questions

There are no questions of a sensitive nature in the district interviews.

• Estimates of Burden Hours

Table 2 shows the estimated burden hours for principals and teachers who will participate in data collection. These estimates are based on REL MA’s experience collecting such data from principals and teachers for similar studies, and on pre-testing in other school districts. REL MA pre-tested the interview protocols with three principals and the online survey with four teachers from a school district in NJ represented in the PLRA. The average completion time for both online surveys was about 30 minutes. Annualized burden estimates over two years are 403 respondents (790 + 16 = 803 ÷ 2 = 403) and 214 minutes (395 + 32 = 427 ÷ 2 = 214).

Table 2. Estimated Response Time for Data Collection

Respondent/Data Request	Number of Targeted Respondents	Expected Response Rate (%)	Number of Respondents	Unit Response Time (Hours)	Total Response Time (Hours)
Principals
Post-implementation In-person interview	16	100%	16	2	32
Teachers
Post-implementation Online survey	930	85%	790	0.5	395
Annualized Totals			403		214

• Estimates of Cost Burden to Respondents

There are no capital or start-up costs for respondents. There will also be no additional operational, maintenance, or purchase of service costs for respondents. WCASD already has computers for staff use and has established email accounts and Internet access for all teachers and principals. The district also has included an annual account for Survey Monkey in its budget.

• Annualized Cost to the Federal Government

The estimated cost of the study to the federal government is $280,722 over two years, representing an average annual cost of $140,361. The estimated total cost of the REL MA five-year contract is $32,353,087, of which this study accounts for $280,722. Study costs were estimated using fully-burdened (“loaded”) hourly rates for staff, multiplied by the estimated number of hours of labor, and adding other direct costs. Multipliers for indirect rates were applied to direct costs.

• Reasons for Program Changes or Adjustments

This is a new data collection.

• Plans for Tabulation and Publication of Results

Table 3. Timeline for Data Collection and Analysis

	2014
	1	2	3	4	5	6	7	8	9	10	11	12
Administer post-implementation online survey of teachers
Analyze post-implementation online teacher survey data
Conduct Principal post-implementation interviews
Analyze of principal interviews
Draft report
Final report and WCASD debrief

• Tabulation Plans

REL MA will use the information gathered from the teacher survey and principal interviews to examine and evaluate district-wide implementation of a PLC initiative. Once the teacher survey dataset is ready for analysis, the study team will explore the distribution of individual responses on all variables of interest for irregular patterns (outliers, missing values, and deviation from normality for continuous variables) and make adjustments as needed. Next, the study team will compare responses across subgroups of teachers who teach at different levels (elementary, middle or high school), different content area, as well as teachers from different departments and schools, to identify potential sources of bias in our data. This is done for data diagnostic purposes and will not be included in public reports. As a rule, data will be reported at the aggregate to protect individuals’ identity.

Audio recordings of principal interviews will be transcribed and the analyzed using Dedoose. Special attention will be given to extracting insights into barriers and facilitators to implementation. The content of teachers’ responses to the open-ended questions in post-implementation online survey will be downloaded and then content analyzed. We will use Dedoose to identify and extract teachers’ reports of most positive and most negative PLC experiences, and to synthesize their recommendations for improving the implementation of PLCs.

Research Question 1 (how do PLC teams implement the study district’s charge?) will be answered primarily through analyzing the post-implementation survey data collected from teachers in Year 2 of the project. The survey will include several items (see Part B, Appendix E) that ask teachers to report retrospectively on aspects of the work routine of the PLC in which they are a member (e.g., degree to which the team meets regularly, level of active participation in the group, group dynamics, and leadership style within the group).

• Responses (on a Likert scale) provided by teachers who are members of a specific PLC team (as ascertained by the team’s unique study identifier) will be summed and averaged to produce team-level measures of these variables.

• These variables will be analyzed next using cluster analysis to develop taxonomies of work routines as they emerge from this exploratory procedure. The study team will determine the specific cluster analysis algorithm to be used once there is an opportunity to assess the specific properties of the data collected from teachers.

• Next, the study team will use findings from content analyzing teachers’ responses to the open-ended questions on the post-implementation online survey (where they describe the most positive and the most negative PLC experience they had and recommend possible improvements) to delineate common challenges/barriers that PLC teams encounter as members collaborate to establish work routines.

The data analysis approach to answering Research Question 2 (how do WCASD teachers and principals evaluate their experience with the program compared to their initial expectations) will also utilize teachers’ responses to the online survey, specifcaly the variables measuring expectations, attitudes, evaluations, and self-reported behaviors.

• Findings from teachers’ responses to the open-ended reflection questions on the post-implementation survey will also be used to answer this question, either by corroborating and/or clarifying themes that emerge from the teacher surveys or adding new insights that were not captured through these surveys. To answer this research question regarding the principals, the study team will analyze qualitative findings from the post-implementation interviews with principals.

• Audio recordings of the interviews will be transcribed verbatim and checked against the original audio recording for accuracy.

• Dedoose, a web-based application for managing, integrating, and analyzing qualitative and mixed methods data, will be used for data analysis. A grounded theory approach will be employed to explore a priori and emergent themes regarding principals’ evaluation of the program. The a priori themes will be defined according to the explicit themes explored in the interview (e.g., role of principal in implementation and allocation and use of school resources to support PLC work). It is anticipated that emergent themes will be focused on facilitators and barriers to implementation as well as on the impact, if any, PLCs had on teachers, students, and/or the school in the course of the first year of the program.

• An extensive list of codes and their definitions that relate to each theme will be derived from the interview data. Two independent coders will use the coding scheme to analyze transcripts of four randomly selected interviews and agreement among coders (intercoder reliability) will be assessed using Cohen’s Kappa. The coding scheme will be adjusted to eliminate disagreements among coders and improve overall reliability (Cohen’s k > .85).

• All codes will be entered into Dedoose and tagged to their associated segments of text for all interviews. Text segments will then be sorted by codes and reviewed by the coding team to identify emergent themes and to identify recurring patterns of responses to assess prevalence of themes.

• In the final step of the analysis, the study team will examine relationships among themes by looking at associations between differentially coded content.

Lastly, the study team will compare key findings about teachers’ perspective on the PLC initiative and its implementation to key findings that emerge from the analysis of the interview data obtained from principals. The study team will note similarities and differences in perspectives as a way of assessing possible changes to school culture post-implementation.

To answer Research Question 3 (specific products produced by PLC teams), the study team will analyze the survey data for teachers’ reports about artifacts produced by their team.

• Specifically, teachers will be asked to indicate the number of instructional units for which the team designed (1) essential learning targets, (2) core formative assessments, and (3) assessment rubrics. They will also be asked to indicate if they shared these artifacts with stakeholders (students, parents, and principal) and whether these artifacts were actually used with students in the course of the first year of implementation. The study team will also reference principals’ assessment of artifacts produced by PLC teams in their school based on analyzing the interview data collected from all principals in the district at the end of the first year of full implementation of the PLC initiative.

• Publication Plans

REL MA will produce a “What’s Happening” report of 15 to 30 pages at the conclusion of the project. The study team will present findings in complementary narrative and tabular formats, and will include user-friendly graphic depictions (i.e., charts and graphs) to highlight and contextualize descriptive findings. The study findings will be presented without reference to any names of individuals or specific schools, in order to avoid deductive disclosure of study respondents. The final report, scheduled for release in November, 2014, on the IES and REL MA websites, will be based on data from the pre-implementation and post-implementation teacher surveys and principal interviews. We anticipate that the “What’s Happening” report will include:

• A brief description of the study district’s PLC initiative and implementation objectives

• A discussion of the evaluation approach and logic model

• Presentation of key findings from the analysis of quantitative and qualitative data

• Discussion of study limitations, the appropriate interpretation of findings, and implications for current knowledge about PLCs, as well as for planning, implementation, and evaluation of PLCs

• A study methodology appendix.

REL MA will conduct a one-day debriefing to share the findings with the district’s leadership and to facilitate a discussion on the potential programmatic implications of these findings.

REL MA will also conduct a separate webinar to disseminate the study results with a broader audience and will use its dissemination network to alert regional stakeholders about the final report and webinar opportunities. REL MA will also work with members of the Professional Learning Research Alliance to identify ways in which they may use this report to guide their current and future professional learning activities. The study team will present this work at academic and professional conferences and at meetings involving Research Alliance members and other regional stakeholders. Finally, the study team anticipates producing several peer-reviewed journal articles based on this project, including a paper to present the logic model used in the project, a paper that will discuss the validation of the survey instrument used in this project, and a paper that introduces the overall methodological approach to the evaluation of the implementation, in addition to papers that describe and discuss the key findings of the evaluation.

• Approval Not to Display the Expiration Date for OMB Approval

The study will display the OMB expiration date.

• Exception to the Certification Statement

ED is not seeking an exception.

References

Baenen, N., & Jackl, A. (2010, June). Evaluation of central services professional learning teams as of spring 2010. Eye on Evaluation. E&R Research Report No. 10.06. Raleigh, NC: Wake County Public School System.

Delaware Department of Education. (2012). DDOE/Data coach project. Retrieved from www.doc.k12.de.us/tleu_files/PLCSurveyReport_2012.pdf.

Drummond, K., Chinen, M., Duncan, T.G., Miller, H.R., Fryer, L., Zmach, C., & Culp, K. (2011). Impact of the Thinking Reader® software program on grade 6 reading vocabulary, comprehension, strategies, and motivation (NCEE 2010-4035). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

DuFour, R., Eaker, R., & DuFour, R. (2005). Recurring themes of professional learning communities and the assumptions they challenge. In R. DuFour, R. Eaker, & R. DuFour (Eds.), On common ground: The power of professional learning communities (pp. 7–30). Bloomington, IN: National Educational Services.

DuFour, R., DuFour, R., Eaker, R., & Many, T. (2010). Learning by doing: A handbook for professional learning communities at work (2^d ed.). Bloomington, IN: Solution Tree.

Grossman, P., Wineburg, S., & Woolworth, S. (2000). What makes teacher community different from a gathering of teachers? Seattle, WA: Center for the Study of Teaching and Policy.

Hall, G., & Loucks, S. (1979). Implementing innovations in schools: A concerns-based approach. Austin, TX: Research and Development Center for Teacher Education.

Hemphill, J. (1956). Group dimensions: A manual for their measurement. Research Monograph No. 87. Columbus, OH: Ohio State University, Bureau of Business Research.

Heppen, J.B., Walters, K., Clements, M., Faria, A., Tobey, C., Sorensen, N., and Culp, K. (2012). Access to Algebra I: The Effects of Online Mathematics for Grade 8 Students. (NCEE 2012–4021). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Hord, S. M. (1997). Professional Learning Communities: Communities of Continuous Inquiry and Improvement. Austin, TX: Southwest Educational Development Lab.

Killion, J. (2006). Collaborative professional learning in school and beyond: A tool kit for New Jersey educators. New Jersey Department of Education Office of Academic and Professional Standards and the New Jersey Professional Teaching Standards Board in cooperation with the National Staff Development Council.

Little, J. (2003). Inside teacher communities: Representations of classroom practice. Teachers College Record, 105(6), 913–945.

Lomos, C., Hofman, R. H., & Bosker, R. J. (2011). Professional communities and student achievement: A meta-analysis. School Effectiveness and School Improvement, 22(2), 121–148.

Martin, T., Brasiel, S. J., Turner, H., & Wise, J.C. (2012). Effects of the Connected Mathematics Project 2 (CMP2) on the Mathematics Achievement of Grade 6 Students in the Mid-Atlantic Region (NCEE 2012-4017). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

National Staff Development Council. (2001). Standards for staff development. Oxford, OH: Author. Retrieved from http://www.nsdc.org/standards/learningcommunities.cfm

Nelson, M. C., Cordray, D. S., Hulleman, C. S., Darrow, C. L., & Sommer, E. C. (2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. Journal of Behavioral Health Services Research, 39(4): 374–96.

Office of Management and Budget (2006). Questions and answers when designing surveys for information collection. Washington, DC: Author.

Olivier, D.F. & Hipp, K.K. (2010). Demystifying professional learning communities: School leadership at its best. Lanham, MD: Rowman & Littlefield Education.

Stoll, L., Bolam, R., McMahon, A., Wallace, M., & Thomas, S. (2006). Professional learning communities: A review of the literature. Journal of Educational Change, 7, 221–258.

Stoll, L., & Louis, K. S. (Ed.) (2007). Professional learning communities: Divergence, depth and dilemmas. Berkshire, UK: Open University Press.

Talbert, J. E. (2010). Professional Learning Communities at the Crossroads: How Systems Hinder or Engender Change. In Hargreaves, A., et al. (eds.). The Second International Handbook of Educational Change, Springer International Handbooks of Education, Volume 23, Dordrecht, Springer, 555–571.

Vescio, V., Ross, D., & Adams, A. (2008). A review of research on the impact of professional learning communities on teaching practice and student learning. Teaching and Teacher Education, 24(1), 80–91.

Wijekumar, K., Hitchcock, J., Turner, H., Lei, PW., and Peck, K. (2009). A Multisite Cluster Randomized Trial of the Effects of CompassLearning Odyssey® Math on the Math Achievement of Selected Grade 4 Students in the Mid-Atlantic Region (NCEE 2009-4068). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.

Wood, D. R. (2007). Professional learning communities: Teachers, knowledge, and knowing. Theory Into Practice, 46(4), 281–290.

Appendix A: Confidentiality Pledge

I understand that the names, and any other identifying facts or information, of individuals, businesses, organizations, and families participating in projects conducted by the Center for Effective School Practices at Rutgers University or its subsidiaries are confidential information. I agree that I will not reveal such confidential information, regardless of how or where I acquired it, to any person unless such person has been authorized by the Director or Project Manager at Center for Effective School Practices at Rutgers University to have access to the information.

I further understand that the unauthorized access to, use, or disclosure of any confidential information is a breach of the terms of my employment, or my consultant agreement with Center for Effective School Practices at Rutgers University and may subject me to court action by any interested party or to other sanctions by Center for Effective School Practices at Rutgers University. I acknowledge that this agreement shall continue to bind me even after the project(s) is (are) completed and/or even though my employment or my consultant agreement with Center for Effective School Practices at Rutgers University has terminated.

In addition, in the course of my employment I may have access to personal information, electronic and otherwise, about fellow employees. I agree that I will treat that information as having the highest confidentiality, and not communicate it to fellow employees or others outside Center for Effective School Practices at Rutgers University. Final determination of whether or not there is a business purpose requiring that I access a fellow employees’ records will be made in consultation with the Director of Human Resources. Failure to uphold this standard is a breach of trust and may subject me to disciplinary action, including termination of employment.

Other than in the course of my authorized employment or my consultant agreement, I further agree that I will not use, nor facilitate the use by any third party, in any way any information deemed confidential by the terms of any contract or other written agreement between Center for Effective School Practices at Rutgers University and any other organization, except by written authorization by both parties. It is my understanding that Center for Effective School Practices at Rutgers University and the contracting organization(s) have the exclusive right to all information acquired or developed under such a contract or other written agreement. I acknowledge that I acquire no right, title, or interest in and to any data or information to which I have access by reason of my employment or my consultant agreement and that I may not remove such data from my assigned work location without prior authorization.

Name:	Signature:
	Date: