Response to OMB Comments

Responses to OMB Comments/Questions Regarding the Department of Education’s “Analysis of State Bullying Laws and Policies”

1. Revision/Clarification

In response to comments, ED revised the language on page 3 of A1 to read, “These factors indicate a need to document and assess how anti-bullying laws and policies are being implemented in a purposive sample of states, districts, and schools.”

2. Confidentiality

In response to concerns regarding use of the term “confidentiality”, language was revised throughout the Supporting Statement and appendices (i.e., state and school district recruitment letters, participant consent forms, and confidentiality statements preceding all protocols) to replace confidentiality assurances with recommended language stating that information identifying respondents will not be released to anyone outside of the study team except as required by law. In addition to changes throughout the Supporting Statement, the following changes were made to the recruitment letters (Appendix D) and participant consent forms (Appendix G):

• State and School District Recruitment Letters: The language in the state, district, and school recruitment letters was revised to read:

“Additionally, we want to assure you that to the extent provided by law, participation in the study will not violate the privacy of individuals who are interviewed, nor will it publicly identify participating schools, school districts, or government officials.”

• Participant Consent Forms: The language in the participant consent form was revised to read:

”We will protect your privacy by not releasing information that identifies you outside the study team except as required by law.”

3. School and District Recruitment Letters

As recommended, the language in district and school site recruitment letters was revised to include the following statement regarding the voluntary nature of the study: “Your state agency’s (school or district) participation in the research study is completely voluntary. There is no penalty for choosing not to participate.” The language of the recruitment letters was also revised to state that the “study is sponsored by the U.S. Department of Education and is being conducted through a contract with EMT Associates, Inc.” and references to PPSS were removed. The language indicating that the letter was being sent “on behalf” of the U.S. Department of Education was also replaced with language clarifying that the letter originated with Stuart Kerachsky.

The state agency recruitment letter to each Chief State School Officer was also reworded to clarify our request to interview a designee within the State Education Agency (SEA) who is knowledgeable about school bullying policy and program development within their state.

4. Description of Analysis Plan

The request for specification of how “comparisons” will be made is an important issue. This response and corresponding revision of the discussion of analysis in the Supporting Statement, Part A: Justification (16. Plans for Tabulation and Publication of Results) address this issue in several ways.

• The term “comparison” is broad and open to various interpretations, and revisions to the Supporting Statement are more explicit about how findings will be displayed and discussed within and across schools, districts, and states. Specific revisions to accomplish this include:

  • Purpose of Study

The purpose of the study stated in B1 has been changed from “compare bullying policy implementation across states, districts and school sites…” to “document and describe the relation of state bullying legislation and policy to district bullying policies and the implementation of bullying policies and practices in specific schools.” Corresponding changes in language are made throughout the statement.

• Selection of Sample

Language is added to the supporting statement, Part A: 16 to more clearly explain the kinds of similarities and differences the sample is designed to support. The study sample is not intended to provide a representative sample at the national, state, or district level. States are selected to incorporate differences in expansiveness in state laws and policies, and to incorporate geographic differences. Districts and schools are randomly selected within strata of urbanicity. In summary, the site sample is designed to provide insight into the implementation of bullying policy and practice in schools and districts across states with substantial differences in the detail and content of bullying legislation and model policies. Districts are selected to ensure the presence of schools in communities diverse in urbanicity.

• The revised statement clarifies that comparisons in this context are made to identify and summarize similarities and differences in the way a) districts and schools set and implement bullying policy in different state policy environments, and b) the ways that groups of respondents perceive bullying policy and practice implementation, including successes and challenges, in their districts and schools. Statements about similarity and difference will be descriptive and will not imply differences in quality or demonstrated effectiveness. Analysis will identify findings and patterns that warrant further consideration and assessment.

Specific revisions to the language in A16 of the Supporting Statement include the following:

“The implementation evaluation will proceed through a series of nested, exploratory analyses that will describe findings within schools and districts and within states, and will summarize findings across states. Information collected through interviews, surveys, and review of documents will be synthesized into a comprehensive set of summaries at the school site, district, within-state, and cross-state level.

• School Site Summaries

The individual respondent samples in each site are designed to support documentation and description of similarities and differences in the way that a broad range of school administrators, teachers and staff perceive and implement policy and practices relevant to bullying. As part of the data analysis process, the site visit team will complete a single site visit protocol that summarizes each of the major topics and questions included in the individual interviews. The site level summary also incorporates information derived from the interviews, from brief questionnaires administered at the time of the interviews, and from additional sources (e.g., from coded documents and records that will be requested during the process of organizing and carrying out the site visits). The following questions are examples of the kind of information that will be provided in school summaries.

• How well informed are teachers and other school staff concerning school policy and procedures concerning bullying? Do they understand their responsibilities within the policy? What, if any, are the differences in understanding across different teacher and staff groups?

• How much and what kind of training or guidance have teachers and other school staff received concerning school policy and procedures concerning bullying? What, if any, differences are reported concerning training across different teacher and staff groups?

• How often have teachers and other school staff taken action under the policies? What have been the successes and challenges they have experienced? What, if any, are the differences in experience across different teacher and staff groups?

• What kinds of bullying incidents are perceived as most prevalent and serious by teachers and other school staff? How does this relate to their exposure to different parts of the school environment? What, if any, differences in perception exist across different teacher and staff groups?

• School District Summaries

Additional brief summaries will be prepared at the district level. The data used in these summaries will include documentation (e.g., district policy and guidance documents, district records and data concerning bullying), an interview with a district representative, and school profiles. District summaries will focus on questions such as:

• What is district policy (using the key elements and criteria developed in the analysis of state legislation and model policies)? What guidance is given to schools regarding policy?

• How well do the school summaries in the district reflect district policy? What are the notable discrepancies if any?

• How similar are the school practices, including perceived successes and challenges, as represented in school summaries?

• Are there any clear differences in school environment (e.g., community type, student body, school characteristics such as size or leadership) that may be notable in understanding differences? If so, the potential issue will be explained.

• Within State Summaries

State summaries will be more extensive, focusing on the major patterns of similarity and difference in implementation across districts and schools. The data used in these summaries will include documentation (e.g., summaries and ratings of state policy in the state legislation and policy analysis, additional state documentation that may be provided, state data on bullying and bullying disciplinary actions, interview with an SEA representative, and district and school summaries. State summaries will focus on questions such as:

• What are the major features of state legislation and policy? Particular emphasis will be placed on issues of state and district interaction, e.g., how much discretion is given to districts in setting bullying policy? … to schools? What are procedural and reporting requirements? is district policy (using the key elements and criteria developed in the analysis of state legislation and model policies)? What guidance is given to districts and schools regarding policy?

• To what degree are components of state legislation reflected in school policy and implementation practices as reported by interviewees? Which components are challenging to districts / schools? Why? How much variation in the degree to which state policy components are reflected in policy and practice is there between districts? … between schools? What school characteristics are associated with these differences?

• Are there any clear differences in district or school environment (e.g., community type, student body, school characteristics such as size or leadership) that may be notable in understanding differences in any of the above areas? If so, the potential issue will be explained.

• Summaries Across States

  • A concluding analysis will review and summarize similarities and differences across states, including characterization of the degree of difference between school implementation within states and what policy or context issues may be associated with those differences. This cross-state summary will a) identify general patterns of guidance or challenge associated with policy attributes, b) identify other influences (opportunities or challenges) that characterize school level policy and practice (e.g., district level guidance and support), and c) identify issues that are important for further monitoring, assessment, and recommendations.”

5. Universe and Sampling Issues

The following items are in response to issues raised regarding the universe of states, school districts and school sites and the proposed sampling design.

1. Inclusion of the District of Columbia and Puerto Rico: The universe of states does not include the District of Columbia or Puerto Rico. As noted in the Analysis of Bullying Laws and Policies report, “The District of Columbia and the U.S. territories were excluded from the analysis because their governance structures (i.e., legislative bodies and administrative agencies) are different from those in states.” Since the states were chosen based on that analysis in the report, DC and PR were not included in the universe.

2. Report Citation: As the report is not yet released, we cannot cite the report. The language in Supporting Statement B on page 2 of B1 was revised to explain that it is a forthcoming report.

3. Sampling of States: The sampling of the four states is purposeful, based on findings from the forthcoming report and diversity with respect to their legislative and policy environments addressing bullying.

4. Sampling of LEAs and School Sites: The sampling of the LEAs and school sites within LEAs is random, within the rural/town, suburban, and urban stratification.

6. Division of Labor between Principals and Vice-Principals

School administrators are the key informants on bullying policy within each school setting. As such, the protocols used to interview administrative personnel cover a broad range of issues concerning policy development, programming, procedural guidelines, and perceptions regarding implementation challenges and impacts of bullying and bullying policy on the school setting. Given the volume of the material, the interview content was divided into two separate protocols in the interest of minimizing burden on any one administrator and minimizing redundancy when attempting to capture more factual interview content (e.g., descriptions of procedures, use of training curriculum, etc.). Although the two versions of the administrator interview have several shared items that are common to all school personnel protocols, they are also specialized according to content area. Specifically, one version focuses largely on issues related to student conduct and one version focuses on policy and program development.

The division of interview items was based on an assumption that vice-principals are traditionally responsible for managing issues of student discipline and policy enforcement, and would be most knowledgeable for responding to questions regarding student conduct. We do recognize, however, that in some school settings the actual division of responsibility between a principal and vice-principal may differ from this assumption, and/or that a school site may have only one administrator, or may have multiple administrators assigned to different functions. Accordingly, as part of our initial recruitment and scheduling process, EMT staff will work with a school representative prior to interview scheduling to identify the most appropriate respondent(s) to complete each protocol by sharing general information about the interview content and the nature of items. The school will be asked to recommend an appropriate school administrator for each interview.

In the event that a school has only one school administrator, that individual will be asked to respond to items on both protocols, but will be given the option of completing the second set of interview items by phone at the administrator’s first convenience. In the event that an administrator indicates at the time of the interview that they would like to defer certain questions to other administrative personnel, the sub-set of questions identified may be completed by telephone with another respondent as follow-up to the site visit.

7. Advantages of Conducting On-Site Interview

Proposed data collection for the implementation of bullying legislation and policy is conducted primarily through face-to-face interviews conducted on-site. On-site interviews conducted through team site visits are preferred for the following reasons:

• Site visits are planned to include a relatively large number of interviews in each site. Interviews will be within groups of administrators, teachers and staff who may frequently interact. In this circumstance the two-person, 1.5 day interview format has logistic advantages that can also impact data quality. Interviews will be formally scheduled ahead of time with the study and school liaisons agreeing on scheduling and ensuring that selected respondents will be available. This will support an efficient completion of interviews within a short window of time, will minimize contamination of responses due to inter-respondent discussion, and will minimize potential contamination by events. Interviewing by telephone would be much more difficult to achieve in a brief window.

• This is an exploratory study, and presence on-site will have advantages in doing careful and rich data collection, and inductive interpretation of data. For example, field researchers have the ability to observe the physical and social context of the schools and to observe interactions of the persons being studied. Specifically:

• There is little existing evaluation or research concerning the implementation of bullying policies, and there is less systematic research across multiple implementation sites (i.e., districts and schools). It follows that interview questions should be relatively open, soliciting the perceptions and insights of respondents in different environments. The study interviews have been developed to provide site visitors with guidance for probes that will achieve uniform response opportunities at all sites. On-site interviews will significantly enhance the ability of interviewers to understand context through observing the schools physical condition (e.g., physical design characteristics that may promote or inhibit bullying) and any visible signs of policy related to bullying (e.g., posters, public displays of policy). This understanding of context enriches probes and promotes an interviewer-interviewee rapport that motivates interviewees to volunteer perceptions and insights.

• The school sites are being selected through a purposive sample that uses sample stratification by urbanicity to increase the planned diversity in the sample. These differences in urbanicity are expected to relate to differences in resources, socio-economic background, and school climate (e.g., size, group heterogeneity) that may impact bullying behavior and the implementation of policy. Observing the setting provides an enhanced understanding of the degree to which specific schools reflect these expected characteristics associated with urbanicity, and to identify the presence of particularly important school community features that may improve understanding of bullying behavior or policy implementation opportunities and challenges.

• The site visit team format is important to our methodology. Team members work together to complete the structured site-level summary protocols that incorporate information from site documents and interviews. The process, on which team members will be thoroughly trained, involves discussion of interpretation and judgments. The presence of both team members on-site and their ability to discuss emerging findings and refine probing questions is important to improving the quality of site summary protocols.

• Being on-site will also allow the team to confirm the completeness of disciplinary records; incident reports; policy statements; communications to students, families and community; and other documents that will be requested, collected and reviewed during the preparation period for the site visits.

8./9. Structure of the Interview Proposals

We agree with the reviewers’ recommendation that interview protocols should be divided into two instruments—a brief survey containing closed-ended, Likert-scaled items (e.g., How frequently do you respond to incidents of bullying on your school campus?), more typical of a survey questionnaire, and an interview protocol containing open-ended items consistent with a semi-structured interview. Accordingly, a new brief survey form (Appendix S) has been developed that will be administered to the interview respondent prior to initiating the interview. For the school district interview, which will be completed by phone, the brief survey questionnaire will be administered as an electronic form that will be e-mailed to the respondent. We will request that the electronic survey form be completed and returned to EMT prior to the scheduled interview date.

All closed-ended items have now been removed from the semi-structured interview protocol and included as Likert-items on the new survey form, or have been revised as open-ended questions. For the remaining interview items, checklists or categories of potential responses will be retained. These categories do not represent fixed-responses, but rather are non-exhaustive lists of potential responses that are used to categorize open-ended elaboration and serve as interviewer prompts.

10. Introductory Consent Statement on Protocols

As recommended, the language in the introductory consent statement for each protocol was shortened and revised to read:

“We will not share information identifying you or your district with anyone outside of the study team except if required by law. No one in your district administration will be told of your responses to interview questions and your responses will not be tied to any review or evaluation of your professional performance. Responses will be summarized in individual case studies for schools and districts to provide examples of policy implementation in a manner that does not associate responses with a specific individual and does not identify a school or district by name.”

The OMB PRA statement was removed from each protocol.

11. Specialization to Personnel Position

As bullying polices often have the same expectations of all school staff (i.e., teachers, special education teachers, bus drivers, yard supervisors), we think it is important that all staff answer similar questions on communication, training, reporting, and investigations. The use of common interview items across interview respondents also allows us to gauge the degree of consistency in policy implementation (e.g., how bullying is defined, under what circumstances bullying incidents are reported) across school personnel.

We think that the principal, vice principal, school counselor, and school resource officer have unique roles to play in implementing bullying policies and their protocols are more unique.

Bullying policies are written to cover all students, and while special education students may be named as a special class, we think it is appropriate that special education teachers answer many of the same questions as other teachers.

In response to OMB comments, PPSS requested a review of the special education protocol by an individual from the Special Education office. The following changes were incorporated into the special education protocol to reflect their recommendations:

• Item 4e previously read “How well does your school bullying policy provide protections for students with special needs?” and currently reads “How well does your school bullying policy provide protections for students with disabilities?”

• Item 5f previously read “Does the training curriculum address special concerns related to bullying for students with special needs?” and currently reads “Does the training curriculum address special concerns related to bullying for students with disabilities?”

• Item 8d previously read “Are there any special modifications made for students with special needs to facilitate reporting or investigation of bullying incidents? “ and currently reads “Are there any special modifications made for students with disabilities to facilitate reporting or investigation of bullying incidents? ”

6