Memo to OMB on SLEPS Cognitive Testing

MEMORANDUM

MEMORANDUM TO: Jennifer Park

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

Michael Planty

Deputy Director

Acting Chief, Law Enforcement Statistics Unit

FROM: Elizabeth Davis

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: August 29, 2016

SUBJECT: BJS request for OMB Clearance to conduct expanded cognitive interviewing for the Survey of Law Enforcement Personnel in Schools (SLEPS), under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) plans to conduct a new data collection to gain a better understanding of the roles, functions, and supporting infrastructure of police officers in schools through a project titled the Survey of Law Enforcement Personnel in Schools (SLEPS). The SLEPS has two primary goals, the first is to identify a national roster of active law enforcement agencies that have law enforcement personnel operating in some capacity in U.S. K-12 schools. The second primary goal is to generate detailed, accurate, and reliable national statistics describing the scope, size, characteristics, and functions of law enforcement personnel that work and interact in a school environment. To accomplish these goals, the SLEPS has a two-phased approach that will survey both law enforcement agencies and officers who work in schools.

This new data collection is a component of a larger school safety agenda. As outlined in the Department of Justice Appropriations Act, 2014 (Public Law 113-76), Congress tasked the National Institute of Justice (NIJ) to undertake the Comprehensive School Safety Initiative (CSSI), a research-focused program to increase the safety of schools nationwide. The CSSI identified a number of school safety-related topics for which more extensive data and research are required; one such topic is the presence of law enforcement in schools. There have been isolated local efforts to empirically examine law enforcement involvement in schools; however, no current comprehensive national-level data exist on the extent of law enforcement involvement in the nation’s schools or on their typical roles and responsibilities. To address this lack of data, NIJ entered into an interagency agreement with BJS, seeking to improve the amount of information pertaining to the roles, responsibilities, and actions of local law enforcement in schools.

Currently, there are limited data available regarding the presence of law enforcement in schools. BJS has collected the number of school resource officers (SROs) or other sworn personnel whose primary duties are related to school safety through the Census of State and Local Law Enforcement Agencies (CSLLEA) and the Law Enforcement Management and Administrative Statistics (LEMAS) Survey. Data are collected about every 4 years for the CSLLEA and about every 3 years for the LEMAS; the question asking about the number of SROs, however, was not included in the most recent administration of the LEMAS (2013). The periodicity of the CSLLEA and the LEMAS along with the narrow scope of the school safety question on these questionnaires limit the utility of these data and provide no insight on the roles of officers in schools or on the infrastructure in place to support these officers.

The National Center for Education Statistics (NCES) conducts the School Survey on Crime and Safety (SSOCS), which surveys principals at a nationally representative sample of over 3,000 public schools. SSOCS asks principals about the presence and activities of school security staff, including sworn law enforcement personnel. The most recent data available cover the 2009-2010 school year and contain only high level information on the school’s security staff as a whole. SSOCS was fielded again in the spring of 2016 with more detailed school security questions, including a distinction between types of security staff and their roles in the school. While there is a level of overlap between the SSOCS and SLEPS data collections, this overlap is not duplicative, but rather complementary. The SSOCS does not provide the information the SLEPS project seeks to gain regarding the number of law enforcement agencies with officers working in schools and the infrastructure within these agencies to support school safety. Additionally, the data collected by the SSOCS on officers working in schools consider only a narrow view of officer roles and responsibilities and do not explore officer characteristics such as training and experience.

To address the need for national-level data on the prevalence and roles of law enforcement in schools, BJS has contracted with RTI International to develop and test the SLEPS. RTI has awarded a subcontract to the Police Executive Research Forum (PERF) to assist with the SLEPS, given PERF’s experience with conducting officer-level surveys and their existing relationships with law enforcement agencies.

The current request is to conduct expanded cognitive testing for the SLEPS under the generic clearance agreement (OMB Number 1121-0339). Project staff from RTI International and PERF will cognitively test (1) a SLEPS law enforcement agency (LEA) survey (Attachment A), which includes a school resource officer rostering form (Attachment B), and (2) a SLEPS school resource officer (SRO) survey (Attachment C) to ensure question clarity and effectiveness.

The expanded cognitive interviewing plan described below is designed to (1) thoroughly test the survey instruments to ensure question clarity and reliability and (2) assess the willingness of law enforcement agencies to provide rosters of their officers working in schools. The project staff will review the results of the cognitive interviews and agencies’ willingness to roster their officers to determine if a small field test of the LEA and SRO surveys should be employed to test the planned data protocol. If a field test is deemed necessary, BJS will submit a request to conduct a field test of the SLEPS surveys and data collection protocol under the generic clearance agreement, tentatively in December 2016. Following the completion of testing, BJS plans to implement the full-scale law enforcement agency portion of the SLEPS by January 2018 and the full-scale school resource officer portion by May 2018.

Request for Expanded Cognitive Interviewing

BJS plans to conduct expanded cognitive testing of the two instruments, agency and officer, to ensure that the content and language of the survey questions are clear and the order of questions is appropriate. The cognitive testing of the agency survey will include a review of the officer roster form, seeking feedback on the willingness of agencies to provide a list of school resource officers. Twenty law enforcement agencies will be selected to participate in cognitive interviews for the agency survey and roster form and 20 officers will be selected to participate in cognitive interviews for the officer survey. The intended respondent for the law enforcement agency survey is the agency’s chief or another staff member who is knowledgeable about the infrastructure supporting the officers who work in schools, such as the supervising officer.

The project team will develop a preliminary list of agencies for cognitive testing based on a few select criteria to maximize the potential spread of agency characteristics. After including agencies PERF previously identified as leaders in SRO programs, the project team will randomly select additional agencies based on the following four stratification categories: (1) agency size, (2) agency urbanicity, (3) recency of the agency’s SRO program, and (4) agency type. The recency of the SRO program will be based upon whether the agency had any SROs according to the 2008 CSLLEA. The project team is confident that the resulting list of agencies prioritized for cognitive interviewing will be diverse in characteristics and will therefore provide a sampling of potential obstacles to the proposed rostering approach.

For officer recruitment, the project team will identify a list of agencies from which to recruit SROs. Using a point of contact within the agency, PERF will ask command to distribute a description of the project along with PERF contact information to all SROs so that individual officers can self-select by reaching out to PERF voluntarily. The project team will accept up to 2 SROs per agency. The project team will use an iterative contacting approach that will start with a minimum of 13 agencies and may reach as many as 26 agencies, depending on the response received from the initial 13 agencies. The initial 13 agencies will be identified through a 2-stage process. First, six agencies known by PERF as having a ‘successful’ SRO program identified through outside sources will be included. The project team will then use a randomly assigned pool to select from the remainder of the listings across the above-mentioned four stratification categories: (1) agency size, (2) agency urbanicity, (3) recency of the agency’s SRO program, and (4) agency type. If agencies or officers decline to participate, the project team will select additional agencies depending on the needed characteristic from the randomized pool.

During the full fielding of the survey, the project team will sample officers from the sampled agencies. The agency survey will request a roster of officers that work in schools and from these rosters, the project team will select a sample of officers to receive the SRO survey. During cognitive testing, however, the officers and law enforcement agencies will be entirely independent of each other. The project team will not recruit officers from the agencies that are participating in the cognitive testing. RTI’s Institutional Review Board (IRB) imposes a higher burden on the cognitive interviewing process due to the qualitative nature of the data collected during the interviews, where the project team is probing participants and soliciting opinions, asking officers for information that they will not be asked to provide during the actual survey. Since the purpose of the cognitive interviews is only to evaluate the questions planned for the survey instruments, they do not need to be conducted through a recruitment process that mirrors the sampling process that will be used during the full fielding.

The recruitment of agencies and officers for cognitive testing and the subsequent cognitive interviews will be conducted over the phone. Upon selection for cognitive testing, the project team will contact the selected participants via phone for recruitment and scheduling. The participants will be asked to provide an email address so that the project team can email participants a consent form (Attachments D and E) for their review and a copy of their respective survey. At the time of the scheduled interview, the interviewer will call the participant and review the consent form with the participant. Upon receiving the participant’s consent, the interviewer will complete the questions at the bottom of the interviewer’s version of the consent form, acknowledging that the interviewer discussed consent with the participant and the participant provided consent to participate in the interview and to have the interview recorded (Attachments F and G). Participants will not receive any compensation for the interview.

The law enforcement agency (LEA) survey topics include the practices, policies, and procedures that agencies use to oversee and support officers working in schools; the number of personnel who work in schools; training, common roles and responsibilities of officers in schools as defined by agency policies and agreements with schools; and the types of officer activities monitored by the agency. Topics covered on the school resource officer (SRO) survey include the extent of the officer’s law enforcement experience, the officer’s activities in schools on a monthly basis, the officer’s role in and experience with school discipline, officer activities that may influence school climate and culture, and the extent of any mentoring or teaching roles of the officer. These surveys were developed to collect data leading to the production of detailed information on the characteristics of law enforcement in schools, including a number of key estimates, such as the number of various types of officers working in schools, the number of schools served by law enforcement agencies, years of officer experience, frequency of training, and the frequency with which officers perform various activities in their assigned schools.

Concepts and topics for both the agency and officer questionnaires were developed and prioritized through ongoing discussions between the SLEPS project team, NIJ, practitioners, researchers, and other key stakeholders. The project team also convened an Expert Working Group (EWG) to solicit input and feedback on topics of interest and utility, following up with a subset of the EWG to review early drafts of both questionnaires. The EWG was designed to include individuals who would provide the perspective of data providers and consumers of the information that will be gathered by the SLEPS. The group was comprised of officers that work in schools, those who supervise officers who work in schools, a representative from the National Association of School Resource Officers (NASRO), experts from the academic field, and representatives from various federal agencies with a vested interest in school safety topics.

BJS requests approval to conduct expanded cognitive testing under the BJS generic clearance to include more rigorous and thorough testing of the SLEPS questionnaires. This testing is needed to ensure that the form and content of the instruments are understandable and appropriate across the range of targeted respondents. For both the agency and officer surveys, the project team developed structured protocols for cognitive testing (Attachments H and I), providing specific probing questions to help determine that the questions are interpreted as intended and are thereby collecting the targeted data. Participants will be asked to take note of any aspects of the instruments that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary.

Attached to the agency survey is an officer roster form. This form requests the agency point of contact to provide a list of officers working in schools which the project team will use to draw the sample of officers for the officer survey. The SLEPS will be BJS’ first attempt at an officer survey and other projects have encountered challenges in their efforts to obtain a list of officers. For these reasons, BJS would like to use the cognitive testing effort to gauge agencies’ willingness to provide a list of school resource officers to help the project team determine the best method to obtain the information needed to draw a sample of officers. To facilitate the collection of this information, included in the attached cognitive interviewing protocol for the agency survey (Attachment H) are scripted probing questions designed to inform the project team about the officer information that agencies are willing to provide and the mode(s) through which they are willing to provide it. The participant will be asked to review the roster form and to provide feedback, but the participant will not be asked to actually complete the form with officer information. The participant will also be asked if their agency would have a need or an interest in viewing the SRO survey prior to either providing the roster of officers or distributing surveys to officers.

Expanded cognitive testing will provide an opportunity for BJS to calculate more reliable burden estimates for law enforcement agencies and school resource officers to complete their respective questionnaires. This information will be taken into consideration as BJS continues the development and design of the SLEPS. BJS will also evaluate the option of conducting a field test following review of the results of the cognitive testing, specifically the willingness of agencies to provide officer rosters. Should BJS determine that a field test of survey protocols should follow cognitive testing, the information gathered during the cognitive interviewing and the revised questionnaires will be integrated into a generic clearance request to conduct a small field test, tentatively in December 2016. All information gathered from the SLEPS testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB by July 2017. BJS plans to fully implement the agency portion of the SLEPS by January 2018 and the officer portion by May 2018. All of the dates noted above are planned and pending final approval.

Burden Hours

The burden hour estimates are provided in the following table.

		Average burden hours per respondent	Total respondents	Est. burden hours
Agency level cognitive interviewing (including roster form)
	Initial contact and scheduling	0.10	24	2.40
	Complete cognitive interview	1.00	20	20.00
Officer level cognitive interviewing
	Initial contact and scheduling	0.10	24	2.40
	Complete cognitive interview	1.00	20	20.00
TOTAL HOURS				44.80

Institutional Review Board

The project team obtained approval from RTI’s IRB to ensure the cognitive testing protocol is compliant with informed consent and data confidentiality standards.

Contact Information

Questions regarding any aspect of this project can be directed to:

Elizabeth Davis

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 305-2667

E-mail: Elizabeth.Davis@ojp.usdoj.gov

Attachments

Attachment A: Law enforcement agency (LEA) survey

Attachment B: Officer rostering form

Attachment C: School resource officer (SRO) survey

Attachment D: LEA participant informed consent

Attachment E: SRO participant informed consent

Attachment F: LEA interviewer informed consent

Attachment G: SRO interviewer informed consent

Attachment H: Law enforcement agency (LEA) survey cognitive interviewing protocol

Attachment I: School resource officer (SRO) survey cognitive interviewing protocol

5