Memo to OMB on Cognitive Interviewing for LEMAS 2023 Supplemental Survey

MEMORANDUM

TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Doris J. James

Acting Director

Bureau of Justice Statistics

Kevin M. Scott

Deputy Director, Statistical Operations

Bureau of Justice Statistics

Alexia D. Cooper

Chief, Law Enforcement Statistics Unit

Bureau of Justice Statistics

FROM: Elizabeth Davis

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: April 19, 2022

SUBJECT: BJS request to conduct cognitive interviewing for the Law Enforcement Management and Administrative Statistics (LEMAS) supplement survey under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) is planning to collect data under the 2023 Law Enforcement Management and Administrative Statistics (LEMAS) supplement (previous OMB Control Number 1121-0354, expired 2/28/2019). The LEMAS supplement survey was last fielded in 2016, with a focus on body-worn camera use among law enforcement agencies. The 2023 LEMAS supplement is part of BJS’s Law Enforcement Core Statistics (LECS) program, for which BJS has a multi-year cooperative agreement with RTI International (RTI) and subawards with the Police Executive Research Forum (PERF) and the National Policing Institute (NPI). The LECS program streamlines BJS’s major law enforcement data collections, which includes the LEMAS core and Census of State and Local Law Enforcement Agencies (CSLLEA), into a common rotating schedule to reduce respondent burden and increase the timeliness of data collection.

The goal of the 2023 LEMAS supplement is to obtain information from law enforcement agencies on two topics: post-academy law enforcement training and agency responses to police suicide. A primary goal of the requested cognitive interview process is to assess whether these two related but distinct topics can be fielded together seamlessly. If so, both topics will comprise the 2023 LEMAS supplement. If not, the two instruments will be separated with the post-academy questions moving forward alone as the 2023 LEMAS supplement.

Questions on post-academy training were developed following a BJS expert panel meeting on the topic. Working with the RTI project team, BJS convened a group of nearly 30 federal, academic, and practitioner experts for a five-hour virtual meeting on September 29, 2021. The goal of the expert panel meeting was to discuss in-service training. The panel focused on three fundamental issues: what we already know about law enforcement in-service training; what we need to know about law enforcement in-service training; and how best to gather information about in-service training from law enforcement agencies. One takeaway was the perceived need to ask about law enforcement training more broadly than originally designed. Though the meeting itself was labeled as a discussion of “in-service” training, panelists immediately noted that post-academy training is a larger topic worthy of full exploration. Additionally, panelists expressed a desire to have national data on the frequency of specific training topics (both the count of agencies adopting a training topic and number of hours per training), including detailed information on a subset of high-priority topics. The expert panel meeting informed the development of the survey instrument draft (Attachment A).

Questions on agency responses to police suicide were developed in furtherance of the potential BJS role in augmenting the Federal Bureau of Investigation’s (FBI) Law Enforcement Suicide Data Collection (OMB Control Number 1110-0082). The goal is for this proposed instrument to complement the FBI’s work by collecting baseline, nationally representative measures on what programs and polices agencies have in place to address suicide risk. The instrument (Attachment B) was developed independently of the post-academy expert panel but with the same project team making contributions to drafting and revising the instrument. Questions were developed using three primary sources. First, the project team consulted the FBI data collection instrument to ensure this LEMAS supplement instrument does not overlap with the FBI effort. Second, the general scope and specific content of questions were based on an extensive literature review focused on the measurement of agency responses to police suicide. The project team reviewed more than 30 resources, including recent federally-funded publications and academic articles, to ensure questions accurately reflect recent scholarship and could contribute to the field. Third, the project team used prior LEMAS surveys to standardize answer choices across BJS collections, when possible.

Request for Cognitive Interviewing

This request is to conduct cognitive testing for the 2023 LEMAS supplement instrument under BJS’s generic clearance agreement (OMB Control Number 1121-0339). BJS has contracted with RTI to develop and test the instruments. Project staff from RTI will be responsible for administering the cognitive testing of the 2023 LEMAS supplement instruments (see Attachment C for a cognitive interview protocol).

The cognitive interviewing plan (described below) is designed to (1) test the survey instruments to ensure question clarity and reliability, (2) assess the capability of law enforcement agencies’ ability to identify a proper respondent for each questionnaire topic and to provide the requested information, and (3) determine the feasibility of including questions on post-academy training and police suicide on the same survey instrument. The project staff will review the results of the cognitive interviews and modify the survey instruments as necessary. Following the completion of testing, the BJS plans to implement a LEMAS supplement survey in September 2023.

The sample selection process will classify law enforcement agencies across two strata: type of law enforcement agency (local police department, sheriff’s office, or primary state police) and agency size (small agencies with less than 100 full-time equivalent sworn personnel¹ or large agencies with 100 or more full-time equivalent sworn personnel). These two strata match those used for selection in prior LEMAS core and supplement samples. Primary state police agencies are the smallest number nationally (N=49) and all such agencies are classified as large.

Forty (40) law enforcement agencies will be selected to participate in cognitive interviews: 19 local police departments,19 sheriffs’ offices, and 2 primary state police. The breakout of desired respondents based on agency type and size is shown in Table 1.

Table 1. Number of completed cognitive interviews by agency type and size

Full-Time Equivalent Sworn count	Local	Sheriff	Primary State	Total
Large (100+)	9	9	2	20
Small (1-99)	10	10	0	20
Total	20	20	2	40

Using the strata as a framework, the project team will use a multi-stage approach to populate the sample. First, selection will reference a list of agencies that have expressed a willingness to participate in future BJS research and development activities. This list is a product of the 2020 LEMAS core survey collection, where agencies were asked to self-select their willingness to be contacted for future activities. Since the 2020 LEMAS core data collection took place from January to September 2021, the list is a likely to be up to date and still reflect agency preferences at this time. From these agencies, the project team will select 20 large and 20 small local police departments and 13 large and 9 small sheriff’s offices.

Second, the project team will select from the Law Enforcement Agency Roster (LEAR), a master listing of agencies in the United States which is maintained by the project team as part of ongoing LECS program work. The LEAR has agency type, size, and contact information as key variables in the dataset. Selection from LEAR will be used concurrently with the first stage to complete the initial sample of sheriffs’ offices by adding 7 large and 11 small offices to the sample.

Third, at the end of each sheriff’s office cognitive interview, the project team will ask each participant if they would be able to suggest other potential sheriff’s offices to contact. Prior LEMAS core and supplement surveys identify sheriffs’ offices as more difficult to obtain an initial response as compared to local police departments. The project team intends for this snowball sampling approach to clarify offices with a potential willingness to participate.

Finally, the LEAR will be the source for identifying and contacting replacement agencies once the prior stages are exhausted. Agencies will be grouped by strata and selected at random. The LEAR was the starting frame for all BJS law enforcement surveys for nearly a decade. As such, any selection will be straightforward and based off a population frame.

As BJS anticipates that some of the sampled agencies will be unable to participate in the cognitive interview effort, a substitute agency will be selected for each of the sampled agencies and contacted only if a replacement is needed. Up to 125 total agencies will be contacted, though completed interviews will be capped at 40 total agencies. Prior experience during the 2020 LEMAS core cognitive interview process had 95 agency contacts result in approximately 30 interviews, leading to the current requested count of 125 outreach agencies.

The intended respondent for the survey is the agency’s chief executive or another staff member who is knowledgeable about the law enforcement personnel and policies. Based on past experience with cognitive interviewing during development of the 2016 LEMAS supplement (OMB Control Number 1121-0354, expired 2/28/2019), 2016 and 2020 LEMAS core surveys (OMB Control Number 1121-0240, expired 6/30/2019 and expiring 7/31/2023, respectively), 2018 CSLLEA (OMB Control Number 1121-0346, expired 4/30/2021), and 2018 Census of Law Enforcement Training Academies (CLETA; OMB Control Number 1121-0255, expired 11/30/2021), the project team is confident that the resulting list of agencies prioritized for cognitive interviewing will be diverse in characteristics and will provide a sampling of potential obstacles to the proposed information collection.

Initial contact with 40 agencies will be conducted by email and led by RTI. The message will request participation and include an invitation letter similar the 2020 LEMAS core cognitive interviewing effort (Attachment D). Non-responders will be contacted within 3-5 business days of initial outreach by resending the invitation email (Attachment D) and then by phone (Attachment E). When an agency is recruited, participants will be asked to confirm or provide a new point of contact (POC) and their email address. If an agency does not choose to participate, a replacement from the same stratum will be selected and outreach will be made to recruit the new agency. The replacement process will be repeated until the required number of respondents in each stratum is obtained.

Within one week of agency recruitment, a survey package will be sent to the respondent either by mail or email. The packet will include a cover letter explaining the purpose of the test (Attachment F), a copy of the draft questionnaires (Attachments A and B), a postage paid return envelope if requested (to send by mail), and an RTI email address (to send by email). Respondents will be asked to complete the questionnaires within one week, make a copy for their reference and return the original to RTI. They will also be asked to record the time spent by all agency staff to complete the form; this information will be captured on the questionnaires.

Concurrently with distribution of materials, RTI will schedule POCs for a 60-minute cognitive interview call, either by phone or virtual platform. During this call, a member of the project team will conduct a retrospective cognitive interview using a standardized interview guide (Attachment C). The POC and the interviewer will both have access to the completed instrument to use as a reference while the interviewer follows the cognitive testing protocol to complete the interview. Participants will be asked to take note of any aspects of the instrument that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. Participants will not receive any compensation for the interview. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary.

Cognitive testing will also provide an opportunity for BJS to calculate reliable burden estimates for law enforcement agencies to complete the questionnaires. This information will be taken into consideration as the BJS continues the development and design of the 2023 LEMAS supplement. All information gathered from the LEMAS testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB by February 2023. BJS plans to fully implement the LEMAS supplement beginning in September 2023. All of the dates noted above are planned and pending final approval.

Burden Hours

The burden hour estimates are provided in the following table. BJS has estimated the respondent burden for the proposed questionnaire development effort at 107 hours (Table 2). The burden is comprised of three components: initial contact and scheduling, completion of the paper form and completion of the cognitive interview. Up to 125 law enforcement agencies may be contacted and asked to participate in cognitive interviewing. However, a maximum of 40 agencies (19 local police departments, 19 sheriffs’ offices, and 2 primary state police) will complete and submit the draft 2023 LEMAS supplement survey in order to (1) obtain tested estimate of burden for the national implementation and (2) provide real data that can be assessed prior to the cognitive interview for quality issues and to serve as the basis for discussion during the interview. The estimate was calculated using 1.5 hours per respondent for the completion of the draft surveys and 1 hour per respondent for the cognitive interview.

Table 2. Estimated burden hours for cognitive interviews.

	Average burden hours per respondent	Total maximum respondents	Est. burden hours
Initial contact	0.03	125	3.8
Scheduling of interview	0.08	40	3.2
Complete questionnaire	1.50	40	60.0
Complete cognitive interview	1.00	40	40.0
TOTAL HOURS			107.0

Institutional Review Board

RTI’s IRB determined that the research conducted under the LECS contract does not constitute research involving human subjects as defined by the U.S. Code of Federal Regulations (45 CFR 46.102) and approval of these activities by the RTI IRB is not necessary (Attachment F).

Contact Information

Questions regarding any aspect of this project can be directed to:

Elizabeth Davis

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Phone: (202) 305-2667

E-mail: Elizabeth.Davis@usdoj.gov

Attachments

Attachment A: LEMAS Supplement Survey – Post-Academy Training questionnaire

Attachment B: LEMAS Supplement Survey – Response to Police Suicide questionnaire

Attachment C: LEMAS Supplement Survey Cognitive Testing Protocol

Attachment D: Agency Invite Letter

Attachment E: Non-Response Follow-Up Phone Script

Attachment F: Cognitive Interview Packet Cover Letter

Attachment G: RTI LECS IRB Determination

1 An agency’s full-time equivalent sworn personnel count is the sum of the number of full-time sworn officers and part-time sworn officers (who are counted as the equivalent of .5 full-time sworn officers) employed by that agency.

7