Memo to OMB on LEMAS Cognitive Interviews

MEMORANDUM

MEMORANDUM TO: Robert Sivinski

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Jeffrey Anderson

Director

Bureau of Justice Statistics

Devon Adams

Acting Principal Deputy Director

Bureau of Justice Statistics

Kevin Scott

Chief, Law Enforcement Statistics Unit

Bureau of Justice Statistics

FROM: Shelley Hyland

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: August 30, 2019

SUBJECT: BJS request for OMB Clearance to conduct cognitive interviewing for the 2020 Law Enforcement Management and Administrative Statistics (LEMAS) core survey, under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) is planning to conduct the next wave of the Law Enforcement Management and Statistics (LEMAS) in fall 2020. The LEMAS survey is the most comprehensive source of national data on law enforcement agencies in the U.S. The LEMAS substantive domains have evolved over time but generally include: the size, staffing, and specialization of agencies; personnel attributes including sex and race/ethnicity of sworn employees; hiring and training practices; agency authorized and provided equipment; the adoption of technology; and agency policies and procedures. The results are used to provide an overview of law enforcement nationally, identify trends, and provide a representative benchmark for individual agency comparison.

Since 1987, BJS has implemented 10 waves of LEMAS surveys approximately every 4 years. In each iteration of LEMAS, BJS has drawn a nationally representative sample of about 3,500 general purpose state and local law enforcement agencies from either the Census of State and Local Law Enforcement Agencies (CSLLEA) (OMB Control Number 1121-0346) or the Law Enforcement Agency Roster (LEAR). The sample is designed to be representative of all general purpose state and local law enforcement agencies in the United States with at least 1 full-time equivalent (FTE) sworn officer, with separate samples drawn for local police departments, sheriffs’ offices, and state agencies. Agencies serving special jurisdictions (such as tribal lands, schools, airports, or parks), or with special enforcement responsibilities (such as conservation laws or alcohol laws), are considered out of scope for the survey. Additionally, sheriffs’ offices without primary law enforcement jurisdiction within the county they serve are ineligible for LEMAS. All agencies with 100 or more FTE officers (approximately 1,000 agencies) receive the survey. Agencies with fewer than 100 FTE officers are selected using a stratified random sample based on FTE and agency type.

Prior to the 2013 LEMAS administration, the LEMAS administered two forms—one long and one short. The long form was administered to agencies with 100 or more FTE officers and the short form was administered to agencies with less than 100 FTE officers. This format was designed to reduce the burden on smaller agencies but the differences between the two forms were minimal. In 2013 and 2016, the LEMAS was administered as one form to all sampled agencies. The overall response rate for the 2013 LEMAS was 86% and for the 2016 LEMAS it was 80%. The response rate for agencies with less than 100 FTE for the 2013 LEMAS was 82.7% and for the 2016 LEMAS it was 78.3%. The response rate for the 2016 LEMAS was lower than the 2013 LEMAS due to a shorter field collection period (7 months vs. 11 months).

The 2016 LEMAS (OMB 1121-0240) was the most recently fielded LEMAS. The 2016 LEMAS instrument largely mirrored what had been collected in previous iterations but was also designed to meet the immediate recommendations of the President’s Task Force on 21st Century Policing. New items added to the 2016 LEMAS included additional demographic items and questions on seat belt use and external investigations of use of force.

There are two significant challenges for the LEMAS survey. First, response rates for surveys, including LEMAS, have declined in recent decades in part because agencies are receiving an increasing number of survey requests and have limited resources to comply with those requests. BJS believes the best option available to encourage response, given the time and cost restraints, is to reduce the scope of questions and subsequent burden for agencies (the 2016 LEMAS took agencies, on average, three hours to complete), thereby lessening the amount of time and/or effort required to take the survey.

The second challenge for the LEMAS survey is whether the data collected are still useful. The instrument has remained largely unchanged since it was first administered in 1987, and many aspects of law enforcement in the U.S. have changed considerably since that time. Survey items must be updated to better reflect the data needs of law enforcement today. As law enforcement continues to become more reliant on data, it is critical that the LEMAS survey ask meaningful questions to serve as a valuable resource of information for practitioners. This also means the items should be meaningful to all law enforcement agencies. The current LEMAS instrument is primarily tailored to local police departments. For example, items on community policing do not apply to sheriffs’ offices or state police agencies. The survey needs to better reflect the needs of differing law enforcement agencies in order to increase response rate and produce more relevant estimates for these groups.

Understanding these challenges, BJS conducted a series of expert panel meetings in order to obtain feedback to address these challenges and revise the 2016 LEMAS for the 2020 administration. Working with awardees, RTI International (RTI) and the Police Executive Research Forum (PERF), BJS convened a group of federal, academic, and practitioner experts for a day-and-a-half long meeting to discuss potential changes to the LEMAS survey for 2020. This meeting took place on July 18-19, 2018 at the Office of Justice Programs building. Since no sheriffs were able to attend the July meeting, BJS and PERF held two conference calls on November 5 and 8, 2018, in order to solicit feedback from four sheriffs.

The expert panel meetings had two primary goals: 1) identify essential items that would form a LEMAS “core” instrument, and 2) obtain feedback on topics that are relevant to law enforcement and how these may differ by agency type. A core questionnaire would serve two primary purposes. First, it would include items that are applicable to all agency types. Second, it would include a reduced number of items in order to capture the essential characteristics of law enforcement agencies and hence reduce the 3-hour survey burden. In order to identify the core items, panel members were asked to review all the items on the 2016 LEMAS and select the most essential. They were also asked to solicit topics that were relevant to law enforcement but were not being captured on the LEMAS. Lastly, panel members were asked about topics on the 2016 LEMAS that were not relevant to their agency type and what missing topics should be included of state police and sheriffs’ offices. Details of these discussions are available in the LEMAS 2020 Expert Panel Report (Attachment A).

As a result of these discussions, the panel members were in favor of a core plus agency type supplement design. The core items would be given to all agencies and the agency type supplement would be administered only for that agency type. The agency supplement would allow BJS to better tailor items for specific types of agencies. Doing so would also allow the LEMAS to be administered to additional agency types in the future, such as special purpose agencies. These agencies are only surveyed about basic functions through the CSLLEA. Some have their own collections (e.g., Survey of Campus Law Enforcement Agencies (OMB Control Number 1121-0334) and Census of Tribal Law Enforcement Agencies (OMB Control Number 1121-0366)), but the items included in these collections largely mirror what is captured in the LEMAS.

The core plus agency type structure would also mirror how BJS has reported data from the LEMAS. There have been two primary publication streams from the LEMAS data: Local Police Department Series and Sheriffs’ Offices Series. While the LEMAS has also collected data from primary state police agencies, there has been no primary state police report published other than the number of full-time and part-time sworn and nonsworn personnel featured in Table 1 of the Local Police Department reports. Going forward, the new instrument structure will better allow BJS to continue producing reports for specific agency types including primary state police and, if expanded, special purpose agencies.

Request for Cognitive Interviewing

This request is to conduct cognitive testing for the two LEMAS 2020 instruments under BJS’s generic clearance agreement (OMB Control Number 1121-0339). BJS has contracted with RTI to develop and test the instruments. RTI has awarded a subcontract to PERF to assist with recruitment for cognitive testing for the LEMAS 2020 because of their longstanding relationship with law enforcement agencies. Project staff from RTI will cognitively test (1) the LEMAS survey for Local Police Departments and Primary State Agencies (Attachment B) and (2) the LEMAS survey for Sheriffs’ Offices (Attachment C).

The cognitive interviewing plan described below is designed to (1) thoroughly test the survey instruments to ensure question clarity and reliability and (2) assess the capability of law enforcement agencies’ ability to identify a proper respondent for the survey and to provide the requested information. The project staff will review the results of the cognitive interviews and modify the survey instruments as necessary. Following the completion of testing, the BJS plans to implement the full-scale LEMAS in September 2020.

The project team will develop a preliminary list of agencies for cognitive testing based on two criteria (agency size and agency type) partially representing how the full sample will be drawn. Of the 3,500 general purpose law enforcement agencies in the full LEMAS sample, approximately 1,000 agencies have 100 or more FTE (about 30% of the sample) and 2,500 having 99 or fewer FTE (about 70% of the sample). The ratio of 100+ FTE and 99 or fewer FTE is about the same for the samples chosen for local police departments and sheriffs’ offices. All primary state police agencies have 100 or more FTE. Within the 99 or fewer FTE category, approximately half of the sample (and proportion within the frame) includes agencies with 9 or fewer FTE and the other half of agencies fall within 10-99 FTE. Following these guidelines, the project team will randomly select agencies based on the following two stratification categories: (1) agency type (local police department and sheriff’s office) and (2) agency size (small: 9 or fewer FTE, medium: 10-99 FTE, and large: 100 or more FTE). Primary state police agencies are excluded from cognitive interviewing due to the small number of agencies (N=49).

Forty (40) law enforcement agencies will be selected to participate in cognitive interviews: 20 local police departments and 20 sheriff’s offices. The breakout of respondents based on agency type and size is shown in Table 1.

Table 1. Number of cognitive interviews by agency type and size.

FTE	Local	Sheriff	Total
100+	6	6	12
10-99	7	7	14
1-9	7	7	14
Total	20	20	40

As BJS anticipates that some of the 40 sampled agencies will be unable to participate in the cognitive interview effort, a substitute agency will be selected for each of the sampled agencies and contacted only if a replacement is needed. The intended respondent for the survey is the agency’s chief or another staff member who is knowledgeable about the law enforcement personnel and policies. Based on past experience with cognitive interviewing for the 2016 LEMAS, 2018 CSLLEA, and 2018 Census of Law Enforcement Training Academies (CLETA; OMB Control Number 1121-0255), the project team is confident that the resulting list of agencies prioritized for cognitive interviewing will be diverse in characteristics and will provide a sampling of potential obstacles to the proposed information collection.

Initial contact with 40 agencies will be conducted by phone and led by PERF. Once contact is established with the agency head (or a designee), the purpose of the test and the scope of agency involvement will be explained. When an agency is recruited, participants will be asked to provide a point of contact (POC) and their email address. If an agency does not choose to participate, a replacement from the same stratum will be selected and outreach will be made to recruit the new agency. The replacement process will be repeated until the required number of respondents in each stratum is obtained.

Within 1 week of agency recruitment, a survey package will be sent to the respondent either by mail or email. The packet will include a cover letter explaining the purpose of the test (Attachment D), a copy of the draft questionnaire, a postage paid return envelope (to send by mail), and an email address and fax number (to send by email). Respondents will be asked to complete the questionnaire within 1 week, make a copy for their reference and return the original to RTI. They will also be asked to record the time spent by all agency staff to complete the form; this information will be captured on the questionnaire. Upon receipt of the completed questionnaire, RTI will contact the respondent by telephone or email to schedule a 60-minute debriefing call. During that call, a member of the project team will conduct a retrospective cognitive interview using a standardized interview guide (Attachments E and F). The POC and the interviewer will both have access to the completed instrument to use as a reference while the interviewer follows the cognitive testing protocol to complete the interview. Participants will be asked to take note of any aspects of the instrument that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. Participants will not receive any compensation for the interview. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary.

Summary of Instruments

The 2020 LEMAS instruments were modified based on recommendations from the expert panel and BJS priorities. As mentioned previously, the most substantive change to the 2020 LEMAS structure is the development of two instruments: one for local and primary state police departments and one for sheriffs’ offices. Both instruments contain the same set of core items which include sections IV-VIII on the local police/primary state police survey (LP) and sections V-IX on the sheriffs’ offices survey (SO). Sections I-III (LP) and I-IV (SO) are unique to the agency types.

Core Items

Selection and Training (5 items): Section IV (LP) and Section V (SO)

Section IV (LP) and Section V (SO) represents the first section of the core items and are related to selection of new officer recruits and training. All of these items have been included on the LEMAS surveys since 2000. Expert panel members felt that selection and training were important items to retain on the core. Due to the CLETA, BJS was careful to only include items captured previously on training rather than expanding this section. For officer selection, the LEMAS asks agencies about their minimum education requirement (Q14a LP; Q17a SO) and if there is a military service exemption to this item (Q14b LP; Q17b SO) and pre-employment screening practices (Q15 LP; Q18 SO). For training, the LEMAS asks about the total number of academy (Q16 LP; Q19 SO), field training (Q16 LP; Q19 SO) and annual in-service training hours (Q17 LP; Q20 SO).

Hiring and Retention (10 items): Section V (LP) and Section VI (SO)

The vast majority of expert panel members stated that questions on hiring and retention practices were needed on the LEMAS. Over the past few years, law enforcement agencies have struggled to maintain staffing levels and panel members wanted to know what is being done by agencies to address this issue. To address this need, BJS included items asked on past LEMAS surveys and from a 2008 Census of State and Local Law Enforcement Agencies (CSLLEA) supplement on hiring and retention. Items from earlier waves of LEMAS include the number of hires (Q18 LP; Q21 SO), number of separations by reason of separation (Q22 LP; Q25 SO), base annual salary by sworn position (Q24 LP; Q27 SO), authorized special pay (Q25 LP; Q28 SO), and standard shift length for patrol officers (Q27 LP; Q30 SO). Items from the 2008 CSLLEA include types of applicants targeted for hiring (Q20 LP; Q23 SO), hiring incentives (Q21 LP; Q24 SO), exit interview policy for those who leave (Q23 LP; Q26 SO), and methods to increase retention rate (Q26 LP; Q29 SO). One new item added at the request of the expert panel members was the average number of weeks to hire (Q19 LP; Q22 SO). Departments that take too long to hire often have more difficulty retaining new recruits and this item will allow BJS to examine what is happening on average nationally and by population served or agency size.

Equipment and Operations (4 items): Section VI (LP) and Section VII (SO)

This section has been a staple of LEMAS since it was first developed. During the expert panel discussions, the local agencies found less utility for these items than the sheriffs. However, all panel members agreed that the items pertaining to weapons and actions (Q31 and Q32 on the 2016 LEMAS) were important. It was suggested that Q29, Q30, Q31 and Q32 on the 2016 LEMAS could be combined into one item. Q28 (LP) and Q31 (SO) on the 2020 LEMAS reflect this combined item and ask about the authorization and on/off-duty use for different types of firearms and less-lethal weapons and techniques. This section also contains an item asking about the number of video cameras (Q29 LP; Q32 SO) which has been used on previous LEMAS surveys. The other two items in this section are about K-9s (Q30a and Q30b LP; Q33a and Q33b SO). Expert panel members from all three agency types expressed a desire to know more about the functions of K-9s, which has not been asked about on previous LEMAS or CSLLEA instruments. Past LEMAS surveys have asked about the number of dogs but no items pertaining to what K-9s engage in. The 2020 LEMAS is also asking about the number of handlers for the dogs and the activities that the K-9s engage in.

Technology (4 items): Section VII (LP) and Section VII (SO)

Past LEMAS waves have also had a number of items pertaining to technology but many of these would be considered out of date. For example, questions asking about what types of data are maintained as computerized files (Q44 on 2016 LEMAS) are not particularly meaningful as the majority of departments have moved to electronic systems in the past few years. Expert panel members felt the detail of some of the items was not necessary, preferring yes/no to whether the agency has a website (Q31 LP; Q34 SO) or uses social media (Q32 LP; Q35 SO) to asking what is specifically used or provided through these mediums. The historic LEMAS item relating to technologies used on a regular basis (Q33 LP; Q36 SO) was retained and updated to be more relevant to agencies. The other item in the section updated Q39 from the 2016 LEMAS, which asked about the tasks for which computers are used. Expert panel members stated they were more interested in how data was being used rather than if computers were being used. Q34 (LP) and Q37 (SO) on the 2020 LEMAS ask agencies if they use data for a number of activities such as budget allocation, hot spot analysis, and patrol allocation.

Policies and Procedures (9 items): Section VIII (LP) and Section IX (SO)

The last section of the core items pertains to policies and procedures, which also contains a number of previously asked LEMAS items. Expert panel members consistently rated this section as one of the most useful components of LEMAS. Items retained from previous iterations of LEMAS include asking about written policy or procedural directives on a number of topics (Q35 LP; Q38 SO), use of an Early Intervention System (Q40 LP; Q43 SO), the number of civilian complaints by disposition status (Q41 LP; Q44 SO), the presence of a civilian complaint review board (Q42 LP; Q45 SO), and external investigation of use of force incidents (Q43 LP; Q46 SO). A new set of items pertaining to immigration policies and practices is also included in the 2020 LEMAS core. These items were added to respond to the President and Attorney General’s emphasis on immigration practices. Items ask agencies whether they check immigration status of persons detained (Q36 LP; Q39 SO), under what circumstances they check immigration status (Q37 LP; Q40 SO), if agencies verify immigration status with the Department of Homeland Security (Q38 LP; Q41 SO) and reasons for not checking immigration status (Q39 LP; Q42 SO).

Agency Specific Sections

Among the sections included to be agency specific, a number of items still overlap substantially between the two forms. Therefore, both instruments will be discussed within these sections. Only Section III (SO) differs substantially between the two versions.

Personnel (8 items): Section I (LP and SO)

The first section of both surveys is on personnel. This section contains what are likely the most essential items of LEMAS. LEMAS is the only national survey to provide race and Hispanic origin of sworn officers. This section is modified to better reflect the types of personnel employed by local police and state police departments compared to those in sheriffs’ offices. Seven items in this section have been asked on previous LEMAS waves. Q1 is the primary staffing count variable. On the 2016 LEMAS there were three types of personnel: sworn with general arrest powers, officers/deputies with limited arrest powers (e.g., jail or court officers in some agencies) and non-sworn employees. The category pertaining to officers/deputies with limited arrest powers was added solely for sheriffs’ offices and has created confusion for respondents in local departments and primary state police agencies. Therefore, this type of personnel is dropped on LP on the 2020 LEMAS but retained on SO. Another modification to Q1 on the 2016 LEMAS was removing “authorized” positions. During data collection and expert review, this item caused confusion and agencies had difficulty distinguishing the difference between “authorized” and “actual” full-time sworn. For the 2020 LEMAS, BJS has created a new item (Q2) asking about the number of full-time sworn officer vacancies. This value can be added to the number of full-time sworn officers in order to obtain the authorized count.

The item requesting the number of personnel by primary job responsibility (Q3) differs the most between the two forms. On LP, there are 4 primary duty areas: administration, operations, support and other. On SO, there are 7 primary duty areas: administration, operations, support, jail-related, court-related, civil process, and other. On both forms, agencies are asked to report the number of patrol/field officers, detectives/investigators and dispatchers. Personnel counts are requested for sworn officers with general arrest powers and non-sworn on LP and for these personnel types as well as officers with limited or no arrest powers on SO.

Race, Hispanic origin and sex of full-time sworn officers is captured through Q4. Q6 captures race, Hispanic origin and sex based on supervisory status and does not differ between the two forms, but are included in the personnel section for continuity. Similarly, Q5 does not differ between the two forms other than adjusting the question stems slightly; chief executive is used in the stem on LP and sheriff is used on SO.

This section also includes an item asking for the number of full-time agency personnel who are bi- or multilingual by type of personnel (Q7). This item only differs between the two forms because SO also includes counts for officers/deputies with limited or no arrest powers.

The last item (Q8) in the section asks about the types of personnel assigned to various problems or tasks. There is no difference in this item across the two forms. Since it deals with personnel, its placement within this section was most fitting.

Budget (4 items): Section II (LP and SO)

Operating budget has been on every wave of LEMAS, and asset forfeiture funds (Q10 LP; Q11 SO) has been on every LEMAS since 1997 (in 2013 total amount was replaced with yes/no option). The primary difference in this section across the two forms is asking for the jail administration budget (Q10a and Q10b SO) separately from the total operating budget (Q9a) for sheriffs’ offices. Past administrations of LEMAS have shown that some sheriffs include the jail budget in the total operating costs and others do not. By allowing them to report separately, this will allow BJS to better understand the budget allocations in sheriffs’ offices. The fiscal year is asked on both forms (Q9b). Lastly, at the request of the expert panel members an item was included on whether or not the budget included a line item for community policing activities (Q9c).

Community Policing (3 items): Section III (LP) and Section IV (SO)

Both surveys include three items on community policing. However, on LP only local police will be asked these items not the primary state police agencies. Based on expert panel feedback and result from previous LEMAS waves, primary state police agencies rarely engaged in these activities. LEMAS has included community policing items since 2000. The three items retained for 2020 have been revised with feedback from the expert panel members, which included the DOJ Community-Oriented Policing Service (COPS) office, to be more relevant. Items in this section ask the types of groups/organizations that the agency had problem-solving partnership with (Q11 LP; Q14 SO), the types of activities that the agency used community feedback on (Q12 LP; Q15 SO) and types of community policing activities the agency engaged in (Q13 LP; Q16 SO).

Service Area (2 items): Section III (SO)

On SO, section III consists of two items pertaining to their service area. Specifically, sheriffs’ offices are asked to enter the total square mileage of their service area (Q12) and the total resident population for which they have primary responsibility for providing law enforcement services (Q13). The sheriffs that participated in the expert panel discussions requested these two items. Additionally, BJS has had difficulty accurately capturing the population for which sheriffs primarily provide law enforcement to and as such typically do not report out by population served. Q13 would allow BJS to report on sheriffs’ offices by population served.

Cognitive testing will also provide an opportunity for BJS to calculate more reliable burden estimates for law enforcement agencies to complete the questionnaires. This information will be taken into consideration as the BJS continues the development and design of the LEMAS 2020. All information gathered from the LEMAS testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB by May 2020. BJS plans to fully implement the LEMAS beginning in September 2020. All of the dates noted above are planned and pending final approval.

Burden Hours

The burden hour estimates are provided in the following table. Estimated burden hours are the same for all 40 respondents regardless if they receive the local police departments/primary state agencies or sheriffs’ offices version of the surveys. BJS has estimated the respondent burden for the proposed questionnaire development effort at 134 hours (Table 2). The burden is comprised of three components: initial contact and scheduling, completion of the paper form and completion of the cognitive interview. Up to 80 law enforcement agencies may be contacted and asked to participate in cognitive interviewing. However, a maximum of 40 agencies (20 local police departments and 20 sheriffs’ offices) will complete and submit the paper version of the draft 2020 LEMAS survey in order to (1) obtain tested estimate of burden for the national implementation and (2) provide real data that can be assessed prior to the cognitive interview for quality issues and to serve as the basis for discussion during the interview. The estimate was calculated using 2 hours per respondent for the completion of the paper survey and 1 hour per respondent for the cognitive interview. The 2-hour estimate to complete the paper survey is based on the 3-hour estimate from the 2016 LEMAS and reflects the reduction in the number of items compared to that instrument.

	Average burden hours per respondent	Total maximum respondents	Est. burden hours
Initial contact and scheduling	0.17	80	13.6
Complete questionnaire	2.00	40	80.0
Complete cognitive interview	1.00	40	40.0
TOTAL HOURS			133.6

Table 2. Estimated burden hours for cognitive interviews.

Institutional Review Board

RTI’s IRB determined that the research conducted under the LECS contract does not constitute research involving human subjects as defined by the U.S. Code of Federal Regulations (45 CFR 46.102) and approval of these activities by the RTI IRB is not necessary.

Contact Information

Questions regarding any aspect of this project can be directed to:

Shelley Hyland

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 616-1706

E-mail: Shelley.Hyland@usdoj.gov

Attachments

Attachment A: LEMAS 2020 Expert Panel Report

Attachment B: LEMAS 2020 Survey: Local Police Departments and Primary State Agencies

Attachment C: LEMAS 2020 Survey: Sheriffs’ Offices

Attachment D: Cover Letter

Attachment E: LEMAS 2020 Local Police/Primary State Cognitive Testing Protocol

Attachment F: LEMAS 2020 Sheriffs’ Offices Testing Protocol

8