Memo to OMB on Cog Interview for 2022 CLETA

MEMORANDUM

MEMORANDUM TO: Robert Sivinski

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Doris James

Acting Director

Bureau of Justice Statistics

Kevin M. Scott

Deputy Director, Statistical Operations

Bureau of Justice Statistics

Alexia D. Cooper

Chief, Law Enforcement Statistics Unit

Bureau of Justice Statistics

FROM: Emily Buehler

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: June 23, 2022

SUBJECT: BJS request for OMB Clearance to conduct cognitive interviewing for the 2022 Census of Law Enforcement Training Academies (CLETA), under the OMB generic clearance agreement (OMB Number 1121-0339)

Introduction

The Bureau of Justice Statistics (BJS) is planning to conduct the 2022 Census of Law Enforcement Training Academies (CLETA) in 2023. This wave of the census will build on the previous four iterations of data collection (conducted in 2018, 2013, 2006, and 2002). The proposed collection includes all state and local law enforcement training academies offering basic law enforcement training programs for recruits during calendar year 2022 in the 50 states and the District of Columbia. BJS plans to field the 2022 CLETA from March through October 2023. This request is to conduct cognitive testing of the revised 2022 CLETA instrument under BJS’s generic clearance agreement (OMB Control Number 1121-0339).

BJS uses CLETA to complement the information obtained in the Census of State and Local Law Enforcement Agencies (CSLLEA), conducted every 4 years since 1992, and the Law Enforcement Management and Administrative Statistics (LEMAS) survey, conducted every
3–5 years since 1987. The CSLLEA and LEMAS collections target agencies that employ sworn personnel who provide law enforcement services nationwide, whereas the CLETA focuses on the training that recruits receive and the institutions that provide the training.

Through the CLETA, BJS describes characteristics of state and local law enforcement training academies, including data on the content of basic training curricula for new law enforcement recruits. The 2022 CLETA will include many of the same topics as the 2018 collection, thus enabling BJS to measure how the nature of police training has changed over time. Topics include the following:

• Types of officer positions trained

• The number and types of training instructors used

• Academy operating and equipment budgets

• The resources accessible through the academies

• The number of instruction hours provided for each training topic

• The types of special training programs offered to basic recruits

• The general training philosophy of the academy (stress/military style or non-stress/ academic style)

• The number of recruits starting and completing basic law enforcement training programs, by race, Hispanic origin, and sex

• The reasons that recruits did not successfully complete their training programs

The 2022 CLETA design efforts will focus on ways to address several challenges facing the collection, including (1) reducing burden on responding academies (and thereby potentially increasing response rates); (2) updating the survey topics to be relevant and applicable to today’s academy operations and needs; and (3) maintaining the relevance of the survey to the field, along with the utility of the survey data.

To address these challenges that were experienced during the 2018 CLETA administration and make improvements for the 2022 iteration of the survey, BJS conducted a series of stakeholder interviews to obtain feedback. The data collection awardees—RTI International, the International Association of Directors of Law Enforcement Standards and Training (IADLEST), and the National Policing Institute (NPI)—and BJS convened five groups of stakeholders:

1. Law enforcement agencies with their own academies

2. Regional and state academies

3. Researchers who have used CLETA data

4. College-based pre-hire academies

5. A mixed group with participants from each category

Each meeting lasted 90 minutes. Meetings were held by Zoom from late February to early April 2022.

Prior to the interviews, data from the 2018 CLETA were reviewed to investigate item nonresponse and validation error rates. This review suggested survey items that warranted discussion during the stakeholder interviews. Interview guides were distributed to participants in advance to allow them to review the items prior to the discussion and to help facilitate discussion during the interviews. After the interviews, feedback was consolidated and incorporated with feedback from BJS, RTI, NPI, and IADLEST to propose a final set of recommendations and to draft the 2022 CLETA instrument (Attachment A).

Request for Cognitive Interviewing

BJS has contracted with RTI to develop and conduct cognitive testing of the revised 2022 CLETA instruments. BJS requests generic clearance under OMB Control Number 1121-0339 for the cognitive testing. RTI has awarded a subcontract to NPI to assist with recruitment for cognitive testing of the instrument so that the project benefits from NPI’s longstanding relationship with law enforcement agencies.

The cognitive interviewing plan described below is designed to thoroughly test the survey instrument to ensure question clarity and reliability and assess the capability of academies to identify a proper survey respondent to provide the requested information. The project staff will review the results of the cognitive interviews and modify the survey instrument as necessary. After testing is complete, BJS plans to implement the full-scale CLETA in March 2023.

BJS will select 60 potential participants: a primary sample of 30 academies and a replacement sample of 30 academies. Academies will be categorized into five strata (Exhibit 1) and selected from various geographic regions to achieve better representation.

Exhibit 1. Cognitive Interview Academy Strata

Academy Strata	Primary Group 1	Primary Group 2	Replacement
State government academies	3	3	6
Municipal police academies	3	3	6
Sheriff’s office academies	3	3	6
Regional academies	3	3	6
Pre-hire/self-sponsored academies	3	3	6
Total	15	15	30

Exhibit 2 details the cognitive interview recruitment protocol.

Exhibit 2. CLETA Cognitive Interview Recruitment Protocol

In preparation for the 2018 CLETA, BJS sampled 15 training academies for cognitive interviews and completed interviews with 5 academies. Given this low participation rate, BJS will adjust its outreach strategy for this round of cognitive interviews, beginning with an email invitation sent from BJS to academy heads asking them to participate in the cognitive interview (Attachment C). This message will add legitimacy to the CLETA cognitive interviewing request and encourage participation.

Recruited participants will be provided with a cover letter explaining expectations for participation (Attachment D) and a cognitive interview version of the instrument (Attachment A) one week before their scheduled interview and instructed to return the partially completed form to RTI before their interview date. Participants will be asked to make note of any aspects of the instrument that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. To ease burden on participants, the core items (18 items that BJS has identified as critical) will be highlighted for all participants to complete, but only half of the remaining items will be highlighted for completion by any individual participant. Additionally, participants will be instructed that, for any items that require consultation of external sources or research to complete, they need only think about how they would go about answering; they do not need to fill out the items. This clarification is relevant for some high-burden items, such as recruit or staff counts and hours of training dedicated to various topics.

Under this approach, 15 participants (Primary Group 1) will complete half of the remaining items, and the second set of 15 (Primary Group 2) will complete the other half. The cognitive interview version of the instrument will be available as a fillable PDF, which will ease the burden of participating by allowing users to place responses directly in the PDF and email it to RTI, rather than completing the form on paper, scanning it, and sending it back.

Each telephone or videoconference interview will be moderated by an RTI cognitive interviewer, who will use the interview protocol (Attachment B) to guide the discussion. Prior to the call, the interviewer will review the completed form and identify issues that warrant probing and that are not already in the protocol; the interviewer will address these issues with the participant as needed. The point of contact and the interviewer will both have access to the completed instrument to use as a reference while the interviewer follows the cognitive testing protocol to complete the interview. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary. Participants will not receive any compensation for the interview.

Cognitive testing will also provide an opportunity for BJS to calculate more reliable burden estimates for law enforcement academies to complete the questionnaires that will be used with burden estimates obtained from the 2018 CLETA. This information will be considered as BJS continues to develop and design the 2022 CLETA. All information gathered from the testing efforts will be integrated into the full OMB clearance package that BJS expects to submit in fall 2022. BJS plans to fully implement the CLETA beginning in March 2023. All dates noted above are pending final approval.

Burden Hours

As shown by the burden hour estimates in Exhibit 3, BJS has estimated the total respondent burden for the proposed questionnaire development effort at 55 hours. The burden comprises three components: initial contact and scheduling, completion of the paper form, and completion of the cognitive interview. Up to 60 academies may be contacted and asked to participate in cognitive interviewing. However, a maximum of 30 agencies will complete and submit the paper version of a portion of the draft 2022 CLETA survey in order to obtain a tested estimate of burden for the national implementation and provide real data that can be assessed prior to the cognitive interview for quality issues and to serve as the basis for discussion during the interview. The estimate was calculated using 0.5 hours per respondent for the completion of the paper survey and 1 hour per respondent for the cognitive interview. The 0.5-hour estimate to complete the paper survey is based on the 2-hour estimate from completing the entire 2018 CLETA and factoring in a reduction in the number of items and the amount of effort spent per item that will be requested from cognitive interview participants.

Exhibit 3. Estimated Burden for Cognitive Interviews

	Average burden hours per respondent	Total maximum respondents	Est. burden hours
Initial contact and scheduling	0.17	60	10.2
Complete questionnaire	0.5	30	15.0
Complete cognitive interview	1.00	30	30.0
TOTAL HOURS			55.2

Institutional Review Board

RTI’s IRB determined that the research conducted under the CLETA contract does not constitute research involving human subjects as defined by the U.S. Code of Federal Regulations (45 CFR 46.102), and approval of these activities by the RTI IRB is not necessary (Attachment E).

Contact Information

Questions regarding any aspect of this project can be directed to:

Emily D. Buehler

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 307-1552

Email: emily.buehler@usdoj.gov

Attachments

Attachment A: CLETA2022 Survey Instrument

Attachment B: CLETA2022 Cognitive Interview Protocol

Attachment C: Academy Invite Letter

Attachment D: Cognitive Interview Cover Letter

Attachment E: IRB determination

6