Download:
pdf |
pdfList of Attachments in PDF
1. 2022 CLETA web screen shots
2. 30-day notice
3. Pre-notification letter
4. Census invitation letter (mail)
5. 2022 CLETA flyer
6. IADLEST letter of support
7. Census invitation letter (email)
8. First reminder (mail)
9. Second reminder (mail)
10. Third reminder (email)
11. Fourth reminder (mail)
12. Telephone Data Quality Follow Up script
13. Telephone Non Response Follow Up script
14. Fifth reminder (UPS mail)
15. Sixth reminder (email)
16. IADLEST Reminder (email)
17. Targeted mailing (FedEx)
18. End-of-study warning (postcard)
19. End-of-study notification (mail)
20. Eligibility confirmation (email)
21. Law Enforcement Organizations Endorsement letter
22. Partial Complete Completion Request (mail and email)
23. Thank you letter (mail)
24. Cognitive Interview Report
2022 CLETA Website Mockups
https://bjslecs.org/cleta2022 Main Page
https://bjslecs.org/cleta2022 Login Screen
https://bjslecs.org/cleta2022 Survey Launch Page
DEPARTMENT OF JUSTICE
Bureau of Justice Statistics
[OMB Number 1121-0255]
Agency Information Collection Activities;
Proposed Collection Comments Requested;
Reinstatement, with change, of a previously approved collection for which approval has
expired: 2022 Census of Law Enforcement Training Academies
AGENCY: Bureau of Justice Statistics, Department of Justice.
ACTION: 30-Day Notice.
====================================================================
SUMMARY: The Department of Justice (DOJ), Office of Justice Programs, Bureau of Justice
Statistics, will be submitting the following information collection request to the Office of
Management and Budget (OMB) for review and approval in accordance with the Paperwork
Reduction Act of 1995. The proposed information collection was previously published in the
Federal Register, Volume 87, Number 219, page 68518 on Tuesday, November 25, 2022,
allowing a 60-day comment period. Following publication of the 60-day notice, BJS did not
receive any comments on the proposed information collection.
DATES: Comments are encouraged and will be accepted for 30 days until [INSERT DATE 30
DAYS AFTER DATE OF PUBLICATION IN THE FEDERAL REGISTER].
FOR FURTHER INFORMATION CONTACT:
Written comments and recommendations for the proposed information collection should be sent
within 30 days of publication of this notice to www.reginfo.gov/public/do/PRAMain. Find this
particular information collection by selecting "Currently under 30-day Review - Open for Public
Comments" or by using the search function.
SUPPLEMENTARY INFORMATION: Written comments and suggestions from the public
and affected agencies concerning the proposed collection of information are encouraged. Your
comments should address one or more of the following four points:
-
Evaluate whether the proposed collection of information is necessary for the proper
performance of the functions of the Bureau of Justice Statistics, including whether the
information will have practical utility;
-
Evaluate the accuracy of the agency’s estimate of the burden of the proposed
collection of information, including the validity of the methodology and assumptions
used;
-
Evaluate whether and if so how the quality, utility, and clarity of the information to
be collected can be enhanced; and
-
Minimize the burden of the collection of information on those who are to respond,
including through the use of appropriate automated, electronic, mechanical, or other
technological collection techniques or other forms of information technology, e.g.,
permitting electronic submission of responses.
Overview of this information collection:
1) Type of Information Collection: Reinstatement of the Census of Law Enforcement
Training Academies, with changes, a previously approved collection for which
approval has expired.
2) The Title of the Form/Collection:
2022 Census of Law Enforcement Training
Academies (CLETA)
3) The agency form number, if any, and the applicable component of the Department
sponsoring the collection: The form number for the questionnaire is CJ-52. The
applicable component within the Department of Justice is the Bureau of Justice
Statistics, in the Office of Justice Programs.
4) Affected public who will be asked or required to respond, as well as a brief
abstract: Respondents will include all state and local law enforcement training
academies in the United States that provide basic training to law enforcement
recruits. BJS has conducted the CLETA regularly since 2002. The 2022
CLETA will be the fifth administration. Historically, the CLETA generates an
enumeration of all state and local training academies that provide basic law
enforcement training in the United States. The CLETA provides details about
the instructors, curricula, resources, and recruits at the approximately 750
training academies operating nationally. The survey asks about the operating
entity; resources available to recruits; total operating budget; full-time and
part-time instructors or trainers and their education, sworn officer experience,
certifications, and ongoing training; sex, race and Hispanic origin, prior
educational attainment, and veteran status of recruits starting and completing
training; and the length and content of basic training curricula offered at the
academy
5) An estimate of the total number of respondents and the amount of time estimated for
an average respondent to respond: BJS estimates approximately 750 law
enforcement academies with a respondent burden of about 2 hours per academy to
complete the survey form and about 10 minutes per agency of data quality follow-up
time for approximately 450 of those academies.
6) An estimate of the total public burden (in hours) associated with the collection: There
are an estimated 1,575 total burden hours associated with this information collection.
Carley M. Lester, Department Clearance Officer, United States Department of Justice, Justice
Management Division, Policy and Planning Staff, Two Constitution Square, 145 N Street NE,
3E.405A, Washington, DC 20530.
Dated:
__________________________________
Carley M. Lester,
Department Clearance Officer for PRA,
U.S. Department of Justice.
Billing Code: 4410-18
«Date»
«Salutation» «ContactFirstName» «ContactLastName»
«Academy Name»
«ContactAddress1» «ContactAddress2»
«ContactCity», «ContactState» «ContactZip»
Dear «Salutation» «ContactLastName»:
I am pleased to announce that the Bureau of Justice Statistics (BJS) has begun preparations for the 2022
Census of Law Enforcement Training Academies (CLETA). CLETA is an important part of BJS’s Law
Enforcement Core Statistics program, which coordinates several law enforcement agency surveys
conducted by BJS. BJS has been administering CLETA since 2002 and periodically gathered information
on the content of basic training curricula for new law enforcement recruits. By comparing data over time,
CLETA is able to show how the nature of law enforcement training has changed.
In the next few weeks, BJS will invite <> to participate in the 2022 CLETA;
specifically, your academy will be asked to complete an online survey focusing on the types of training
offered, the types of agencies and positions for which basic training is provided, academy personnel and
recruits, and other topics related to the training you provide.
I appreciate that you receive a number of data requests throughout the year, and I thank you for your
support for CLETA. If you have questions about CLETA, please contact the data collection team via
phone or e-mail at <> or cleta@rti.org. If you have any general comments about
this data collection, please contact the Bureau of Justice Statistics Program Manager Emily Buehler at
202-598-1036 or Emily.Buehler@usdoj.gov.
Sincerely,
Dr. Alexis R. Piquero, Director
Bureau of Justice Statistics
1
«TITLE» «ACADEMY HEAD/POC NAME» OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
I am writing to ask for your participation in the 2022 Census of Law Enforcement Training Academies (CLETA). Since
2002, CLETA has periodically gathered information on the basic training curricula for new law enforcement recruits.
Your response is critical to the Bureau of Justice Statistics effort to produce national estimates of personnel, resources,
curricula, trainees, policies, and practices of the academies that train state and local law enforcement officers. It has been
endorsed by the law enforcement training community and a letter of support along with information on the CLETA is
enclosed.
To complete your survey, please access the questionnaire online at https://bjslecs.org/cleta2022. You may start and stop as
needed. Your training academy-specific information is:
Username:
Password:
«WebUsername»
«PIN»
Please complete this questionnaire online by [DATE].
The questionnaire takes approximately 2 hours to complete including time to find information you may not have readily
available. You may download a copy of the survey from the website to assist you in gathering the necessary data.
If you need to change the point of contact for your training academy or update your contact information (including email
address), go to https://bjslecs.org/cleta2022 using the username and password shown above and follow the instruction
provided on the website. If you have questions, please contact the CLETA data collection team via phone or e-mail at
<> or CLETA@rti.org. If you have any general comments, please contact me at 202-598-1036
or Emily.Buehler@usdoj.gov.
BJS uses the data collected in CLETA only for research and statistical purposes, as described in Title 34, USC §10134.
RTI International, the CLETA data collection agent, is required to adhere to BJS Data Protection Guidelines, which
summarize the many federal statutes, regulations, and other authorities that govern all BJS data and data collected and
maintained under BJS’s authority. The Guidelines may be found at
http://www.bjs.gov/content/pub/pdf/BJS_Data_Protection_Guidelines.pdf.
Thank you in advance for your agency’s participation in CLETA. We appreciate your time and effort.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Enclosures: CLETA Flyer, IADLEST Endorsement Letter
2022 Census of Law Enforcement Training
Academies (CLETA)
Although completing basic academy training is an
important step in a law enforcement officer’s career,
the features of these programs can vary significantly
across the United States. Through the Census of
Law Enforcement Training Academies (CLETA), the
Bureau of Justice Statistics (BJS) is able to describe the
characteristics of basic academy training for new recruits
for all state and local law enforcement academies in the
United States.
The CLETA has been administered approximately
every 5 years since 2002. The next CLETA will begin
administration in spring 2023. BJS will send the
CLETA survey to approximately 750 local and state law
enforcement training academies. Data will be used to
produce national estimates describing the basic training
curricula, characteristics of trained recruits, and instructor
qualifications of these academies.
Conducted by:
Bureau of Justice Statistics, U.S. Department of Justice
RTI International
2022 Census of Law Enforcement Training
Academies Survey Content
• Types of training offered by the academy
• Types of agencies and positions for
which basic training is provided
• Characteristics of the training personnel
(e.g., sworn status, law enforcement
experience, education)
• Academy resources (e.g., funding)
• Number of recruits starting and
completing programs and number/
reasons for not completing
In addition to providing a snapshot of the current
state of basic law enforcement training, BJS will be
able to compare the results of the 2022 CLETA with
data from prior years to determine how training has
changed over time. Results from the 2022 CLETA
will be made available in 2024 to law enforcement
personnel, researchers, lawmakers, and other
stakeholders to facilitate nationwide discussions on
law enforcement training.
CLETA is part of a coordinated program of law
enforcement agency surveys conducted by BJS.
Also included in these are the Census of State and
Local Law Enforcement Agencies (CSLLEA), the Law
Enforcement Management, Administrative Statistics
(LEMAS) core survey and the Survey of Campus Law
Enforcement Agencies (SCLEA). BJS is working with
RTI International to administer these surveys.
2022 CLETA Survey Schedule
Winter–Spring 2023
• BJS will send a letter inviting law enforcement
training academy directors to participate in
CLETA
• Academy directors can designate a point of
contact to complete the survey
• RTI will provide directors and points of contact
with access to the survey website to allow
review of survey items and submission of data
• Training academies will send their data to RTI
either online or using a paper questionnaire
Summer–Fall 2023
• Data quality assessments
• Data compiled and processed
Spring–Summer 2024
• Results analyzed
• BJS will publish a report on the findings
For more information about CLETA, visit bjslecs.org/CLETA2022.
For additional information on the LECS suite of BJS data collection efforts, visit bjslecs.org.
You may also contact:
Tom Scott, PhD
CLETA Principal Investigator
RTI International
3040 E. Cornwallis Road
Research Triangle Park, NC 27709
tlscott@rti.org
781-370-4020
Emily Buehler, PhD
CLETA Program Manager
Bureau of Justice Statistics
810 Seventh Street NW
Washington, DC 20531
Emily.Buehler@usdoj.gov
202-598-1036
International Association of Directors of
Law Enforcement Standards and Training
Headquarters: 152 S. Kestrel Pl., Ste 102, Eagle, ID 83616
Washington DC: 1775 “I” Street NW, Suite 1150, Washington, D.C. 20006
Main: (208) 288-5491
Fax (800) 783-6438
www.iadlest.org
ERIK BOURGERIE
President
Colorado Peace Officer Standards
&Training Board
1300 Broadway, 9th Floor
Denver, CO 80203
(720) 508-6736
Erik.Bourgerie@coag.gov
MIKE AYERS
First Vice President
Georgia Peace Officer Standards
& Training
PO Box 349
Clarkdale, GA 30111
(706) 421-8217
Mike.Ayers@gapost.org
CHRIS WALSH
Second Vice President
Wyoming Peace Officer Standards
& Training
1556 Riverbend Drive
Douglas, WY 82633
(307) 358-8215
Chris.walsh1@wyo.gov
TREVOR ALLEN
Treasurer
North Carolina Academy
200 West College Street
Salemburg, NC 28385
(919) 926-6000
tjallen@ncdoj.gov
STEPHANIE PEDERSON
Secretary
Wisconsin Department of Justice
Training & Standards Bureau
17 West Main Street, PO Box 7070
Madison, Wisconsin 53707
(608) 261-8641
pedersonse@doj.state.wi.us
MICHAEL N. BECAR
Executive Director, CEO
152 S. Kestrel Place, Suite 102
Eagle, ID 83616-5137
(208) 288-5491 Fax: (800) 783-6438
mikebecar@iadlest.org
John V. Scippa
Northeast Region
New Hampshire
(603) 271-1793
John.V.Scippa@pst.nh.gov
Amanda Yarbrough
Central Region
Arkansas
(501) 682-4750
Amanda.yarbrough@arkansas.gov
November 2022
The Bureau of Justice Statistics (BJS), working with RTI International
(RTI), is fielding the 2022 Census of Law Enforcement Training
Academies (CLETA). This survey is sent to all of the approximately
750 law enforcement training academies in the United States and the
BJS team is hoping that you will participate in this important project.
The information that will be produced by this survey will be very
valuable not only to BJS, but to all academy directors. From the survey
results, you will be able to learn about how other academies conduct
their business. For instance, you will learn about curriculum content,
training methods, staff credentials and training, facilities, budgets,
policies and more. You will be able to assess your own academy in
relation to national norms.
I write to strongly encourage you to complete the survey. The receipt
of information from each and every academy will greatly enhance the
value of the data produced by this project. We know that you and your
staff have many responsibilities and limited time, but we hope that you
will provide the requested information and contribute to this effort. Your
participation will help ensure that the 2022 CLETA is a success and that
the results can be used with confidence.
Thank you in advance for your cooperation with this important
endeavor.
Sincerely,
Michael N Becar
Executive Director
Glen W Hopkins
South Region
Florida
(850) 410-8660
glenhopkins@fdle.state.fl.us
Darin Beck
Midwest Region
Kansas
(620) 694-1400
dbeck@kletc.org
Matt Giordano
West Region
Arizona
(602) 774-9350
mattg@azpost.gov
Joe Trindal
International Region
Washington DC
(202) 359-8505
jtrindal@iadlest.org
Charles “Skeet” Brewer
Federal Representative
Georgia
(912) 267-2654
Charles.brewer@fletc.dhs.gov
SUBJECT: 2022 Census of Law Enforcement Training Academies
Dear «TITLE» «NAME»:
Last week, we sent you the 2022 Census of Law Enforcement Training Academies (CLETA). This email
message requests confirmation that you successfully received your invitation. We encourage you to contact us if
you have any questions related to the data collection or did not receive the materials.
The mailing contained two enclosures that you can view on the survey web site:
An informational flyer describing the goals and schedule for the 2022 CLETA – <>
A letter of support from the International Association of Directors of Law Enforcement Standards and
Training– <>
Please reply to this message to confirm that we have reached:
«Academy Name»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
In the event you did not receive the packet, the information contained in the mailed materials is provided below.
Thank you,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
«TITLE» «ACADEMY HEAD/POC NAME» OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
I am writing to ask for your participation in the 2022 Census of Law Enforcement Training Academies (CLETA). Since
2002, CLETA has periodically gathered information on the basic training curricula for new law enforcement recruits.
Your response is critical to the Bureau of Justice Statistics effort to produce national estimates of personnel, resources,
curricula, trainees, policies, and practices of the academies that train state and local law enforcement officers. It has been
endorsed by the law enforcement training community and a letter of support along with information on the CLETA is
enclosed.
To complete your survey, please access the questionnaire online at https://bjslecs.org/cleta2022. You may start and stop as
needed. Your training academy-specific information is:
Username:
Password:
«WebUsername»
«PIN»
Please complete this questionnaire online by [DATE].
The questionnaire takes approximately 2 hours to complete including time to find information you may not have readily
available. You may download a copy of the survey from the website to assist you in gathering the necessary data.
If you need to change the point of contact for your training academy or update your contact information (including email
address), go to https://bjslecs.org/cleta2022 using the username and password shown above and follow the instruction
provided on the website. If you have questions, please contact the CLETA data collection team via phone or e-mail at
<> or CLETA@rti.org. If you have any general comments, please contact me at 202-598-1036
or Emily.Buehler@usdoj.gov.
BJS uses the data collected in CLETA only for research and statistical purposes, as described in Title 34, USC §10134.
RTI International, the CLETA data collection agent, is required to adhere to BJS Data Protection Guidelines, which
summarize the many federal statutes, regulations, and other authorities that govern all BJS data and data collected and
maintained under BJS’s authority. The Guidelines may be found at
http://www.bjs.gov/content/pub/pdf/BJS_Data_Protection_Guidelines.pdf.
Thank you in advance for your agency’s participation in CLETA. We appreciate your time and effort.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Enclosures: CLETA Flyer, IADLEST Endorsement Letter
«TITLE» «ACADEMY HEAD/POC NAME»
OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
The Bureau of Justice Statistics (BJS) recently mailed you a link to a web questionnaire asking
about your law enforcement training academy for the 2022 Census of Law Enforcement Training
Academies (CLETA). The questionnaire due date is [DUE DATE] and we hope you will be able
to respond by then. If you have already completed the questionnaire, please accept my sincere
thank you.
If you have not completed your questionnaire, please complete it as soon as possible. No other
national data collection can provide such comprehensive data on the functions, personnel, and
training at law enforcement training academies. Developing and maintaining an accurate picture
of the nation’s law enforcement training is paramount to understanding the current state of law
enforcement in the United States.
Please complete the questionnaire by using the following link: https://bjslecs.org/cleta2022
and entering the following information:
Username: <>
Password: <>
If you would prefer to complete the questionnaire on paper, you may download and print a paper
version upon entering your questionnaire access code on the CLETA website. You may also
request a paper questionnaire by emailing RTI International at CLETA@rti.org or calling <>. If you have any general comments about this data collection, please
contact me at 202-598-1036 or Emily.Buehler@usdoj.gov.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
«TITLE» «ACADEMY HEAD/POC NAME»
OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
On behalf of the Bureau of Justice Statistics (BJS), RTI International is conducting the 2022
Census of Law Enforcement Training Academies (CLETA). RTI has been reaching out to
<> since March. We hope to receive your questionnaire soon so that the
census data reflect the variety of training provided by academies of all types and sizes.
Information from your academy is needed to ensure the quality of the study.
The due date is [DUE DATE].
Please complete the CLETA questionnaire as soon as possible. I understand that you receive a
number of survey requests and I genuinely appreciate your attention to this request.
You may access the questionnaire online at https://bjslecs.org/cleta2022 and entering the
following information:
Username: <>
Password: <>
If you have questions about CLETA, need to change the point of contact at your training
academy, or need to update your contact information, please contact the RTI team via phone or
e-mail at <> or CLETA@rti.org. If you have any general comments
about this data collection, please contact me at 202-598-1036 or Emily.Buehler@usdoj.gov.
Thank you for your time and consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
SUBJECT: Census of Law Enforcement Training Academies Reminder
Dear «TITLE» «NAME»:
Over the past 2 months, materials related to the 2022 Census of Law Enforcement Training Academies
(CLETA) were sent to you by mail. This email message is to request confirmation that we have successfully
reached you and encourage you to contact us if you have any questions related to the data collection.
Please reply to this message to confirm that we have reached <>.
The due date is [DUE DATE].
The information contained in the letter that we mailed most recently (on <>) is provided below.
Thank you,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Dear «TITLE» «NAME»:
On behalf of the Bureau of Justice Statistics (BJS), RTI International is conducting the 2022 Census of Law
Enforcement Training Academies (CLETA). RTI has been reaching out to <> since March.
We hope to receive your completed questionnaire soon so that the census data reflect the variety of training
provided by academies of all types and sizes. Information from your academy is needed to ensure the quality of
the study.
The due date is [DUE DATE].
Please complete the CLETA questionnaire as soon as possible. I understand that you receive a number of survey
requests and I genuinely appreciate your attention to this request.
You may access the questionnaire online at https://bjslecs.org/cleta2022 and entering the following
information:
Username: <>
Password: <>
If you have questions about CLETA, need to change the point of contact at your training academy, or need to
update your contact information, please contact the RTI team via phone or e-mail at <>. or CLETA@rti.org. If you have any general comments about this data collection, please contact me
at 202-598-1036 or Emily.Buehler@usdoj.gov.
Thank you for your time and consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
«TITLE» «ACADEMY HEAD/POC NAME» OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
«Academy Name» has been asked to participate in the Bureau of Justice Statistics’ (BJS) Census of Law
Enforcement Training Academies (CLETA). CLETA data will be used by law enforcement agencies, academy
directors, policy makers, and researchers to better understand and respond to needs regarding training of law
enforcement personnel. No other national data collection can provide comprehensive data on the curricula and
characteristics of training academies. Since CLETA is a census, «Academy Name» cannot be replaced with
another training academy.
I recognize that you may not have received the previous correspondence or that you may not have responded
because of time constraints. I appreciate that your time is limited; however, the reliability of the study directly
depends on the participation of your training academy. The questionnaire includes items that are relevant to all
academies and your responses are essential to our ability to provide the information needed by local law
enforcement and other stakeholders. The CLETA is endorsed by the law enforcement community.
Please complete the questionnaire by using this link https://bjslecs.org/cleta2022 and entering the
following information:
Username: <>
Password: <>
Alternatively, you can submit your data by mail using the enclosed questionnaire and business reply envelope.
The questionnaire due date was [DUE DATE]. Please submit your questionnaire as soon as possible. If you
have questions about the CLETA survey or having difficulty accessing the website, please contact the CLETA
data collection team via phone or e-mail at <> or CLETA@rti.org. If you have any
general comments about this data collection, please contact me at 202-598-1036 or Emily.buhelr@usdoj.gov.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Enclosures: CLETA questionnaire; Business reply envelope; CLETA endorsement letter
Sample Call Script for Data Quality Follow‐up Calls
[IF CALL RINGS TO A GATEKEEPER]
Hello, this is <> calling on behalf of the Bureau of Justice Statistics in the U.S.
Department of Justice regarding the 2022 Census of Law Enforcement Training Academies. I am
following up on a survey invitation that we sent addressed to <>. May I speak with <>?
[IF CALL RINGS TO POC]
Hello, this is <> calling on behalf of the Bureau of Justice Statistics in the U.S.
Department of Justice regarding the 2022 Census of Law Enforcement Training Academies. It is
important that we obtain complete data from all law enforcement training academies. I’m calling now
to confirm that we have everything recorded correctly and completely for your academy. This should
only take a few minutes of your time.
BEGIN READING QUESTION(s) THAT IS (ARE) MISSING INFORMATION OR HAVE INCONSISTENT
RESPONSES.
Thank you for your time.
Sample Call Script for Nonresponse Telephone Calls
[IF CALL RINGS TO A GATEKEEPER]
Hello, this is <> calling on behalf of the Bureau of Justice Statistics in the U.S.
Department of Justice regarding the 2022 Census of Law Enforcement Training Academies. I am
following up on a survey invitation that we sent addressed to <>. May I speak with <>?
[IF CALL RINGS TO POC]
Hello, this is <> calling on behalf of the Bureau of Justice Statistics in the U.S.
Department of Justice regarding the 2022 Census of Law Enforcement Training Academies. A few
months ago, we sent you a letter and an email message inviting your agency to participate in the survey.
We did not hear back from your agency and I wanted to follow up with you to confirm that you received
the request.
Have you received our communications?
[IF YES]
[IF QUESTIONS ABOUT THE SURVEY]
‐
‐
‐
‐
The Census of Law Enforcement Training Academies (CLETA) has been conducted
periodically since 2002; the last time was in 2018 and 2013 before that.
The CLETA measures changes in the content of basic training curricula for new law
enforcement recruits and shows how the nature of law enforcement training has
changed over time.
BJS will use the data collected through this survey only for research and statistical
purposes. Results – at the national level, not at the academy level – will be shared with
law enforcement agencies, academy directors, policy makers, and other stakeholders.
The survey will take approximately 2 hours to complete, including gathering some of the
information and numbers you might need to compile.
[OFFER ASSISTANCE TO COMPLETE]
Is there anything I can do to assist you in completing the survey? A paper version is available
if you would prefer to submit the information by mail.
[IF PROMPTING AGENCY TO COMPLETE ONLY CRITICAL ITEMS]
BJS considers the following questions to be most critical: <>. Would you be
able to provide responses to just those questions? I can record your answers now or
schedule a time to call you that would be most convenient.
[IF AGENCY SAYS THEY DO NOT INTEND TO RESPOND]
Thank you for letting us know. Would you be able to provide responses to just those
questions? I can record your answers now or schedule a time to call you that would be most
convenient. Would you be willing to share with us why you have chosen not to participate?
[IF NO]
Let me review the information we have on file for your agency. [REVIEW E‐MAIL ADDRESS AND
MAILING ADDRESS.]
Ask for the POC’s preferred method of contact and offer to re‐send the information.
«TITLE» «ACADEMY HEAD/POC NAME»
OR CURRENT CHIEF EXECUTIVE
«TRAINING ACADEMY»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
The Bureau of Justice Statistics (BJS) and RTI International (our data collection agent) launched
the 2022 Census of Law Enforcement Training Academies (CLETA) in March and invited
<> to participate in the study. Unfortunately, we have not received
your survey. I recognize that you may not have received the previous correspondence or that
you may not have responded because of time constraints. I appreciate that your time is limited;
however, the reliability of the study directly depends on the participation of your academy.
It is because your agency is so important to the success of the project that we have sent this
letter via UPS instead of standard mail delivery. Please access the questionnaire online at
https://bjslecs.org/cleta2022. Your agency-specific information is:
Username: <>
Password: <>
If you have any general comments about this data collection, please contact me at 202-598-1036
or Emily.Buehler@usdoj.gov.
Thank you for your time and consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
TO: «TITLE» «ACADEMY HEAD/POC NAME»
OR CURRENT CHIEF EXECUTIVE
«TRAINING ACADEMY»
SUBJECT: Census of Law Enforcement Training Academies Reminder
Dear «TITLE» «NAME»:
A few weeks ago, a letter related to the 2022 Census of Law Enforcement Training Academies
(CLETA) was sent to you by mail. We have not received a response from you, so we are sending
this message to request confirmation that we have successfully reached you and encourage you
to contact us if you have any questions related to the data collection.
Please reply to this message to confirm that we have reached <>.
The information contained in the mailed materials is provided below.
Thank you,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Dear «TITLE» «NAME»:
The Bureau of Justice Statistics (BJS) and RTI International (our data collection agent) launched
the 2022 Census of Law Enforcement Training Academies (CLETA) in March and invited
<> to participate in the study. Unfortunately, we have not received
your survey. I recognize that you may not have received the previous correspondence or that
you may not have responded because of time constraints. I appreciate that your time is limited;
however, the reliability of the study directly depends on the participation of your academy.
It is because your agency is so important to the success of the project that we have sent this
letter via UPS instead of standard mail delivery. Please access the questionnaire online at
https://bjslecs.org/cleta2022. Your agency-specific information is:
Username: <>
Password: <>
If you have any general comments about this data collection, please contact me at 202-598-1036
or Emily.Buehler@usdoj.gov.
Thank you for your time and consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Subject: Census of Law Enforcement Training Academies
Dear <Name>:
The International Association of Directors of Law Enforcement Standards and Training (IADLEST)
supports the efforts of the Bureau of Justice Statistics (BJS) and RTI International to conduct the 2022
Census of Law Enforcement Training Academies (CLETA). This survey was sent to all of the
approximately 750 law enforcement training academies in the United States. The study team hopes that
the <<ACADEMY>> will participate in this important project before data collection closes in the next
few weeks.
As of DATE, RTI has not received a survey from <<ACADEMY>>. I write to strongly encourage you
to complete the survey. Information from your academy cannot be replaced. The receipt of information
from each and every academy will greatly enhance the value of the data produced by this project. We
know that you and your staff have many responsibilities and limited time, but we hope that you will
provide the requested information and contribute to this effort. CLETA is the only national data
collection that helps law enforcement agencies, academy directors, policy makers, and researchers to
understand and respond to the training needs of law enforcement personnel.
You may access the survey at https://www.bjslecs.org/CLETA2022 and log-in using the below
credentials:
Username: <<CaseID>>
Password: <<PIN>>
If you have questions about the CLETA survey, the CLETA Help Desk is available via phone or email at [RTI NUMBER] or cleta@rti.org.
Thank you in advance for your cooperation with this important endeavor.
Sincerely,
Michael N. Becar
Executive Director
«TITLE» «NAME» OR CURRENT CHIEF EXECUTIVE
«TRAINING ACADEMY»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear [Title] [Name],
Over the past few months, we have been reaching out to the [ACADEMY NAME] to encourage participation in
the 2022 CLETA. We are in the final stages of data collection and hope to hear from you soon so that the results
of this work reflect the experiences of your agency.
Participation from [State government academies; Municipal police academies; Sheriff’s office academies;
Regional academies, Pre-hire/self-sponsored academies] like yours is particularly important. Data will be
used to produce national estimates describing the basic training curricula, characteristics of trained recruits, and
instructor qualifications of these academies.
Since CLETA is a census, <<ACADEMY NAME >> cannot be replaced with another training academy.
Please complete the questionnaire by using the following link: https://bjslecs.org/cleta2022 and entering
the following information:
Username: «WebUsername»
Password: «PIN»
If you have questions about CLETA or need to update your contact information (including e-mail address),
please contact the CLETA data collection team via phone or e-mail at [RTI NUMBER] 800-845-7883 or
CLETA@rti.org. If you have any general comments about this data collection, please contact me at 202-5981036 or Emily.Buehler@usdoj.gov.
I greatly appreciate your consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
CLETA
Bureau of Justice Statistics
U.S. Department of Justice
c/o RTI International
5265 Capital Boulevard
Raleigh, NC 27616
ATTN: Data Capture
____________________________________________________
ADDRESS SERVICE REQUESTED
*CaseID* <MailStage>‐<CaseID>‐<Control>
«SurveyContactTitle» «FirstName»
OR CURRENT CHIEF EXECUTIVE
«ACADEMY NAME»
«ADDRESS1»
«CITY», «STATE» «ZIP»
Dear «Title» «FirstName»,
«ACADEMYNAME» was invited to participate in the Bureau of Justice Statistics’ (BJS) 2022 Census of Law
Enforcement training Academies (CLETA). Our records show that as of [DATE], we have not received
your completed questionnaire.
If you have completed the questionnaire since [DATE], please accept my sincere thanks.
If you have not completed the questionnaire, please use the following information to log onto the
LEMAS website (https://bjslecs.org/CLETA2022).
Username: «CaseID»
Password: «Pin»
Your response is very important. Your agency cannot be replaced by another agency. As we are
nearing the end of data collection, we kindly ask that you submit your questionnaire as soon as
possible. If you have any questions, please contact RTI International (BJS’s data collection agent for
CLETA) by phone at [RTI NUMBER] or by e‐mail at CLETA@rti.org.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics, Office of Justice Programs, Department of Justice
«TITLE» «NAME» OR CURRENT CHIEF EXECUTIVE
«TRAINING ACADEMY»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
We have made several attempts to contact you over the past few months regarding the participation of
<<TRAINING ACADEMY>> in the Bureau of Justice Statistics’ (BJS) Census Law Enforcement Training
Academies (CLETA). Your responses are vital to the success of the project.
I am writing today to notify you that there are only a couple of weeks remaining to complete the questionnaire.
We must receive your response soon to ensure that the study results accurately reflect the characteristics and
activities of your training academy. The reliability of the study’s results directly depends on the participation of
all law enforcement training academies. Since CLETA is a census, <<TRAINING ACADEMY >> cannot be
replaced with another training academy.
Please complete the questionnaire by using the following link: https://bjslecs.org/cleta2022 and entering
the following information:
Username: «WebUsername»
Password: «PIN»
Alternatively, if you would prefer to complete the questionnaire on paper, we are happy to send you a hard copy
or you may download and print a paper version upon entering your questionnaire access code on the CLETA
questionnaire website.
If you have questions about CLETA or need to update your contact information (including e-mail address),
please contact the CLETA data collection team via phone or e-mail at [RTI NUMBER] 800-845-7883 or
CLETA@rti.org. If you have any general comments about this data collection, please contact me at 202-5981036 or Emily.Buehler@usdoj.gov.
I greatly appreciate your consideration.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
SUBJECT: Census of Law Enforcement Training Academies
Dear «TITLE» «NAME»:
Recently, a letter related to the 2022 Census of Law Enforcement Training Academies (CLETA)
was sent to you by mail. We have not received a response from you, so we are sending this
message to request confirmation that we have successfully reached you and encourage you to
contact us if you have any questions related to the data collection.
Please reply to this message to confirm that we have reached <<TRAINING ACADEMY>>
and that this academy is still operational.
The information contained in the mailed materials is provided below.
Thank you,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
(DATE)
Dear Chief Executive:
The Bureau of Justice Statistics, working with RTI International, is fielding the 2022 Census of Law
Enforcement Training Academies (CLETA). Conducted periodically since 2002, the CLETA is the only
national-level data collection providing a complete enumeration of state and local academies providing basic
law enforcement training in the U.S.
Recognizing the importance of this work, the nation’s law enforcement community as represented
by the Major County Sheriffs of America, Major Cities Chiefs Association, and the International
Association of Directors of Law Enforcement Standards and Training provide their full support of
these efforts. These organizations recognize the importance of this national data collection and encourage
their members and other law enforcement training academy professionals to participate.
Your participation will help ensure that the 2022 CLETA is a success and that the results can be used by law
enforcement, policymakers, academy directors, and researchers with confidence. We know that your staff
have many responsibilities and limited time, but we hope that you will provide the requested information and
contribute to this effort. No other data collection provides such a complete account of law enforcement basic
training curricula, instructors, and recruits.
We thank you in advance for your participation in this important study.
Sincerely,
Sheriff Dennis M. Lemma, President
Major County Sheriffs of America
Michael N. Becar, Executive Director
International Association of Directors of Law
Enforcement Standards and Training
Laura Cooper, Executive Director
Major Cities Chiefs Association
Subject: 2022 CLETA Survey Submission
Dear <<SurveyContact>>,
Data collection for the 2022 Census of Law Enforcement Training Academies (CLETA) will close soon. We
noticed you provided data in the online survey system for the <<ACADEMYNAME>> on <<DATE>>;
however, you did not submit the form. Due to the survey deadline quickly approaching, we ask that you
please log back in to complete and submit the questionnaire as soon as possible.
Thank you for already providing responses to some of the survey questions. In the next few weeks, we
will process your survey on your behalf so that we can accept those data.
If you have questions about CLETA, please contact the CLETA data collection team via phone or email at
[RTI NUMBER] or CLETA@rti.org.
Sincerely,
Ryan Weber
RTI CLETA Project Director
Email: CLETA@rti.org
Phone: [RTI NUMBER]
«TITLE» «NAME»
OR CURRENT CHIEF EXECUTIVE
«AGENCY»
«ADDRESS1», «ADDRESS2»
«CITY», «STATE» «ZIP»
Dear «TITLE» «NAME»:
On behalf of the Bureau of Justice Statistics (BJS) and RTI International, I would like to thank
you for your participation in the 2022 Census Law Enforcement Training Academies (CLETA).
I truly appreciate your support in completing this survey. Your participation ensures that we are
a step closer to providing a complete enumeration of the nation’s law enforcement training
academies.
This letter confirms that we have received your survey and are currently processing the data.
RTI will contact you if there are any questions about the answers your agency has submitted.
We anticipate all survey responses will be collected by the end of September 2023. A copy of
the report will be available through BJS and the CLETA website in 2024.
If you have any general comments or questions, please feel free to contact me at 202-598-1036
or Emily.Buehler@usdoj.gov. If you have questions about CLETA, need to change the point of
contact at your agency, or need to update your contact information (including email address),
please contact the CLETA support team at [RTI NUMBER] or CLETA@rti.org.
Sincerely,
Emily D. Buehler, Ph.D.
Program Manager
Bureau of Justice Statistics
Census of Law Enforcement Training Academies
(CLETA)
Cognitive Interview Report
PREPARED BY:
RTI International
December 12, 2022
1
1.
Introduction
In preparation for the administration of the 2022 Census of Law Enforcement Training
Academies (CLETA), RTI conducted several activities to refine the existing 2018 CLETA
instrument. These activities included:
1. A review and analysis of data obtained from the 2018 CLETA, involving an evaluation of
item nonresponse and validation error rates observed when data were initally provided by
respondents, which was prior to data quality follow-up (DQFU), as well as an evaluation
of DQFU activities and results.
2. A synthesis of feedback obtained from a series of five CLETA stakeholder expert
meetings.
3. A synthesis of feedback obtained from cognitive testing of a revised survey instrument.
This report describes the issues found during cognitive testing and recommended changes to the
instrument following a series of interviews conducted to test the latest iteration of the 2022
CLETA survey instrument. Additionally, the final changes to the instrument are provided for
each quesiton.
2.
2.1
Methodology
Design and Procedure
Participants were recruited by email and phone using a list of academies provided by BJS. All
efforts were made to ensure an even distribution among the five strata (state government,
municipal police, sheriff, regional, and pre-hire/self-sponsored academies) that would participate
in the 2022 CLETA.
Since the full CLETA instrument would be burdensome for an academy to complete for a
cognitive interview, we asked the academy contact to complete half of the questionnaire. Two
cognitive interviewing protocols were developed, each containing scripted and concurrent
probing for approximately half of the CELTA instrument. The protocols are appended to this
report beginning on page 43. Participants were randomly assigned to one of two groups and the
corresponding questionnaire was sent to them to complete in advance of the interview. After
receiving the completed questionnaire, RTI cognitive interview staff reviewed the responses in
advance of each interview.
2.2
Participant Interviews
Interviews were conducted in August and September of 2022 and were administered over Zoom
by one of five RTI cognitive interviewers. Each interview was expected to last about an hour.
While participants were not specifically asked to think aloud while answering the questions, they
were encouraged to comment on anything that was confusing or unclear to them.
After reviewing the findings of the initial 16 interviews, we determined that we had reached
saturation with several survey items, as similar comments were being received. RTI and BJS
agreed to develop an abbreviated protocol, comprised of 6 survey items that we felt could still
2
benefit from additional interviews. As a result, we continued recruitment efforts with potential
survey respondents to see if they would be willing to take part in a 15–20-minute shortened
interview about these 6 items only. Ultimately, 8 participants participated in the Group 1
interview, 8 participated in Group 2, and 8 received the shortened questionnaire, for a total of 24
completed interviews.
The following table shows the distribution of completed interviews by type of academy.
Academy Type
1. State government academies
2. Municipal police academies
3. Sheriff’s office academies
4. Regional academies
Group 1
4
0
1
1
Group 2
2
2
1
2
Short
Questionnaire
1
2
2
0
2
8
1
8
3
8
5. Pre-hire/self-sponsored academies
TOTAL
Total
7
4
4
3
6
24
Initially, we established targets of 6 completed interviews from each type of academy (3 from
Group 1 and 3 from Group 2) for a total of 30 interviews. Due to recruitment challenges, we fell
short by 6 interviews but felt that saturation was met with the survey items that were tests.
3
3.
Question-by-Question Findings and Recommendation
The following are question by question findings and recommendations. The question presented
during cognitive testing is displayed first, followed by the findings, recommendations, and if
changes were made to the question, the final version of the question.
Cognitive Interview Findings:
This question was not presented to the cognitive participants. No changes will be made to this question
for the 2022 CLETA.
4
Cognitive Interview Findings:
The participants did not have any difficulty when answering this question. The definition of BASIC
training was understood by participants and no further challenges where identified.
Cognitive Interview Recommendations:
No recommendations. No changes will be made to this question for the 2022 CLETA.
Cognitive Interview Findings:
The participants understand the question’s intent and its concepts without difficulty. One participant
mentioned that the last sentence, an example of CALEA and IADLEST, was useful to understand what
was being asked.
Cognitive Interview Recommendations:
No recommendations. No changes will be made to this question for the 2022 CLETA.
5
Cognitive Interview Findings:
The concept of "partner institution" was understood by most participants. However, one participant
from a state agency found it confusing since their academy is not associated with a college or
university. Nonetheless, he understood the intent of the question.
When participants were asked about the intent of the question, they explained that the recruits would
receive credit for completion of basic training.
Cognitive Interview Recommendations:
To provide further clarification of what is meant by partner institution, consider adding an example,
(e.g., local 2- or 4-year college or university). In addition, we recommend a slight change in question
wording for improved clarity:
In 2022, did your academy offer an associate’s or bachelor’s degree or offer college credit at a partner
institution?
Final Version of the Question:
6
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
7
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
8
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
Most participants understood the question and were able to provide an appropriate response.
One participant mentioned that they are centralized; that they are the post and the academy.
Therefore, their numbers are the same for "a" and "b".
The participants also mentioned that gathering the information was straightforward and not
burdensome.
When asked to define what "core BASIC training" meant, most participants agreed that it is
the "Completion of the curriculum required to receive certification", "The minimum
requirements from the state", or “What is required to receive certification.”
Participants acknowledged that they read the instructions and only considered the hours for
the requested information, and because POST requires the value in hours, most respondents
found it easy to provide a response in hours.
Cognitive Interview Recommendations:
Based on cognitive testing, the question appears understood by participants. Therefore, we
would not recommend any additional changes. However, for consistency, we recommend
either adding "core" to question 8a as the question references "core BASIC training" or
removing the term from the question.
Final Version of the Question:
9
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
Most participants understood the term "field training" as the "practical portion of the training
component.” One participant explained that the recruit is assigned to a field training officer for a set
number of hours. Another participant from a state agency explained, "Field training is what occurs post
academy in terms of the recruits being evaluated or instructed while working on the street.” To this
participant, field training could mean a lot of different things and adding a definition of "Field training"
would be helpful.
In terms of the unit of measurement, academies all measure this activity in either weeks or hours. For
those that measure in weeks, they can easily report in hours, if needed.
Cognitive Interview Recommendations:
Considering the participants seemed to understand what was meant by field training, we do not
recommend additional edits to the question or feel a definition is necessary. No changes will be made
to this question for the 2022 CLETA.
10
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
Cognitive Interview Findings:
Some participants thought the question was clear, several stated that concepts used in the question
were unclear.
The participants mentioned that they were unclear about the meaning of "community members". One
participant asked, “Do you mean community members not affiliated with our office in some capacity or
the civilians that teach here?" Another asked, "Citizens from the street or citizens with interest in the
academy?”
11
When asked about their interpretation of formal and informal methods, the participants gave different
responses, "I see ‘formal’ as having an advisory committee, with a board that provides oversight. While
‘informal’ would include guest lecturers or scenarios. In these cases, we have instructors that source
people and volunteers, but it is not a formalized process." Another participant mentioned that he thinks
"formal” is either some type of meeting where you are sitting down with community members and
speak to them about training content or some type of survey that is provided, while “informal” could
include random citizens or other people providing written feedback or comments on Facebook or
something similar.
Two other participants mentioned that they did not understand the concept of "formal/informal
methods". One participant mentioned that he would benefit from receiving examples of these terms.
Note: with two alternative measures of this question that were provided to CLETA stakeholders in the
expert meetings, there was also confusion about who should be included in the term “community
members”. This documentation is provided on the last page of this report.
Cognitive Interview Recommendations:
Considering the challenges participants had in interpreting the meaning of “community members” and
“formal” versus “informal” methods, we recommend adding examples of community members and
definitions or examples of formal and informal methods to the question.
Specifically, we recommend the following examples be included after the term ‘community members’ in
the question text: “(e.g., local residents; members of the general public; community
advocates/representatives)”. Additionally, based on the LEMAS post-academy survey, we recommend
the following definitions be included as notes to the item:
a. Examples of formal methods include: involvement in an advisory board or commission report;
having a contractual position/arrangement.
b. Examples of informal methods include: opportunities to offer feedback/input online or inperson; having a non-contractual position/arrangement.
For response option b, we recommend removing the text “Involvement in” since the question already
uses the term “involve” and removing the term “guest” from the example since this is captured in
informal methods.
Final Version of the Question:
12
Cognitive Interview Findings:
13
This question was understood by most participants, and they reported that gathering the information is
a straightforward process.
Some of the participants that reported difficulty were from larger academies, where it was hard to
determine where the personnel should be entered since many come in just for their training portion and
are not full-time staff. However, they don’t fit within the part-time category either. Some participants
were confused because they have full-time officers who are brought in to help with areas of expertise
which would be considered part-time. In one case a participant was unsure of who to count because
their academy "might have instructors coming in from other agencies that are paid by that agency."
Most participants explained that retrieving the data needed to complete the question was easy, except
for a few large academies, where they reported having about 150 part-time adjunct trainers. These
latter participants explained that they have the total number but breaking down the part-time trainers
would be difficult. They would need to review background sheets on each instructor to assign them into
each category, which would be burdensome.
When asked if there could be instructors who would fit into multiple categories, the participants
explained that they could include each instructor into one category. However, one participant explained
that there could be potential overlap between the "Not employed” and “Employed by academy”
categories since someone from a law enforcement agency may be paid by their agency to come
support their academy training. However, they could also come here on their own time and be paid by
the academy as an adjunct/contractor. Another participant asked, “Employed vs Not employed...do you
mean employed by the agency or employed by our academy?”
Finally, three participants mentioned that they considered all job responsibilities (not just their training
role) when answering this question. One participant explained, "In terms of the number of hours they
work here, I would determine it by how they are hired, since full-time staff receive benefits, while parttime staff do not."
Cognitive Interview Recommendations:
One point of confusion that arose during cognitive testing was the difference between being a full-time
employee and a full-time trainer or instructor. The question seems to define full-time status based on
overall employment and not hours dedicated to the academy. Consider changing the instruction under
the question to: Full-time trainers or instructors are those regularly scheduled for 35 hours or more per
week at the academy.
Final Version of the Question:
14
15
Cognitive Interview Findings:
The question format was clear to participants, and they understood that they should select only one
response per column.
When asked about the difficulty of responding to the question, participants had to make some
interpretations. For example, two participants mentioned that they had no minimum educational
requirements, but a high school diploma is a minimum requirement for law enforcement certification in
the state, so they selected that option. One participant had problems interpreting "employed by your
academy," and another participant mentioned education could be substituted with experience as an
officer, so they selected “No minimum education required for our academy trainers or instructors.”
Two participants did not see the instruction under the question stem to check the box if they did not
have full or part-time instructors when they should have checked it.
Cognitive Interview Recommendations:
We recommend removing the instruction below the question stem and adding a row for: “Not applicable
as we do not employ these staff” with radio buttons for each column. Include an instruction: Select “Not
applicable” if your academy does not employ full-time or part-time trainers or instructors.
For example:
16. In 2022, what was the minimum formal education requirement for trainers or
instructors employed by your academy? Please select one response for each column.
Select “Not applicable” if your academy does not employ full-time or part-time trainers or
instructors.
Full-time
trainers and
instructors
Graduate degree required
Four-year college degree required
Two-year college degree required
Some college but no degree required
High school diploma or equivalent required
No minimum education required for our academy
trainers or instructors
Not applicable - academy does not employ these
staff
Final Version of the Question:
16
Part-time
trainers and
instructors
Cognitive Interview Findings:
There were no challenges identified by participants when answering this question.
When experience was required, participants agreed that the number of years is easy to provide, given
that is a requirement made clear by the agency. In some cases, the requirement is set by the HR
posting, while other participants explained that it is an internal requirement.
Two participants mentioned that they have minimum requirements, but they are measured by skills, not
in years. One participant explained, "They need job related experience but no specific number of years
of experience are required.” One participant mentioned they have an hourly requirement, so he had to
do the math to convert it to years. Another participant checked the wrong N/A box, because he
mentioned that he saw "part-time" underlined, and looked at it but ignored the rest, so was not aware
that it was for no part-time instructors.
17
Cognitive Interview Recommendations:
We recommend removing the instruction below the question and adding a row for: “Not applicable as
we do not employ these staff” with radio buttons for each column. Include an instruction: Please select
“Not applicable” if your academy does not employ full-time or part-time trainers or instructors.
As for the skills requirement in lieu of years of experience, if this is of interest to BJS we could include a
row with the following response option: Do not require experience as a sworn officer but do require skill
proficiency. If this option is taken, we suggest adding “or skill proficiency” to the last row before the
word “requirements”.
Final Version of the Question:
18
Cognitive Interview Findings:
For question 18, we probed for understanding and feedback on the format of the question, given that it
has a different structure than the rest of the instrument (in terms of the horizontal orientation of FT/PT
as opposed to the vertical orientation presented in other questions).
In terms of general understanding, most participants mentioned that the question was clear and
explained that the question was asking about the "state requirements to be an instructor." Two
participants expressed confusion regarding the "Other" category, since they were not sure what other
certification requirements there were. One participant thought about including academic experience in
the "other" category. One participant suggested that we add "Instructor certification" as an example of
“Subject matter or specialty certification.”
Three participants provided responses in both tables when they should have only completed it for
either part-time or full-time.
We presented participants with two alternate versions that included the FT/PT categories in the
standard vertical format. Most participants preferred version B which presents Yes/No response
categories by FT and PT in a vertical format.
Cognitive Interview Recommendations:
Considering most participants preferred the vertical format presented in Alternate B, we recommend
using this format.
We recommend removing the instruction below the question and adding a row for NA - academy does
not employ these staff.
We recommend keeping the other certification requirement field in case there are write-in requirements
which could be upcoded during analysis.
For example:
18. In 2022, were the following certifications required for your academy’s trainers or
instructors? Please enter a response for each row for the full-time and part-time columns.
Please select “Not applicable” if your academy does not employ full-time or part-time trainers or
instructors.
a.
State- or POST-certification
b. Academy/instructor certification
c.
Subject matter or specialty certification (e.g., firearms
certified, driving certified, first aid certified, etc.)
d. Other certification requirement (please specify):
e.
Not applicable - we do not employ these staff
19
Full-time
trainers and
instructors
Part-time
trainers and
instructors
Yes
Yes
No
No
Final Version of the Question:
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item.
Final Version of the Question:
20
Cognitive Interview Findings:
Participants were asked what they thought “general instruction” meant in the context of this question.
The participants interpreted this as, "Overall instruction or training to instructors to enhance their
teaching skills.”
One participant mentioned that he was confused as to what it meant, but his interpretation, "Craft of
instructing," was in line with the intention of the question.
Cognitive Interview Recommendations:
No additional recommendations for this item.
Final Version of the Question:
21
Cognitive Interview Findings:
Of the 8 academies that received question 20, only one participant received question 21 since its part
of the skip logic from question 20. No comprehension problems were mentioned.
Cognitive Interview Recommendations:
No additional recommendations based on cognitive testing. Because this item was not presented in the
expert meetings and only 1 cognitive interview participant provided feedback on the item, we suggest
BJS consider the item’s analytic value. Please also reconsider the addition of the term “internallyprovided” to this question.
Final Version of the Question:
22
Cognitive Interview Findings:
All the participants mentioned that the list was clear, including “Use of force training simulator” and
“Mental health services.”
Some participants mentioned that we should add "Laptop computer" to the list, given that they provide
laptop computers to their recruits. This could be a stand-alone category or combined with question A2.
A couple participants mentioned they were unfamiliar with question A5 "Mobile data terminals.”
We asked participants whether they understood "Use of force training simulators" as including firearms
simulators as well as other types of scenario-based training. The participants understood the concept,
which was defined as broader than the FATS simulator.
When asked whether they only considered resources available within the academies or also
considered external resources, most participants mentioned that they thought about both, especially
those academies located within a college.
23
Cognitive Interview Recommendations:
Consider adding "Laptop computer" in question A2. For example: “Laptop computer/electronic
tablet/iPad”.
Consider whether "Mobile data terminal" should be renamed, clarified, or an example provided.
Final Version of the Question:
24
Cognitive Interview Findings:
We asked participants what they would include in the "total operating budget”, and they provided
responses such as, “Payroll, salaries, and total budget of what it costs to run the program.”
Some of the participants provided estimates, but overall, they felt it was relatively easy to determine a
value to report, regardless of the academy stratum.
One participant explained their reason for estimating the total value was that they knew their total
budget, but given the instructions on the survey question, they had to remove some items such as
equipment, and that detail resulted in an estimate. In was clear to some participants that dependent on
the size of the academy, and who is answering the survey, the question burden can increase.
Cognitive Interview Recommendations:
The participants generally felt this question increased their burden but was not as challenging as the
next item asking about their equipment budget. While we expect some difficulty in obtaining this value
and a higher rate of estimation, we do think the quality of the total operating budget estimate will be
sufficient to justify keeping this item and we recommend its retention in the 2022 CLETA if it is
determined to be analytically useful to stakeholders.
Final Version of the Question:
Cognitive Interview Findings:
Most participants mentioned that this was a difficult question to answer given that many of them do not
have access to a detailed budget.
The participants explained that it was difficult to provide an accurate value, and in most cases the value
provided was an estimate, as it would require a significant amount of time for academies to collect the
data. One participant explained, “This requires someone to do the calculations and would take time to
split it all out.” Another participant explained, "As a satellite academy, I don't have a separate budget."
Another felt that "Estimating the life expectancy of an item is hard to assess,” explaining that he really
doesn’t know what is going to last 5 years or less. This was in reference to the definition of equipment
provided in the question.
Cognitive Interview Recommendations:
The participants generally felt this question was more burdensome to answer compared to the item
asking about the total operating budget. We should consider removing this item from the 2022 CLETA
as it appears the level of burden would be high and accuracy of the estimate to be a poor
representation of the true equipment budget for the academy.
25
Final Version of the Question:
It was decided that this question will be removed from the 2022 CLETA.
Cognitive Interview Findings:
Most participants reported that gathering this information was easy, depending on the systems they
have in place. Most provided exact values and did not have problems understanding the instructions
for the estimate box.
Some mentioned that they had difficulty answering the question since training programs are not
necessarily aligned to a calendar year. One participant mentioned that numbers could contain
duplicates since "Students that fail can start again, and in those cases the database counts them
again.”
When asked about how they would collect other gender categories, like "non-binary" or other, most
participants mentioned that they do not collect this information but acknowledge the importance of
doing so going forward. One participant explained, "We do not collect at this time, but know we have to
consider that going forward. We have not had someone designate themselves in that manner yet but
would support the addition of a non-binary category.” In some cases, they are making ongoing changes
to quantify it, while in other cases they measure gender as an open field or have an, "other" category.
Two participants mentioned that the values reported are not self-reported from participants, but
observations made by the academy or through medical examination.
26
Cognitive Interview Recommendations:
It should be determined whether it is acceptable to have duplicate records for those who start/fail and
restart the program. If we ask academies to exclude these, the burden of reporting would be higher
along with the estimate rate. An instruction could be included within the question.
As for the non-binary categories, we do believe this should be included in the 2022 CLETA considering
academies believe they will see more non-binary recruits in the future and the addition of the category
to BJS data collections could increase its measurement.
We recommend revising the estimate boxes to be consistent with previous recommendations and
including an instruction in the question.
For example:
25. For the BASIC recruit academy class(es) that ended in 2022, please indicate the total
number of recruits who started BASIC training, and the total number who completed it,
by sex. Please check the estimate box if an estimate was provided.
Number of recruits who
STARTED training
a.
Number of recruits who
COMPLETED training
Male
Estimate provided
Estimate provided
Estimate provided
Estimate provided
Estimate provided
Estimate provided
b. Female
c.
Other / Non-binary
d. Total for classes ending in
2022 (Sum of Rows a
through c)
Final Version of the Question:
27
Cognitive Interview Findings:
Several participants explained that they do not collect this information.
In some cases, participants answered the question based on their own interpretation of recruit’s
race/ethnicity. A few participants explained that they will make race/ethnicity decisions based on review
of recruit photographs.
For several academies that do record race, they explained that they only can provide a single race and
not provide two or more races for multi-racial recruits. Some participants mentioned that they collect
race demographic information, but not with the start/completed distinction, so it took time for them to
answer this question.
Rather than entering information in the "Not known" field, some participants tend to estimate a value
based on observation.
Cognitive Interview Recommendations:
It is concerning that participants are self-identifying the race/ethnicity of their recruits. There seems to
be a high level of burden in capturing accurate values when this is possible. This is clearly an
analytically useful item to preserve. Considering the importance of this item, we should retain it but may
have to accept an increased level of error in reporting. We recommend adding an estimate box below
each response.
Final Version of the Question:
28
26a. In 2022, how many recruits, upon entering BASIC
training …?
a.
Recruits
… were military veterans?
b. … possessed an associate’s degree?
c. … possessed a bachelor’s degree?
Cognitive Interview Findings:
This new question was presented to twelve participants. When presented with the proposed question,
most participants explained that they do collect this information.
Veteran status was tracked by most academies; two participants reported that they did not collect the
information or that the information was collected informally or by the external hiring agency.
Data on recruits possessing an associate degree or bachelor degree was less often available, but most
participants mentioned that they could get that information. Four participants mentioned they do not
track this information. One participant explained, “Recruits are hired by the agency (not the academy),
so we don’t have record of this information and could not provide an estimate."
Agencies that do collect this information explained that it was not a burdensome process to obtain
these data and the information can be easily retrieved.
Cognitive Interview Recommendations:
The question may result in a moderate amount of burden on participants, but stakeholders felt these
data would be analytically useful. If it is decided to retain this item for the 2022 CLETA, we can identify
a way to measure whether the information is collected and whether the number provided is an
estimate.
Final Version of the Question:
29
Cognitive Interview Findings:
The participants all explained that they do record this information and it was easy to access.
Furthermore, they would be able to provide an exact value.
One participant explained that they could not break this down by gender. One participant mentioned
that he did not include someone who had to drop from training because an injury that happened prior to
coming to the academy. He interprets this question about people leaving because of something that
happened during their time at the academy. Another participant explained that there was overlap
between rows "a" and "i", given that an agency could withdraw a recruit due to an injury.
30
One participant mentioned that the list could add a “Skills-based or Scenario-based” failure to qualify
and that this should be added to the list.
One participant mentioned that he did not include 8 or 9 people that did not pass their physical test on
the first week, given that it is their first chance to weed people out. The recruits are enrolled, but since
this withdraw occurs in the first week of academy, they would not consider them as non-completion of
the program.
Another participant mentioned that if someone goes out for any reasons, but comes back, then he
would not include those.
In sum, there appears to be some complexity in how participants would complete the table but that the
data are available. The skills-based failures mentioned by one participant would be captured under
27b-27d.
Cognitive Interview Recommendations:
Following one cognitive interview participant raising a concern about not being able to break down the
total values by sex/gender, we considered how one might respond in the case that they can count the
number or recruits who do not complete training by reason but cannot disaggregate those counts by
sex/gender. We worry in this case respondents could provide inaccurate data. We would like to discuss
ideas for addressing this concern with BJS.
Consider underlining the term “primary reason” given the length of the item and confusion from one
participant about the categories not being mutually exclusive.
Final Version of the Question:
31
Cognitive Interview Findings:
The participants all understood the question and the term "training philosophy". When asked what this
meant to them, they provided examples of their approach to training. One participant explained, "What
is our approach as an academy in the stress/nonstress model. We have a high academic component in
contrast to a physical/force component."
Cognitive Interview Recommendations:
No additional recommendations. No changes will be made to this question for the 2022 CLETA.
32
33
Cognitive Interview Findings:
Most of the participants explained that the items from the list are clear and easy to understand. One
participant mentioned that the term "Intelligence gathering" was too broad, so he was unsure what to
consider.
Participants explained that they would need to provide an estimate for several of the subject areas,
mainly those where there is no specific class about the topic, but it is a topic covered across several
lessons. One participant mentioned that it was an overwhelming question because it requires him to
break down 600+ hours into distinct categories.
When asked about the difficulty in reporting hours, the participants mentioned that there were topics
that overlap (such as de-escalation). For these, most participants selected and provided an estimated
value.
When asked about how they calculate the number of hours, several participants mentioned that the
"numbers are coming straight from the lesson plan," and "I would probably just count the blocks to
classroom training and then add hours where these are practiced within the scenarios.”
If a topic directly relates to a class on the lesson plan, participants explained that the value provided
was exact as opposed to an estimate. Values reported for topics covered across subject areas were
likely estimates.
Participants mentioned overlap across several topics such as:
Defensive tactics and less lethal weapons for example could have subjects that fall into both
those categories;
Intel gathering and analysis could also overlap with investigations;
Mediation/conflict management overlaps with de-escalation/verbal judo and talking someone
down can be considered mediation; and
34
Crime mapping may have a little overlap with research.
For cases with overlapping topics, participants reported that they did their best estimate to divide the
hours, but there is some overlap between them.
Cognitive Interview Recommendations:
Based on the initial data review and stakeholder engagement, we knew the challenges in terms of
overlapping topic areas resulting in a higher estimate rate. There may not be much we can do to
address this as it will require some calculations by respondents. However, it appears that they will
provide their best estimate based on how they map their curricula with the categories we provide.
Consider providing some examples of “Intelligence gathering and analysis.”
Final Version of the Question:
35
36
37
Cognitive Interview Findings:
We asked what participants thought "Formal assessments" meant in the context of this
question and they thought it meant the methods used to evaluate and grade recruits. One
participant said that it is a "Measurable mandate the state has for the academy." However,
one participant felt we needed to include a definition of "formal assessments," as they
selected all the assessments; this participant also noted difficulty calculating the number of
assessments. Traditional classroom exams are easy to quantify, but recruits are being
evaluated all the time. Another participant explained that they are continually evaluating the
cadets, but also use these as teachable moments.
The participants felt that the names of the formal assessments match what they call them at
their academy. However, one explained that the state competency exam can mean a lot of
different things. He explained, "You can have a state competency exam in physical fitness, in
written or scenario-based assessments dependent on where you are in the country. State
competency is a broad term that may encompass sub items A-E."
When asked how they would define the number of required assessments, one participant said
that they went by number of grades that a recruit would receive under each category. Other
participants based it on how it is defined by the state POST or by the number of critical skills
recruits must pass. One participant explained that they would provide an estimate as the
number may vary. In general, participants felt they could provide the number of formal
assessments required.
Cognitive Interview Recommendations:
Consider adding a definition of “Formal assessment” as “A graded test or exam used to evaluate
performance and required to complete basic training.”
Consider removing “State competency exams” as these are likely to overlap with existing categories.
Final Version of the Question:
38
Cognitive Interview Findings:
When asked what “role play scenarios and activities” meant to the participants, they explained that you
had to physically interact or pretend to respond to active shooter, arrest someone, or something
similar. One participant explained that it, "puts a cadet into a semi real life environment to practice their
skills."
The participants generally preferred the term "Role play scenarios and activities" compared to “realitybased (mock) scenarios” as it seemed clearer to them, and they felt the activities in the list were clear.
Cognitive Interview Recommendations:
No additional recommendations. No changes will be made to this question for the 2022 CLETA.
39
Cognitive Interview Findings:
For one participant, the meaning of "weapons training” was unclear. Specifically, the participant was
not sure if testing proficiency and qualifying the recruit was a required aspect of “training,” or if
classroom lecture/demonstration was sufficient. At their academy, they talk about all of these weapons
and introduce them to recruits so they know how to render them safe, but they do not train and qualify
on them. This participant interpreted the question as requiring training, testing proficiency, and
qualifying.
Another participant thought "Revolver" was an antiquated term.
Cognitive Interview Recommendations:
If the intention of the question is to learn whether the academy qualifies recruits on the use of each of
these weapons, then the question makes sense as written. However, if classroom training on the topic
is sufficient (i.e., not necessarily a skills assessment), we would need to clarify this in the question.
No changes will be made to this question for the 2022 CLETA.
40
Cognitive Interview Findings:
Some participants felt that "Use of force training simulator" was preferrable to “Firearms training
simulator” considering the various simulations that can be done without firearms.
However, one participant suggested "Use of force/firearms training simulators"
One participant was not sure what “Training with off duty weapons” meant.
Cognitive Interview Recommendations:
No additional recommendations. No changes will be made to this question for the 2022 CLETA.
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
41
Cognitive Interview Findings:
Cognitive interview participants were not asked for feedback on this item. No changes will be made to
this question for the 2022 CLETA.
42
Appendix A – Cognitive Interview Protocol - Group 1
P#________
DATE: ___________
START TIME: ______
INTERVIEWER: ______
Section 1: Introduction
Hello, I’m [NAME] with RTI, speaking with you on behalf of the Bureau of Justice Statistics.
Thanks for agreeing to help us develop the questionnaire for the 2022 Census of Law Enforcement Training
Academies, or CLETA. The call will take about an hour. I’d like to start with a short summary of the goals for
today’s call and explain a bit about how I’ll conduct the interview.
‐
‐
IF NO LONGER A GOOD TIME, OFFER TO RESCHEDULE
IF STILL GOOD TIME, CONTINUE
As you may know, the Bureau of Justice Statistics and RTI are preparing to conduct the CLETA in early 2023. As
we get ready for the study, we are asking academy staff like you to review the draft questionnaire. During this call,
I’ll ask for your reactions and feedback to these draft questions– including things like how the questions are worded,
your understanding of concepts, ways to clarify instructions, and the information your academy tracks that is related
to the survey questions.
Your participation in this interview is completely voluntary. You may stop at any time or skip questions that you do
not want to answer. The data we collect will be de-identified, meaning that your personal information, such as your
name, will never be stored together with your responses. With your permission, I would like to record this call for
the sole purpose of capturing comments that I may miss when taking notes. The recording would only be accessible
to me and would be deleted no later than 2 weeks from this interview when the responses from this call are finalized.
Do I have your permission to record this call?
Please keep in mind that there are no right or wrong answers to my questions. One of our main goals is to draft
questions that make sense, so if anything about the questions is confusing or unclear, you can help by pointing this
out to me. Also, if you’re not sure how you would respond to any of the questions, please tell me that, too. Finally,
please let me know if you believe any questions or topics were omitted, or any answer choices or response
categories that were missing or insufficient.
I am interested in hearing all of your feedback on the survey, but because there are a lot of topics to discuss and we
only have an hour, sometimes I might ask that we move on to the next question before you’ve had a chance to share
everything on your mind. At the end of the interview, you can share any important feedback that you didn’t have a
chance to share earlier.
Do you have any questions before we begin?
Before we start discussing the questions, did you complete the questionnaire?
‐ IF YES: Do you happen to recall approximately how much time you spent completing the questionnaire?
Please include the time you and any others at your academy spent gathering information needed to answer
the questions.
‐
__________ HOURS __________ MINUTES Go to Section 2.
IF NO: [INTERVIEWER: RECORD NOTES ABOUT REASONS OF NOT COMPLETING THE
INSTRUMENT. RECORD ANYTHING THE R MENTIONED EARLIER OR AFTER SAYING THEY
DID NOT COMPELTED IT]
Section 2: Item reviews
43
I’m planning to discuss only some of the questions on the questionnaire, but if you have comments or concerns
about any of the questions I skip, please feel free to share them with me at any time.
P1. (BOTH) The first question I’d like to discuss is Question 2. It asks about BASIC training. Please read the
question to yourself.
PROBE: Do you have any difficulty understanding the definition of BASIC training?
PROBE: Is there anything you would add/remove to the definition of BASIC training?
P2. (GROUP 1) Now let’s review Question 3.
PROBE: Is it clear that this question asks about accreditation from a body that is not affiliated with a
state POST requirement?
P3. (BOTH) Next I’d like to review Question 4 which asks about degrees and college credits.
PROBE: Is it clear what “partner institution” means here?
PROBE: Do you interpret this question to mean degrees and/or college credits are awarded based on
completing BASIC training?
P4. (BOTH) Now we will move on to Question 8. It asks about the length of your academy’s BASIC training
program.
PROBE: What do you think of this question?
PROBE: Can you tell me how you went about answering this question?
PROBE: What does, “core BASIC training program” mean to you?
PROBE: How do you interpret, “TOTAL number of hours required to complete basic training”?
How about, “Number of hours mandated by state POST…”?
PROBE: Did you read the instruction under the question? Did your response include any of
these?
PROBE: Does your academy measure the length of BASIC training in hours?
[IF NOT] What process would you use to convert your answer if you needed to report the length of your
program in hours?
44
P5. (BOTH) [ONLY ASK IF Q10 = YES] Please now read aloud Question 11. It asks about the length of field
training for academies that oversee these programs.
PROBE: In your own words, what does, “…core field training program” mean to you?
PROBE: Does your academy measure length of its field training program in hours?
[IF NOT] What process would you use to convert your answer if you needed to report the length of your
program in hours?
P7. (BOTH) Now we are going to review questions related to academy personnel. We’ll start with Question 15.
Please read the question and each category to yourself. Let me know if there are any descriptions that are
unclear or difficult to understand.
PROBE: Does your academy record this information separately for full-time and part-time instructors?
PROBE: How difficult is it to obtain this information for each of the categories listed in the table?
PROBE: Do any of your trainers or instructors fit within more than one category?
[IF YES: PROBE TO FIND OUT IF THEY REPORTED THEM IN MULTIPLE ROWS. IF ONLY
REPORTED IN ONE ROW, HOW DID THEY DECIDE WHICH ROW?]
PROBE: When determining who to count as full-time and who to count as part-time, did you think about
their status as a trainer or instructor, or did you think about their status with respect to all job
responsibilities, not just training?
P8. (BOTH) Now let’s go to Question 16. This question asks about education requirements for full and part time
instructors.
PROBE: What challenges, if any, did you have in answering this question?
a. PROBE: Was it clear that you could only provide one response for each column?
P9. (BOTH). Question 17 asks about requirements for instructors as sworn officers.
PROBE: Did you have any challenges answering the field for, “Sworn officer status…”?
PROBE: [IF SWORN OFFICER OPTION IS SELECTED, DETERMINE HOW NUMBER OF YEARS
OF EXPERIENCE WAS CALCULATED]
P10. (BOTH) Please now turn your attention to Question 18. It asks about certifications required for your
academy’s instructors.
PROBE: Can you explain to me with your own words what this question is asking about?
45
PROBE: Is the format of the question clear? [PROBE TO SEE IF ITS CLEAR OR UNCLEAR THAT
AN ANSWER IS REQUIRED FOR EACH ROW AND THE DISTINCTION OF FULL-TIME/PARTTIME]
PROBE: I am going to show you the same question, but with a different format. What do you think
about this alternative format? Do you have a preference? [INTERVIEWER SHARES SCREEN AND
DISPLAY ALTERNATIVE FORMAT]
P11. (GROUP 1) Now let’s review Question 20, which ask about refresher training for academy instructors.
PROBE: What does “general instruction” mean to you?
PROBE: Did you have any challenges understanding or responding to this question? How so?
P12. (GROUP 1) [ASK ONLY IF Q19 = 1 OR 3] Question 21 is a follow up of the previous question. Please
review this question.
PROBE: What does “internally-provided training sessions” mean to you?
P14. (GROUP 1) The CLETA questionnaire asks two questions related to your academy’s budget. Please have a
look at Question 23. Please read through the question to yourself and let me know if anything is unclear or
confusing.
PROBE: Can you tell me, in your own words, what you consider as total operating budget?
PROBE: [IF R COMPLETED THE SURVEY]: How did you define the total amount? [IF ESTIMATE –
HOW DID THEY COME UP WITH THE ESTIMATE?]
PROBE: How difficult or easy do you think it would be to get an accurate value for this question?
Would that value be an estimate?
P15. (GROUP 1) Now let’s look at Question 24. Again, please read through the question to yourself and let me
know if anything is unclear or confusing.
PROBE: Can you tell me, in your own words, what you consider as budget for equipment?
PROBE: [IF R COMPLETED THE SURVEY]: How did you define the total amount? [IF ESTIMATE –
HOW DID THEY COME UP WITH THE ESTIMATE?]
PROBE: How difficult or easy do you think it would be to get an accurate value for this question?
Would that value be an estimate?
46
P16. (BOTH) The next section of the questionnaire contains questions about basic academy trainees. Let’s start by
discussing Question 25.
PROBE: Did you have any challenges completing this question? What challenges did you have?
a. PROBE: [IF NOT COVERED] How difficult was it to obtain the values for this question?
PROBE: The question asks to check a box if your values are estimates and not exact. How likely would
you be to provide an estimate as opposed to providing exact values?
b. PROBE: [IF ESTIMATE BOX WAS SELECTED] Why did you provide an estimate? / Do you
have access to the exact values? / How difficult would it be to obtain exact values?
PROBE: Does your academy collect information on recruits who do not align with the categories
“Male” and “Female” (e.g. non-binary, unknown)? How would you report such recruits? Would your
academy benefit from having additional response options of “Non-binary” or “Gender not known”?
P17. (BOTH) Now I’d like to hear your feedback about Question 26. It asks about the number of recruits that
started and completed basic training by race and ethnicity.
PROBE: Did you have any challenges completing this question? What challenges did you have?
PROBE: [IF ESTIMATE NOT CHECKED] Did you see the instruction under the question to check the
box if your response was an estimate? Were any of your responses’ estimates?
PROBE: [IF THERE IS A VALUE IN “NOT KNOWN” ROW] Could you tell me more about those
recruits included in the “not know” category? [PROBE TO UNDERSTAND IF THEY DON’T
COLLECT DATA ON RACE/ETHNICITY]
P18. (BOTH) Imagine that the survey asked you the following question [INTERVIEWER SHARES SCREEN
WITH QUESTION TEMPLATE]: “How many recruits, upon entering BASIC training were a) military veterans, b)
Possessed an associate degree, c) Possessed a bachelor’s degree?”
PROBE: Would you be able to provide the total number of recruits that were military veterans?
a. PROBE: How about those have earned an associate’s degree?
b. PROBE: How about those have earned a bachelor’s degree?
PROBE: How burdensome would this question be to answer?
P19. (BOTH) Let’s look at Question 27. Please read each reason listed for why recruits did not complete BASIC
training and let me know if anything is unclear or hard to understand.
PROBE: How difficult was it for you to report these data for each of the categories, by sex
classification?
PROBE: Are these accurate values or are they estimates?
47
PROBE: Are there any other frequent reasons not listed in the table?
P20. (BOTH) The final portion of the questionnaire contains questions related to your academies core basic training
curriculum. Let’s first review Question 28.
PROBE: What does “…training philosophy of academies…” mean to you as used in this question?
P21.A (BOTH) Question 29 asks you about the topics covered in BASIC training. If the topic is covered, then you
are asked to provide the number of hours of instructions required. It also requests you to select if the number of
hours is an estimate or if it is an exact number. Please read all topics under “A. Operations” and “B.
Weapons/Defensive Tactics” to yourself.
PROBE: What do you think about these subject areas? Are any subject areas unclear?
PROBE: How easy or difficult is it for your academy to report the number of hours on each topic
covered in BASIC training?
PROBE: For those topics covered, can you provide an exact number of hours or an estimate?
P21.B (BOTH) Please keep reading the remaining of the topics to yourself. Please let me know if you think there is
any topic that is unclear to you, or you find any difficulty on responding.
PROBE: Do you think there is overlap between these subjects? IF YES: Which ones?
a.
PROBE: How would you report the hours of those overlapping subjects? [PROBE TO SEE
IF R WOULD INCLUDE HOURS ON ALL SUBJECTS, ADD THEM TO ONE AND
EXCLUDE THEM FROM THE REST, OR SOME OTHER WAY]
DEBRIEF QUESTIONS (BOTH)
P26. Now I’d like you to think about the entire survey. Did you have difficulty with any aspects of the survey that
we have not already discussed?
P27. How do you feel about the length of the survey and the time needed to answer the questions?
P28. Before we conclude, do you have any other feedback or suggestions to improve the questionnaire?
48
Thank you very much for taking the time to provide your thoughts and feedback, it has been very helpful. We will
combine your comments with feedback from other participants into an overall report. That report will help BJS
evaluate the survey and determine whether to make changes to the questionnaire.
Thanks again!
END TIME: ___ ___ : ___ ___ AM / PM
Appendix B – 2022 CLETA: Cognitive Interview Protocol – GROUP 2
P#________
DATE: ___________
START TIME: ______
INTERVIEWER: ______
Section 1: Introduction
Hello, I’m [NAME] with RTI, speaking with you on behalf of the Bureau of Justice Statistics.
Thanks for agreeing to help us develop the questionnaire for the 2022 Census of Law Enforcement Training
Academies, or CLETA. The call will take about an hour. I’d like to start with a short summary of the goals for
today’s call and explain a bit about how I’ll conduct the interview.
‐
‐
IF NO LONGER A GOOD TIME, OFFER TO RESCHEDULE
IF STILL GOOD TIME, CONTINUE
As you may know, the Bureau of Justice Statistics and RTI are preparing to conduct the CLETA in early 2023. As
we get ready for the study, we are asking academy staff like you to review the draft questionnaire. During this call,
I’ll ask for your reactions and feedback to these draft questions– including things like how the questions are worded,
your understanding of concepts, ways to clarify instructions, and the information your academy tracks that is related
to the survey questions.
Your participation in this interview is completely voluntary. You may stop at any time or skip questions that you do
not want to answer. The data we collect will be de-identified, meaning that your personal information, such as your
name, will never be stored together with your responses. With your permission, I would like to record this call for
the sole purpose of capturing comments that I may miss when taking notes. The recording would only be accessible
to me and would be deleted no later than 2 weeks from this interview when the responses from this call are finalized.
Do I have your permission to record this call?
Please keep in mind that there are no right or wrong answers to my questions. One of our main goals is to draft
questions that make sense, so if anything about the questions is confusing or unclear, you can help by pointing this
out to me. Also, if you’re not sure how you would respond to any of the questions, please tell me that, too. Finally,
please let me know if you believe any questions or topics were omitted, or any answer choices or response
categories that were missing or insufficient.
I am interested in hearing all of your feedback on the survey, but because there are a lot of topics to discuss and we
only have an hour, sometimes I might ask that we move on to the next question before you’ve had a chance to share
everything on your mind. At the end of the interview, you can share any important feedback that you didn’t have a
chance to share earlier.
Do you have any questions before we begin?
Before we start discussing the questions, did you complete the questionnaire?
‐ IF YES: Do you happen to recall approximately how much time you spent completing the questionnaire?
Please include the time you and any others at your academy spent gathering information needed to answer
the questions.
‐
__________ HOURS __________ MINUTES Go to Section 2.
IF NO: [INTERVIEWER: RECORD NOTES ABOUT REASONS OF NOT COMPLETING THE
INSTRUMENT. RECORD ANYTHING THE R MENTIONED EARLIER OR AFTER SAYING THEY
DID NOT COMPELTED IT]
49
Section 2: Item reviews
I’m planning to discuss only some of the questions on the questionnaire, but if you have comments or concerns
about any of the questions I skip, please feel free to share them with me at any time.
P1. (BOTH) The first question I’d like to discuss is Question 2. It asks about BASIC training. Please read the
question to yourself.
PROBE: Do you have any difficulty understanding the definition of BASIC training?
PROBE: Is there anything you would add/remove to the definition of BASIC training?
P3. (BOTH) Next I’d like to review Question 4 which asks about degrees and college credits.
PROBE: Is it clear what “partner institution” means here?
PROBE: Do you interpret this question to mean degrees and/or college credits are awarded based on
completing BASIC training?
P4. (BOTH) Now we will move on to Question 8. It asks about the length of your academy’s BASIC training
program.
PROBE: What do you think of this question?
PROBE: Can you tell me how you went about answering this question?
PROBE: What does, “core BASIC training program” mean to you?
PROBE: How do you interpret, “TOTAL number of hours required to complete basic training”?
How about, “Number of hours mandated by state POST…”?
PROBE: Did you read the instruction under the question? Did your response include any of
these?
PROBE: Does your academy measure the length of BASIC training in hours?
[IF NOT] What process would you use to convert your answer if you needed to report the length of your
program in hours?
P5. (BOTH) [ONLY ASK IF Q10 = YES] Please now read aloud Question 11. It asks about the length of field
training for academies that oversee these programs.
PROBE: In your own words, what does, “…core field training program” mean to you?
50
PROBE: Does your academy measure length of its field training program in hours?
[IF NOT] What process would you use to convert your answer if you needed to report the length of your
program in hours?
P6. (GROUP 2) Now let’s review Question 14. This question asks about how community members are involved
with different aspects of your academy. Please read it to yourself.
PROBE: Do you feel this question is clear or unclear? Why?
PROBE: What does “directly involved community members” means to you as used in this question?
PROBE: In your own words, what do “formal” and “informal” methods mean to you?
[IF NO EXAMPLES ARE PROVIDED BY R IN THEIR DEFINITION: Could you give us an example
of a [formal] and/or [informal] method?]
PROBE: FOR ANY YES RESPONSE: Can you tell me more about why you answered “Yes” to X?
P7. (BOTH) Now we are going to review questions related to academy personnel. We’ll start with Question 15.
Please read the question and each category to yourself. Let me know if there are any descriptions that are
unclear or difficult to understand.
PROBE: Does your academy record this information separately for full-time and part-time instructors?
PROBE: How difficult is it to obtain this information for each of the categories listed in the table?
PROBE: Do any of your trainers or instructors fit within more than one category?
[IF YES: PROBE TO FIND OUT IF THEY REPORTED THEM IN MULTIPLE ROWS. IF ONLY
REPORTED IN ONE ROW, HOW DID THEY DECIDE WHICH ROW?]
PROBE: When determining who to count as full-time and who to count as part-time, did you think about
their status as a trainer or instructor, or did you think about their status with respect to all job
responsibilities, not just training?
P8. (BOTH) Now let’s go to Question 16. This question asks about education requirements for full and part time
instructors.
PROBE: What challenges, if any, did you have in answering this question?
a. PROBE: Was it clear that you could only provide one response for each column?
P9. (BOTH). Question 17 asks about requirements for instructors as sworn officers.
PROBE: Did you have any challenges answering the field for, “Sworn officer status…”?
51
PROBE: [IF SWORN OFFICER OPTION IS SELECTED, DETERMINE HOW NUMBER OF YEARS
OF EXPERIENCE WAS CALCULATED]
P10. (BOTH) Please now turn your attention to Question 18. It asks about certifications required for your
academy’s instructors.
PROBE: Can you explain to me with your own words what this question is asking about?
PROBE: Is the format of the question clear? [PROBE TO SEE IF ITS CLEAR OR UNCLEAR THAT
AN ANSWER IS REQUIRED FOR EACH ROW AND THE DISTINCTION OF FULL-TIME/PARTTIME]
PROBE: I am going to show you the same question, but with a different format. What do you think
about this alternative format? Do you have a preference? [INTERVIEWER SHARES SCREEN AND
DISPLAY ALTERNATIVE FORMAT]
P13. (GROUP 2) Next, I would like to discuss items related to academy resources. Please go to Question 22. This
table contains five general categories of resources. Let’s look at section A. Please let me know if you find
anything unclear or confusing [ASK FOR EACH SECTION AFTERWARDS. ASK PROBES BELOW AFTER
THE CORRESPONDING SECTION IS REVIEWED BY THE RESPONDENT]
PROBE: [INDICATE OVERALL SECTION COMMENTS HERE]
PROBE: [AFTER SECTION B] What does “Use of force training simulators” this mean to you?
o Do you think this is similar or not to “firearms training simulators”?
PROBE: [AFTER SECTION C]: What does “Mental Health Services” mean to you?
PROBE: [AFTER R FINISHES READING ALL SECTIONS] Are there any resources that are missing
from this list?
PROBE: How did you know what to select for each resource? For example, were you thinking about
resources within your academy or also considering external resources?
P16. (BOTH) The next section of the questionnaire contains questions about basic academy trainees. Let’s start by
discussing Question 25.
PROBE: Did you have any challenges completing this question? What challenges did you have?
a. PROBE: [IF NOT COVERED] How difficult was it to obtain the values for this question?
PROBE: The question asks to check a box if your values are estimates and not exact. How likely would
you be to provide an estimate as opposed to providing exact values?
b. PROBE: [IF ESTIMATE BOX WAS SELECTED] Why did you provide an estimate? / Do you
have access to the exact values? / How difficult would it be to obtain exact values?
52
PROBE: Does your academy collect information on recruits who do not align with the categories
“Male” and “Female” (e.g. non-binary, unknown)? How would you report such recruits? Would your
academy benefit from having additional response options of “Non-binary” or “Gender not known”?
P17. (BOTH) Now I’d like to hear your feedback about Question 26. It asks about the number of recruits that
started and completed basic training by race and ethnicity.
PROBE: Did you have any challenges completing this question? What challenges did you have?
PROBE: [IF ESTIMATE NOT CHECKED] Did you see the instruction under the question to check the
box if your response was an estimate? Were any of your responses’ estimates?
PROBE: [IF THERE IS A VALUE IN “NOT KNOWN” ROW] Could you tell me more about those
recruits included in the “not know” category? [PROBE TO UNDERSTAND IF THEY DON’T
COLLECT DATA ON RACE/ETHNICITY]
P18. (BOTH) Imagine that the survey asked you the following question [INTERVIEWER SHARES SCREEN
WITH QUESTION TEMPLATE]: “How many recruits, upon entering BASIC training were a) military veterans, b)
Possessed an associate degree, c) Possessed a bachelor’s degree?”
PROBE: Would you be able to provide the total number of recruits that were military veterans?
a. PROBE: How about those have earned an associate’s degree?
b. PROBE: How about those have earned a bachelor’s degree?
PROBE: How burdensome would this question be to answer?
P19. (BOTH) Let’s look at Question 27. Please read each reason listed for why recruits did not complete BASIC
training and let me know if anything is unclear or hard to understand.
PROBE: How difficult was it for you to report these data for each of the categories, by sex
classification?
PROBE: Are these accurate values or are they estimates?
PROBE: Are there any other frequent reasons not listed in the table?
P20. (BOTH) The final portion of the questionnaire contains questions related to your academies core basic training
curriculum. Let’s first review Question 28.
PROBE: What does “…training philosophy of academies…” mean to you as used in this question?
P21.A (BOTH) Question 29 asks you about the topics covered in BASIC training. If the topic is covered, then you
are asked to provide the number of hours of instructions required. It also requests you to select if the number of
53
hours is an estimate or if it is an exact number. Please read all topics under “A. Operations” and “B.
Weapons/Defensive Tactics” to yourself.
PROBE: What do you think about these subject areas? Are any subject areas unclear?
PROBE: How easy or difficult is it for your academy to report the number of hours on each topic
covered in BASIC training?
PROBE: For those topics covered, can you provide an exact number of hours or an estimate?
P21.B (BOTH) Please keep reading the remaining of the topics to yourself. Please let me know if you think there is
any topic that is unclear to you, or you find any difficulty on responding.
PROBE: Do you think there is overlap between these subjects? IF YES: Which ones?
a.
PROBE: How would you report the hours of those overlapping subjects? [PROBE TO SEE
IF R WOULD INCLUDE HOURS ON ALL SUBJECTS, ADD THEM TO ONE AND
EXCLUDE THEM FROM THE REST, OR SOME OTHER WAY]
P22. (GROUP 2) Next, let’s go to Question 30
PROBE: What does FORMAL ASSESSMENTS mean to you as used in this question?
PROBE: Do these names of formal assessment match with the way they are called in your academy? IF
NO: Which one?
How is it called in your academy?
PROBE: [IF YES TO ANY] For those required assessments, how did you define the number of
assessments required?
PROBE: Is the “Number of assessments required” a value that you can access at your academy? IF NOT
READILY AVAILABLE: How difficult would it be to report on these?
P23. (GROUP 2) Question 31 asks about the use of role play scenarios for different activities during BASIC
training.
PROBE: What does “role play scenarios and activities” mean to you as used in this question?
a. PROBE: Can you tell me what types of training do you think this term include? For example, do
you think it includes less formal classroom scenario exercises?
b.
PROBE: In the last CLETA, the term “reality-based (mock) scenarios” was used instead. Do you
think there is a difference between the terms “role play scenarios and activities” and “realitybased (mock) scenarios?
PROBE: Are the categories provided in the list clear or unclear to you? Did you have difficulty
understanding any of them?
54
P24. (GROUP 2) Now let’s review Question 32, which asks about various types of weapons used in training.
PROBE: Do you have any confusion on the use of the term “Revolver” in 32a? Do you have a better
suggestion for a term to use?
P25. (GROUP 2) Moving to Question 33. This question asks about firearms or related training.
PROBE: In the last CLETA, 33a was referred to as: Firearms Training Simulator. Do you feel “Use of
force training simulators” is better to include here? Why?
DEBRIEF QUESTIONS (BOTH)
P26. Now I’d like you to think about the entire survey. Did you have difficulty with any aspects of the survey that
we have not already discussed?
P27. How do you feel about the length of the survey and the time needed to answer the questions?
P28. Before we conclude, do you have any other feedback or suggestions to improve the questionnaire?
Thank you very much for taking the time to provide your thoughts and feedback, it has been very helpful. We will
combine your comments with feedback from other participants into an overall report. That report will help BJS
evaluate the survey and determine whether to make changes to the questionnaire.
Thanks again!
END TIME: ___ ___ : ___ ___ AM / PM
55
</pre><Table class="table"><tr><Td>File Type</td><td>application/pdf</td></tr><tr><Td>Author</td><td>buehlere</td></tr><tr><Td>File Modified</td><td>2023-01-20</td></tr><tr><Td>File Created</td><td>2023-01-20</td></tr></table></div></div></div><hr>
© 2024 OMB.report | <a href="/privacy_policy.php" rel="nofollow">Privacy Policy</a>
<hr >
</div>
</body>
<script defer async src="/js/instant.page.3.0.0.js" type="module" data-cfasync="false"></script>
</html>