B. Collection of Information Employing Statistical Methods
Universe and Respondent Selection
The potential victim officer universe of Forms 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed include all local, state, tribal, and federal law enforcement officers (LEOs) who:
Wore/carried a badge (ordinarily);
Carried a firearm (ordinarily);
Were duly sworn and had full arrest powers;
Were members of a law enforcement agency (LEA);
Were acting in an official capacity, whether on or off duty, at the time of the incident; and
If killed, directly related to the injuries received during the incident.
In 2017, 176 victim officers were feloniously/accidentally killed, or assaulted and received injury with a firearm or knife/other cutting instrument. Of those, 94 agencies completed Form 1-701a and 82 agencies completed Form 1-701.
Procedures for Collecting Information
The Federal Bureau of Investigation (FBI) Uniform Crime Reporting (UCR) Program’s Law Enforcement Officers Killed and Assaulted (LEOKA) data collection collects all LEO deaths/assaults from LEAs in the United States; sampling methodologies are not used. The UCR LEOKA Program does not apply estimation procedures in the LEOKA data collection, but presents the reported number of officers killed and assaulted in the LEOKA publication.
The LEOKA Program receives an electronic communication (EC) containing details of the officer’s death from the appropriate FBI field office (FO) via Sentinel, as well as an Executive Situation Report from the corresponding FO by email. Sentinel is an internal communication system, which is the next generation information and case management system and has moved the FBI from a primarily paper-based reporting system to an electronic record-keeping system. Sentinel significantly improved how the FBI manages its investigative, intelligence, and administrative information workflow by providing users a new way to enter, review, approve, and research case information.
Upon receipt of a death notification, LEOKA personnel author an EC, giving a synopsis of the request, the LEOKA Program’s mission, and directions for providing pertinent information about the victim officer. LEOKA staff then forwards the EC, via Sentinel, to the assigned FBI Special Agent (SA) at the FO. The EC contains a random computer-generated reference number and reference key with detailed directions for the officer’s agency to access the specific incident in the LEOKA database via the Law Enforcement Enterprise Portal (LEEP). The SA forwards the information within the EC to the victim officer’s agency, providing the 30-day deadline to finalize and submit the data submission to the LEOKA Program. Following the data entry process, the database contains a FINALIZE AND SUBMIT button on the screen. Upon submission, LEOKA staff and the submitting agency receive a confirmation email verifying a successful submission. A retention document is included as an attachment, providing a list of the submitted data for the agency’s records. Upon receipt, LEOKA staff reviews the data for accuracy and completeness.
LEOKA staff partners with other law enforcement organizations to ensure the LEOKA Program receives all death notifications. Throughout the year, LEOKA staff compares a list of victim officers with the list of other nonfederal organizations (the Officers Down Memorial Page, the National Law Enforcement Officers’ Memorial Fund, the Public Safety Officers’ Benefits Program, and Concerns of Police Survivors) to account for all death notifications.
The FBI UCR Program receives monthly submissions from participating agencies on all officer assaults. LEOKA staff are notified of all officer assaults sustaining injury from a firearm or knife/other cutting instrument. From there, LEOKA staff forwards an email to the victim officer’s state UCR Program. These emails provide a synopsis of the request, the LEOKA Program’s mission, and directions for providing pertinent information about each assault with injury incident. Like the officer death notifications, these emails include random computer-generated reference numbers and reference keys for each assault with injury incident. Also included are detailed directions for the victim officer’s agency to access the specific incident in the LEOKA database via LEEP. The State Program contact forwards the information within the email to the officer’s agency and informs them of the 30-day deadline to finalize and submit the data submissions to the LEOKA Program. At the end of the data entry process, the database contains a FINALIZE AND SUBMIT button on the screen. Upon submission, LEOKA staff and the submitting agency receive confirmation emails verifying successful submissions. An attachment contains a retention document, which provides a list of the submitted data for the agency’s records. Upon receipt, LEOKA staff reviews the data for accuracy and completeness.
Methods to Maximize Response
The FBI maximizes response rates through liaison with FOs and LEAs. Communications encouraging data submission occur frequently between LEOKA staff and the FOs, state UCR Programs, and local agencies. LEOKA staff possess a strong understanding of contextual challenges agencies face in reporting valid and reliable data and regularly work to overcome nonresponse issues when such challenges occur. The mission of the LEOKA Program is to acquire LEOKA data, establish guidelines for the collection, and publish LEOKA data. Although the FBI makes every effort through its editing procedures, training practices, and correspondence to ensure the validity of the data it receives, the accuracy of the statistics depends primarily on the adherence of each contributor to the established standards of reporting. LEOKA staff also follow up on incident reports not returned by the assigned deadline.
The new LEOKA database/application will allow for the following:
Integration of LEOKA data sources eliminating multiple, purpose built data repositories within the LEOKA database;
Reduction of administrative workload and burden for internal and external stakeholders to submit data;
Reduction of incomplete and inconsistent data submissions due to built-in features and display rules within the LEOKA database;
Elimination of current work functions by automating procedures currently being performed manually;
Expansion of the ability to efficiently collect, store, edit, review, query, report, and publish data;
Improvement of the accessibility and timeliness of releasable (nonpersonally identifiable information) data to the public;
Increase the agility of the LEOKA database to modify the data collection;
Automation of correspondence and communications with stakeholders from the LEOKA Program; and
Ensure the collection and timely distribution of more accurate and complete LEOKA data.
Response Rates
The UCR Program recognizes the importance of response rates to meet the Office of Management and Budget’s (OMB) publication guideline. In reviewing the LEOKA response rates from 2015-2017, the assault rates ranged from 63.5 percent to 69.1 percent. The felonious deaths ranged from 92.1 percent to 97.8 percent, while the accidental deaths ranged from 82.7 percent to 98 percent.
Technical Response to Address Agency Nonresponse
The recommendation of the Criminal Justice Information Services (CJIS) Advisory Policy Board (APB) to create an FBI-sponsored and FBI-maintained tool is in direct response to issues which have continually provided impediments to the adoption of modifications to the FBI UCR Program. Traditionally, the FBI UCR Program has provided to both agencies and state UCR Programs a set of technical specifications for data submissions of the UCR data collections. However, this method is under the presumption agencies and state UCR Programs assume the responsibility to build and maintain a data system for collecting the data. In the case of the LEOKA Program’s Forms 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed, the FBI is sponsoring and maintaining an electronic data collection tool which is accessible through LEEP. This portal capability enables agencies to provide their data directly to the FBI.
The LEEP data collection tool assumes agencies have consistent connectivity to the Internet and have maintained an active account on LEEP. However, it is unlikely all agencies will have proactively enrolled in LEEP. LEOKA staff has formulated a plan with the LEEP Program Office to expedite enrollment for agencies not currently enrolled in LEEP.
Additionally, LEOKA staff recommended a future technical enhancement to address agency nonresponse issues. The idea is to develop an ‘automatic email and tracking mechanism’ within the new UCR-Technical Refresh (TR) system and the LEOKA database. The recommendation allows the UCR-TR system to obtain the reference numbers and keys for the assault with injury incidents from the LEOKA database. The UCR-TR system will then automatically generate an email to the state UCR Programs/individual agencies, providing the reference number and key, to obtain the assault with injury incident data to be reported to the FBI UCR Program. A secondary feature will allow the UCR-TR and the LEOKA database to track these requests and responses.
National Incident-Based Reporting System (NIBRS) Transition
The UCR Program is actively working to increase NIBRS participation by partnering with the Bureau of Justice Statistics on the National Crime Statistics Exchange, working with advocacy groups to emphasize the importance of NIBRS data for the public and the law enforcement community, and transitioning the FBI UCR Program to a NIBRS-only data collection by 2021. This transition plan is continuously communicated during UCR Program teleconferences and external advocacy and APB meetings.
The NIBRS law enforcement reporting structure assists LEOKA staff in determining a LEA’s eligibility for reporting LEOKA assault data. The NIBRS structure identifies the extent of injury to victim officers, which assists the FBI in determining the incidents requiring the detailed “assault with injury” data. LEOKA staff request the detailed report when victim officers receive more than a minor injury involving a firearm or knife/other cutting instrument.
Additionally, the NIBRS reporting structure identifies the Incident Date and Incident Number. When LEOKA staff request the detailed “assault with injury” data, the specified incident data (Incident Date and Incident Number) are included in the request, making it more convenient for the agencies to locate the incident within their records.
Testing of Procedures
In March 2010, the OMB approved an expanded collection of the 1-701 and 1-701a forms. The LEOKA Program began disseminating the revised forms on January 1, 2011. However, a database was not available to accommodate the expanded forms. The percentage of data collected since 2011 not captured in the LEOKA database includes 68 percent of each felonious incident and 91 percent of each accidental death. Since January 1, 2011, the LEOKA Program has received an average of 202 data collection forms per year.
To meet the needs of the LEOKA Program, the FBI developed a new LEOKA database/application within LEEP. The new database will allow users to electronically complete Forms 1-701 and 1-701a. This database/application provides a graphical user interface tool to walk the preparer through a series of questions relevant to the LEOKA incident. Upon completion, the incident data will be electronically submitted to the LEOKA Program.
In 2017, during the first phase of developing the database/application, the LEOKA Program worked with FBI Information Technology (IT) staff to first develop a database, providing a solution to allow LEOKA staff to enter the new data elements approved in 2010. The LEOKA Program implemented this database in late November 2017.
During the second phase of this initiative, the LEOKA Program presented recommendations to the CJIS APB to further expand the 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed. Since approval of these forms in 2010, the LEOKA Program has received numerous requests for additional data elements from various agencies throughout the law enforcement community. Based on these data requests, the LEOKA Program worked to improve both data collection forms through research and analysis, to add data elements, reword questions, and eliminate unnecessary questions. The FBI UCR Program created a focus group, which included representatives from the FBI, Major City Chiefs Association (MCCA), International Association of Chiefs of Police (IACP), and the National Sheriffs’ Association (NSA) to review and provide feedback on the new categories. The LEOKA Program presented the forms to the focus group and asked its members to select actual incidents of officers killed and assaulted within their agency, and to complete the forms using those incident reports. This process allowed the LEOKA Program to gain input from the focus group and to test the usability of the forms.
Additionally, the National Use-of-Force (UoF) Data Collection Task Force reviewed the expanded elements to determine the significance of these new components. The FBI established the UoF Task Force, represented by members of the IACP, NSA, MCCA, Major County Sheriffs of America, Police Executive Research Forum, Association of State Criminal Investigative Agencies, National Organization of Black Law Enforcement Executives, and the Association of State UCR Programs. The UoF Task Force determined the scope of the UoF data collection, specific responsibilities related to the reporting and handling of UoF data, and a final set of data elements for consideration. For review of the LEOKA data collection, UoF Task Force members received the LEOKA forms via email and were provided a month to assess the documents. The LEOKA Program then spoke to each member via teleconference to discuss any comments, questions, or changes. Following the teleconferences, the LEOKA Program compiled all the suggestions and made the appropriate changes on the forms.
The LEOKA Focus Group and UoF Task Force identified comprehension problems associated with question terminology and criteria requirements. In some cases, LEOKA staff rephrased a question and incorporated additional collection options into others, e.g., mass casualty and post-traumatic stress disorder incidents.
The LEOKA Program began working with the FBI IT staff to complete phase three, development of the new LEOKA application in January 2018, to finalize the application’s requirements. The two groups continued the development and internal testing phases of the new LEOKA database from February through June 2018. During the same timeframe, LEOKA staff introduced the new collection recommendations during the 2018 Spring CJIS Advisory Policy process (March–June).
Cognitive Testing
Purpose of the Research
The primary goal of the LEOKA Program is to reduce the number of LEO line-of-duty deaths and assaults by providing data, research, and instructional services relative to law enforcement safety. The LEOKA Program provides data regarding officer deaths and assaults via its annual publication and responds to special data requests. The data is used to:
Identify circumstances and trends in which officers are killed and assaulted in the line of duty;
Aid LEAs in developing policies and training programs to improve officer safety;
Include in the curriculum of the LEOKA Officer Safety Awareness Training; and
Provide information to LEOKA Liaison Specialists for the composition of officer safety articles monthly.
Currently, when a LEOKA incident occurs, the FBI sends Form 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted, or Form 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed, to deceased officer’s law enforcement agency. Certain portions of the forms may not apply to a particular incident and may be cumbersome and time consuming for the preparer to complete. Since the implementation of Forms 1-701 and 1-701a in 2011, the LEOKA Team has identified many inconsistencies on completed forms which were not identified during the original cognitive testing of the forms. The LEOKA staff believe the inconsistencies exist due to the preparer(s) misperception when completing the form. To streamline this process for all involved and alleviate these issues, the FBI is automating this collection.
Following APB recommendations in June 2018, the expanded forms underwent cognitive testing. The new application of the LEOKA database was pretested with 20 law enforcement participants. Sixteen of the participants were LEOs, while the remaining four were civilian law enforcement employees. The twenty participants were also comprised of four representatives from a city police department, four participants from a county sheriff’s office, four representatives from a university campus law enforcement agency, four representatives from a state police agency, and four representatives from a federal law enforcement agency.
Methodological Plan
The cognitive interviews assessed two aspects of the revised collection. The first component focused on whether the new web-based collection introduced problems with comprehension or general usability through the transition from one mode of collection to another. The second component specifically targeted possible comprehension problems associated with terminology, definitions, and criteria requirements.
Twenty law enforcement personnel participated in the cognitive interviews. The method involved intensive, one-on-one interviews in which the participants were asked to "think aloud" as he or she completed a submission based on specific incident details. Several different techniques were involved, such as asking respondents to paraphrase questions or asking probing questions to determine how respondents came up with their answers. The objective was to identify problems of ambiguity or misunderstanding, identify potential appearance improvements, flow, and instructions, or highlight other difficulties respondents had in answering questions. Participants were provided three incident examples which included an accidental death, a felonious killing, and an assault with injury which involved a firearm, knife, or other cutting instrument. These incident examples were used to evaluate the impact of converting from a paper-based instrument to a web-based instrument and to identify any other issues as described above.
A team of two individuals from the LEOKA Team conducted the cognitive interviews with each participant individually. This allowed both cognitive interviewers to take detailed notes of responses provided by the participant and record behavioral cues. One cognitive interviewer directly viewed how the participant filled out the data collection on a computer and was primarily responsible for recording any behavioral cues which indicated difficulty with the mode of collection. Examples of this behavior included hesitation, hovering over specific items, or misunderstanding terminology. The second cognitive interviewer was primarily responsible for recording the verbal information provided by the participant as part of the “think aloud” aspect of the cognitive interview.
The areas addressed in the cognitive interviews and usability testing included the following:
Identified whether the change from a paper-based collection to a web-based collection reduced or introduced any unforeseen issues with comprehension for the respondents;
Assessed the comprehension of terms and definitions included in the collection;
Improvement in response rate; and
Elimination of reporting errors, made by the preparer, when completing the paper forms.
Evaluation/Analysis
The interviews found the following general observations:
The change in mode from a paper-based collection to a web-based collection reduced the response time by 52 hours annually.
The respondents/observers did not detect or report comprehension issues.
The respondents provided favorable comments about the flow of the database, ease of use, and the capability of the database to eliminate unnecessary questions.
Participants were unsatisfied with:
The response time (page refresh delay after data input) of the LEOKA database/application;
The order of text fields; and
The inconsistent requirements of nonalphabetized dropdown lists or lack of dropdown lists.
Participants requested the following:
Concise instructions on the home screen for improved navigation throughout the forms;
Relocation of response buttons for improved functionality;
Automation features such as identifying which fields were in error;
Improved functionality of the tabbing throughout the form with ease;
To combine some fields (feet/inches);
Spanish translation feature; and
Adjust the background colors for ease of reading.
LEOKA staff shared the cognitive test findings identified above with IT staff. IT staff reviewed the problem areas and successfully integrated these changes and participant requests into the LEOKA database/application.
Contacts for Data Collection
Amy C. Blasher
Crime Statistics Management Unit Chief
<acblasher@fbi.gov>
(304) 625-4840
Cynthia Barnett-Ryan
Statistician
<cbarnett-ryan@fbi.gov>
(304) 625-3576
Melissa J. Blake
Management/Program Analyst
<mjblake@fbi.gov>
(304) 625-4738
Kristi L. Donahue
Management/Program Analyst
<kldonahue@fbi.gov>
(304) 625-2972
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Donahue, Kristi L. (CJIS) (FBI) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |