Memo to OMB

MEMORANDUM

MEMORANDUM TO: Shelly Wilkie Martinez

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Lynn Murray

Clearance Officer

Justice Management Division

William J. Sabol, Ph.D.

Acting Director

Bureau of Justice Statistics

FROM: Howard Snyder

Deputy Director

Bureau of Justice Statistics

DATE: November 25, 2013

SUBJECT: BJS request for OMB Clearance to conduct an agency-level survey with non-reporting agencies in NIBRS states for the National Crime Statistics Exchange under the generic clearance agreement OMB Number 1121-0339.

Introduction

The Bureau of Justice Statistics (BJS) is spearheading the National Crime Statistics Exchange (NCS-X), an initiative designed to support the development of national incident-based data on crimes known to law enforcement agencies. Such data are currently submitted to the FBI’s National Incident-Based Reporting System (NIBRS) by more than 5,000 law enforcement agencies across 32 states, providing data on a wide range of crimes. However, these data are not nationally representative and thus are not able to generate national estimates. BJS has calculated that if a scientifically selected sample of 400 law enforcement agencies would agree to become new NIBRS participants, the resulting NIBRS data could generate statistically sound and detailed national estimates of crime known to law enforcement.

NCS-X is a collaborative undertaking, supported by the Federal Bureau of Investigation (FBI) and other Department of Justice agencies (see Attachment A for the joint statement of support), designed to increase participation in the FBI’s NIBRS program by providing funding and technical assistance to the selected local and state law enforcement agencies, enabling them to report incident-based data to their state Uniform Crime Reporting (UCR) program and subsequently on to the FBI. Collectively, this initiative will increase our nation’s ability to monitor, respond to, and prevent crime by producing timely, detailed, and accurate national measures of crime incidents. To facilitate these efforts, BJS has entered into a cooperative agreement with RTI International to develop implementation plans for NCS-X, assess the needs of local law enforcement agencies selected to participate in NCS-X, and to provide technical assistance to these agencies and their state UCR reporting programs (see Attachment B for an overview of the NCS-X ). The NCS-X also has the support of the International Association of Chiefs of Police’s (IACP) Criminal Justice Information Sharing (CJIS) Committee, which serves as the primary liaison with the FBI on all issues involving information sharing efforts related to criminal justice (see Attachment C for the IACP resolution of support).

The initial phase of the NCS-X will focus on obtaining the necessary background information to 1) assess the viability of and costs for the sample of 400 local law enforcement to participate in NIBRS, 2) identify potential barriers for agency participation, and 3) develop a set of possible incentives to encourage candidate agencies to participate in NIBRS. This background information will provide the foundation for BJS to implement the necessary funding and technical assistance mechanisms to recruit the sample of 400 law enforcement agencies and enable the agencies to begin participating in NIBRS.

The first step in gathering this background information was to conduct semi-structured telephone interviews with the UCR/NIBRS coordinators in each of the 50 states and Washington, DC. These interviews were conducted between August 2013 and October 2013 to better understand how state UCR/NIBRS programs are organized and to gather key information about how incident-based data are collected from local law enforcement agencies in each state, processed by the state, and transmitted to the FBI. The existing processes and capabilities of each state to collect and handle these incident-based data will be the foundation upon which the planning and implementation activities of NCS-X will be based. For example, some states have a state-wide, NIBRS compliant records management system available to any law enforcement agency in the state which makes it easier for an individual agency to become a NIBRS reporting agency. These interviews also yielded the type(s) of technical assistance each state might need to facilitate the NCS-X efforts. Types of technical assistance identified by states during the interviews included funding for training, additional staff, and technical resources such as analysis tools.

The next step is to gather more detailed information specific to each of the candidate law enforcement agencies. To that end, BJS is requesting approval to implement an agency-level survey of the 400 sampled agencies, all of which do not currently report data to NIBRS. The survey will provide BJS with details regarding each agency’s existing reporting capabilities, as well as how BJS can best support the agencies in a transition to reporting incident-based data in the NIBRS format.

Request for developmental work

BJS plans to conduct developmental work for NCS-X under the generic clearance agreement (OMB Number 1121-0339). Specifically, BJS is requesting clearance to conduct an agency-level survey of 400 law enforcement agencies across the 50 states and DC. The goal of the NCS-X agency survey is to determine what barriers exist that have prevented the sampled agencies from contributing to multi-jurisdictional incident-based reporting systems, and how BJS and the U.S. Department of Justice can work to reduce or possibly eliminate these difficulties. BJS considers this survey critical to gathering the background technical assistance information needed to assist the sampled agencies in implementing NIBRS-reporting. The survey will be sent to 400 agencies that do not currently report to NIBRS.

In addition to collecting general agency characteristics, the survey covers four substantive topical areas: 1) learning how agencies currently collect, use, report, and share crime offense or incident information for such crimes as homicide, rape, robbery, assault, burglary, theft, motor vehicle theft, and other offenses; 2) the types of information that agencies record and store when a crime is reported (as well as how agencies record this information and in what format); 3) why agencies have chosen not to contribute to a regional, state, or federal incident-based reporting system; and 4) what might be done to ease their transition to reporting to such a multi-jurisdictional incident-based system.

The survey includes questions that will identify: how agencies collect, record, and store offense/crime incident reports (i.e., paper offense/incident forms, electronic entry, a combination of the two); whether agencies use an automated records management system; the ways in which agencies analyze or use their crime incident data; and the type of information they record and store when a crime is reported. Agencies will also be asked whether they report individual offense/incident reports to any state or regional criminal records repository or information sharing programs, which may be potential sources of incident-based data from a large number of agencies. In addition, agencies will be asked about the resources they may require to begin contributing to their state’s incident-based reporting system.

This information will be used to help BJS develop cost estimates and feasibility assessments for NCS-X implementation by providing an understanding of the technical capacities of non-reporting agencies. Agency capability to participate in incident-based reporting via NIBRS varies greatly, especially among smaller agencies, and understanding the nature of this variability is critical for NCS-X planning and cost estimation (See Attachment D for the final draft of the survey instrument.).

Design of the Survey

Pilot Testing:

The pilot test of the National Crime Statistics – Exchange (NCS-X) Agency Survey involved a thorough testing of the survey instrument, which was developed by RTI International (RTI) and the Police Executive Research Forum (PERF) with support from the Bureau of Justice Statistics (BJS) and the NCS-X Team. The goal of the pilot testing was to ensure that the instrument’s form and content were understandable, appropriate, and not too burdensome. We received completed pilot surveys from 5 law enforcement agencies (LEAs); a sixth provided feedback only. These LEAs represented small, mid-size, and large agencies (in terms of their number of sworn officers) from geographic locations across the US.

The instrument was sent to each agency with instructions to complete the survey just as they would if they had received the survey as part of the regular sample of 400 agencies. They were asked to take note of any aspects of the instrument that were unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing, and to mark these comments directly on the survey instrument.

The team conducted follow-up cognitive interviews¹ via telephone with a representative from each of the selected LEAs to discuss their comprehension of the questions and to focus on the cognitive processes that respondents used in responding – both covert, hidden processes and overt, observable ones.^2,3 A thorough interview guide was prepared for these cognitive interviews that focused on the most important questions asked in the survey, as well as the technical language and terms used in the instrument to be sure they were clear and straightforward.

As a result of the pilot study, several items on the survey were modified. For example, it became apparent during the cognitive interviews that respondents were unsure whether to respond to RMS questions based on the software installation date or the software upgrade date (item 7). This was clarified with the addition of a question asking about the last software update. In item 9, agencies were asked about their participation in the National Data Exchange (N-DEx). Some agencies participating in the pilot were unfamiliar with N-DEx. For clarification, a footnote was added to explain N-DEx’s purpose and goals. Later in the survey, the ordering of a Likert scale was adjusted to remain consistent between questions, specifically for questions 22 and 23. Finally, in item 25, a response choice was added to give agencies a greater range of options regarding the benefits to reporting crime data.

Additionally, more substantive wording changes were also made to the survey. In earlier drafts, the survey referred repeatedly to “multi-jurisdictional incident-based reporting systems.” This wording was replaced with “regional, state, or federal incident-based reporting systems” to be as inclusive and straightforward as possible. The research team also addressed a concern from the pilot testing that affected a series of questions about the types of information that agencies record and store when a crime is reported. Agencies were unable to successfully differentiate between two of the response choices; this problem was resolved by emphasizing certain words in italics and adding an example in parentheses. The final editing change made to the survey affected item 22, which asked agencies about their reasons for reluctance in contributing to NIBRS or a similar system. A response choice was added to address possible concerns with the privacy issues that might arise from reporting rape and sexual assault crimes.

Selection of NCS-X Sample

The NCS-X sample design is a stratified random sample of police jurisdictions identified by the FBI as of 2011. The list of all jurisdictions was divided into groups or strata using three characteristics: CY2011 participation in the National Incidence Based Reporting System (NIBRS), number of sworn officers in 2011, and government type (State, County or Township, Municipal, Tribal, and Other). A total of 12 strata were created using these three characteristics (see Attachment E for a table describing all 12 strata).

These strata were created after extensive study of the 2009 and 2010 FBI Return A report data for jurisdictions reporting crime in those years. The same 2009 and 2010 FBI Return A data were examined in a simulation study to determination an allocation for each of five types of crime (Murder, Rape, Robbery, Simple Assault, and Aggravated Assault). Based on the results of these simulations it was determined that if a sample of 400 law enforcement agencies would agree to become new NIBRS participants, that these agencies data in combination with the current NIBRS data could generate statistically sound and detailed national estimates of crime known to law enforcement.

Agencies will be allocated across the 12 stratum, with all NIBRS jurisdictions and all non-NIBRS jurisdictions with more than 750 sworn officers being included with certainty in Strata 1 and 2. The remaining agencies will be allocated across 10 additional strata. The stratum sizes were determined on the basis of expected levels of precision of subsequent estimates using an average of the Neyman minimum variance allocations across the five types of crime for each of the 10 non-NIBRS strata. Accordingly, the final sample will consist of all NIBRS and all Non-NIBRS jurisdictions with at least 750 sworn officers in 2011 selected with certainty, and simple random samples of jurisdictions from each of the remaining 10 strata (see Attachment I for a table showing the expected number of agencies that will be in each stratum). A reserve sample of additional agencies will be selected for each of the non-certainty stratum to replace agencies who decline to participate or who are determined after selection to be ineligible agencies (e.g., agencies that have combined with other agencies or that have closed since selection).

Data Collection Procedures

Due to the technical nature of the survey instrument, it is essential that a knowledgeable representative from each agency complete the survey. For this reason, we will employ a nomination process during data collection, whereby the chief from each agency will be asked to identify the appropriate person to complete the survey and ensure that we receive this person’s contact information, so that further correspondence and information about NCS-X and the agency survey can be conveyed directly to him or her. This process will help us ensure that the correct person, who has some knowledge of the agency’s technical capabilities and policies for crime data storage and reporting, will be the person who fills out the survey (see Attachment F for an overview of the data collection process).

We will first contact the chief via a lead letter (see Attachment G) to inform him or her that the survey is forthcoming. Accompanying this lead letter will be the joint statement about the NCS-X from BJS and the FBI, the IACP CJIS Committee Resolution of Support, and a one-page summary that describes the NCS-X. This letter will also inform chief that the survey will be forthcoming within two weeks. At that time, the chief will receive a second (cover) letter (see Attachment H) that will ask him or her to review the online version of the survey and, if they wish, to appoint an informed proxy respondent who can take the survey on behalf of the agency. We will ask the chief or the proxy to send contact the proxy representative’s information to the NCS-X e-mail inbox (ncsx@rti.org) for any future correspondence. Non-responding agencies will also receive a follow-up letter sent either to the Chief or their proxy reminding them to complete the survey (see Attachments I and J).

The NCS-X survey will be converted by PERF into IBM SPSS Data Collection survey software to produce both the Internet and hardcopy survey. The survey interface is user-friendly, which ensures more accurate responses. Because online submission is such an important response method, close attention will be paid to the formatting of the Internet survey instrument. Sample screenshots of the online version of the survey are in Attachment K.

Burden Hours for the Survey

The survey respondents from the six pilot agencies that completed the survey were knowledgeable about their agency’s database administration, technical capabilities, and procedures for collecting, storing, and reporting crime and incident reports. Consequently, they were able to advise BJS on the reactions we were likely to encounter to the survey questions. The pilot test found that the items were easily understood and elicited the information needed to construct a model of agency capability to implement NIBRS reporting in these agencies and the barriers to doing so. The survey required approximately one hour total to complete.

Based on the pilot tests, it is estimated that the burden to conduct the NCS-X agency survey in the 400 sampled agencies is 533.4 hours. The burden hour estimates are divided into 3 tasks: 1) review of the online survey by the chief upon receipt of the cover letter (10 minutes); 2) assigning a knowledgeable proxy representative for the agency to take the survey and sending this person’s contact information to RTI or PERF (10 minutes); and 3) completion of the survey (one hour). The burden hour estimates are based on the pilot tests in the six agencies and the experience of NCS-X team in conducting similar surveys for other studies. Further details about the burden hour estimates are provided in the following table.

Table 1.

Tasks	Average burden per jurisdiction	Total estimated burden hours
Review of online survey by chief	10 minutes	400 agencies X 10 minutes = approximately 66.7 hours
Chief assigns a proxy respondent to take survey and sends contact information to RTI and PERF	10 minutes	400 agencies X 10 minutes = 66.7 hours
Agency representative completes survey	1 hour	400 agencies X 1 hour = 400 hours
Total respondent burden for all agencies = 533.4 hours

Analysis Plan

Data obtained from the NCS-X agency survey data will be analyzed for several primary uses. The data will first to be used to provide detailed agency-level information on the data collection processes and record management systems (RMS) that the sampled NCS-X agencies currently use in their respective agencies. For example, the types of RMS and associated databases that agencies employ for collecting incident-based data, the capabilities of these systems, and the processes and procedures agencies use for collecting incident reports in the field and entering them into these systems. In addition, the analysis will reveal the NIBRS required data elements that agencies can and cannot report using their existing systems. Such information will allow the NCS-X project to develop agency-level technical profiles which will be used to assess the needs and requirements necessary to move NCS-X agencies to NIBRS reporting status.

A second and related use of the data will be to analyze the survey data at the state-level to better understand the technical capabilities and needs within and across states. This assessment will be extremely valuable in determining if particular states are likely to require predominately technical assistance and support to move sampled agencies within their state to NIBRS reporting or if there are other factors (including greater capacity and support from the state programs to the local agencies) that are impeding progress in incident-based reporting.

An additional use of the agency survey data will be to identify barriers that currently prohibit agencies from reporting to the NIBRS program or to other multi-jurisdictional incident-based reporting systems. For example, the survey data will be analyzed to identify the most common barriers overall along with barriers that may be specific to subsets of agencies (e.g., smaller agencies). This process will help BJS determine if there are incentives that might be beneficial for certain types of agencies as well as identify state-specific incentives that would encourage reporting. The survey will also provide data on how incident-based data can be made more useful by identifying how agencies use these data for crime analysis and what needs exist around better utilization of incident-based data (e.g., crime analyst training, improved analysis platforms, etc.).

Finally, the agency survey data will be used to establish categories of sampled agencies based on their technical capabilities to collect, store, and record incident-based crime data. This categorization will serve as an initial step in the calculation of an overall cost estimate for the full implementation of NCS-X for all 400 sampled agencies, which will allow BJS to plan for and evaluate required resources moving forward. Once places into categories, a smaller number of candidate agencies within each category will selected and invited to participate in a pilot implementation process. The pilot implementation process will involve a more detailed assessment of each selected agency’s technical capabilities and the implementation and testing of the necessary technical steps required for incident-based reporting. This will allow BJS to generate a cost estimate both by category and for all 400 NCS-X sampled agencies.

Informed Consent and Data Confidentiality

The lead and cover letters that will be sent to agencies, in addition to the NCS-X 1-page overview, the joint statement written by the FBI and BJS, and the IACP CJIS Committee Resolution of Support will fully explain the purpose of NCS-X and the objectives of the survey specifically. The survey is not intended to collect information on individuals or information that would otherwise be considered sensitive in nature. As such, the activities associated with this task are not considered human subjects research. However, participants will be informed that their participation in the survey is voluntary, that they may decline to answer any and all questions, and may stop their participation at any time.

Data Security

PERF utilizes IBM SPSS Data Collection survey software for its online surveys, and the NCS-X agency survey will be programmed using this software.  IBM SPSS Data Collection allows for the creation of unique user names and passwords. Using unique user names and passwords provides a first layer of security when accessing the survey. Additionally, the data and survey are also maintained behind a firewall that protects the NCS-X survey and data.  Data collected through this survey will be stored on a PERF server and is not processed by an outside server location. To ensure confidentiality of data and contact information collected on behalf of the NCS-X project, the following procedures will be implemented where possible:

• Only research team members and BJS will have access to the survey data.

• Data files will be stored on the PERF computer network, which is password-protected.

• Any staff person engaged in the collection and/or management and/or analysis of data will be required to sign a confidentiality agreement, which will specify requisite privacy regulations. In signing the confidentiality agreement, staff indicates their willingness to comply with privacy regulations.

• Intensive monitoring by the PERF Deputy Director of Research, Bruce Kubu, will be done to ensure compliance with confidentiality protocols.

• At the conclusion of the project, PERF will purge all of its records of any information (e.g., contact logs, notes, names within databases, etc.) that could be used to identify the participating agencies or the responding personnel.

• The final agency survey data set will be provided to BJS.

Contact Information

Questions regarding any aspect of this project can be directed to:

Howard N. Snyder, Ph.D.

Deputy Director

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street NW, Room 2326

Washington, DC 20531

Office Phone: 202-616-8305

Fax: 202-616-1351

E-Mail: Howard.Snyder@ojp.usdoj.gov

Attachments

Attachment A: FBI/BJS Joint Statement of Support for NCS-X

Attachment B: Brief Overview of the NCS-X

Attachment C: IACP CJIS Committee Resolution of Support

Attachment D: NCS-X Final Agency Survey Draft

Attachment E: Sample Design Tables

Attachment F: Graphical Representation of Data Collection Procedures

Attachment G: Agency Survey Lead Letter

Attachment H: Agency Survey Cover Letter

Attachment I: Agency Survey Follow-up Letter – sent to Chief

Attachment J: Agency Survey Follow-up Letter – sent to Chief’s proxy

Attachment K: Online Survey Sample Screenshots

1 Areas covered during these interviews included the respondents’ understanding of the intent of each question; the meaning of specific words and phrases in the questions; the types of information respondents need to answer the questions; the respondents’ ability to match their answer to the response categories provided in the instrument; and the types of cognitive strategies used by respondents to retrieve the information.

2 Tourangeau, R. (1984). “Cognitive science and survey methods.” In T. Jabine, M. Straf, J. Tanur, & R. Tourangeau (Eds.), Cognitive aspects of survey methodology: Building a bridge between disciplines. Washington, DC: National Academy Press.

3 Willis, G., DeMaio, T., Harris-Kojetin, B. (1999). Is the bandwagon headed to the methodological promised land? Evaluation of the validity of cognitive interviewing techniques. In M. Sirken, D. Herrmann, S. Schechter, N. Schwartz, J. Tanur, & R. Tourangeau, (Eds.), Cognition and survey research. New York: Wiley.