OMB Memo on Cognitive Interview for LEMAS-FSS

MEMORANDUM

MEMORANDUM TO: Jennifer Park

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

Gerard Ramker

Deputy Director

Acting Chief, Law Enforcement Statistics Unit

FROM: Shelley Hyland

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

Anthony Whyde

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: March 3, 2017

SUBJECT: BJS request for OMB Clearance to conduct cognitive interviewing for the 2017 Law Enforcement Management and Administrative Statistics Forensic Services Supplement (LEMAS-FSS), under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) is planning to conduct a new data collection to better understand the scope of forensic activity carried out by law enforcement agencies (LEAs) through the Law Enforcement Management and Administrative Statistics Forensic Services Supplement (LEMAS-FSS). The primary purpose of the LEMAS-FSS is to provide national estimates of the types of forensic services provided by sworn law enforcement officers and non-sworn employees, accreditation for these services, certification for those engaged in forensics services, evidence processing and storage, and the number of agencies who have forensic units. This survey will ask about law enforcement personnel who collect evidence, analyze and interpret evidence, or testify in court on forensic evidence in the following forensic science disciplines:

• Biological examinations (e.g., DNA)

• Controlled substances

• Crime scene (e.g., evidence collection, reconstruction, forensic photography)

• Digital and multimedia examinations (e.g., computer, network, mobile device, video, audio)

• Document examinations

• Fingerprints (e.g., latent prints, ten prints)

• Fire investigation

• Firearms, ballistics and tool mark examinations

• Footwear/Tire tread impressions

• Toxicology

• Trace evidence

• Traffic accident reconstruction

This new data collection is a component of a larger forensic science agenda established by the Department of Justice through the National Commission on Forensic Science (NCFS), which is tasked with enhancing the practice and reliability of forensic science. In August 2014, the NCFS passed a directive that stated “the Attorney General should direct the Bureau of Justice Statistics to create a proposal for the development of a nationally representative survey to determine forensic capabilities for those who write reports and offer testimony within federal, state and local law-enforcement agencies and for medical examiner and coroner offices.” The purpose of this proposed data collection was to gain a better understanding of the work being done in law enforcement forensic units, since policies on standards, accreditation and certification created by NCFS will directly impact this community. The NCFS stated a primary goal of the BJS law enforcement survey was to identify the LEAs conducting forensic science since the entire universe of forensic science service providers is currently unknown. Without the data, the NCFS will not know if their recommendations for improving the practice and reliability of forensic science will be feasible. The Attorney General approved the NCFS’s directive on September 8, 2014.

On December 16, 2016, the President enacted the Justice for All Reauthorization Act of 2016 (Public Law 114-324), which tasked the Department of Justice to study and report on the status and needs of the forensic science community. Specifically Section 16. Needs Assessment of Forensic Laboratories, states that “the Attorney General shall conduct a study and submit a report to the Committee on the Judiciary of the Senate and the Committee on the Judiciary of the House of Representatives on the status and needs of the forensic science community” no later than October, 1 2018. Furthermore this report shall consider the BJS’s past surveys on publicly funded crime labs and medical examiner/coroner offices and the proposed survey on law enforcement forensic providers.

This generic clearance request is to properly test the instruments for law enforcement forensic service providers that has been developed to meet the needs of the NCFS’s directive and inform the report required by the Justice for All Reauthorization Act of 2016.

The BJS has conducted two surveys that have informed the forensic science landscape: the Census of Medical Examiners and Coroners (CMEC; OMB # 1121-0296) and Census of Publicly Funded Forensic Crime Laboratories (CPFFCL; OMB # 1121-0269). The CMEC was first conducted in 2004. The BJS is working on an interagency agreement with NIJ in order to field a second CMEC in 2018 to provide current information on the personnel, budgets, and workload of medical examiner and coroner’s offices with forensic science evidence.

The CPFFCL has been conducted four times: 2002, 2005, 2009 and 2014. The CPFFCL focuses on the forensic services performed by publicly funded crime laboratories capturing data on budget, staffing, workload, outsourcing and accreditation. In 2014, the CPFFCL was administered to the 409 publicly funded forensic crime laboratories with one or more full-time scientists who possess a minimum of a bachelor’s degree in a natural science, criminalistics, or forensic science and whose principal function is the examination of physical evidence in criminal matters and testimony to courts with respect to such evidence. Additionally, crime labs that only examine digital or multimedia evidence were include for the first time in 2014.

Approximately 135 of the forensic crime labs in the CPFFCL are located within local, county or state law enforcement agencies. These agencies fall within the certainty strata for LEMAS sampling and are necessary in order to have national representation. Therefore, the currently proposed LEMAS-FSS will have two instruments: one for agencies who received the 2014 CPFFCL and one for agencies that were not in the 2014 CPFFCL (were not eligible for the survey).

While the instruments will largely be the same across the two groups, having separate instruments will ensure that we are not duplicating what was collected in the 2014 CPFFCL. Specifically, for CPFFCL agencies, we will ask about personnel who engage in forensic science but only for those who work primarily outside the forensic crime laboratory (e.g., sworn officers/detectives that collect or analyze evidence). We will also ask about evidence processing and storage which was outside the scope of the CPFFCL. Non-CPFFCL agencies will be asked about the structure of forensic science services in their agency (i.e., crime lab, forensic unit, no unit/lab but some personnel who engage, no personnel who engage). Additionally, for non-CPFFCL agencies, the instrument will focus on any personnel who engage in forensic science regardless of whether the agency has a forensic laboratory since there is currently no data on the forensic activities occurring in these agencies.

Request for Cognitive Interviewing

This request is to conduct cognitive testing for the two LEMAS-FSS instruments under BJS’s generic clearance agreement (OMB Number 1121-0339). The BJS has contracted with RTI International (RTI) to develop and test the instruments. RTI has awarded a subcontract to the Police Executive Research Forum (PERF) to assist with recruitment for cognitive testing for the LEMAS-FSS because of their long standing relationship with law enforcement agencies. Project staff from RTI will cognitively test (1) the CPFFCL respondent survey (Attachment A) and (2) the non-CPFFCL respondent survey (Attachment B).

The cognitive interviewing plan described below is designed to (1) thoroughly test the survey instruments to ensure question clarity and reliability and (2) assess the capability of law enforcement agencies’ ability to identify a proper respondent for the survey and to provide the requested information. The project staff will review the results of the cognitive interviews and modify the survey instruments as necessary. Following the completion of testing, the BJS plans to implement the full-scale LEMAS-FSS beginning in October 2017.

Forty (40) law enforcement agencies will be selected to participate in cognitive interviews: 20 LEAs who were in the CPFFCL and 20 LEAs who were not included in the CPFFCL (non-CPFFCL). The intended respondent for the survey is the agency’s chief or another staff member who is knowledgeable about the law enforcement personnel that engage in forensic science. For CPFFCL agencies this will require the agency’s chief executive to identify a respondent who is familiar with personnel engaging in forensic science that work primarily outside the forensic crime laboratory. For non-CPFFCL agencies, the agency’s chief executive may be the respondent or will need to select a representative that is knowledgeable about the law enforcement personnel who provide forensic services regardless of a laboratory or unit structure of the agency.

The project team will develop a preliminary list of agencies for cognitive testing based on a few select criteria partially representing how the full sample will be drawn. The project team will randomly select agencies based on the following three stratification categories: (1) CPFCCL Status (CPFCCL vs. non-CPFCCL), (2) agency size (small: less than 250 full-time sworn, medium: 250-999 full-time sworn, large: 1000 or more full-time sworn), and (3) agency type (local police department and sheriff’s office). The project team is confident that the resulting list of agencies prioritized for cognitive interviewing will be diverse in characteristics and will provide a sampling of potential obstacles to the proposed information collection.

The recruitment of agencies for cognitive testing will be conducted by phone and led by PERF. Upon selection and agreement to participate in cognitive testing, the participants will be asked to provide a point of contact (POC) and their email address. RTI will email the POC a link to the online survey and a request to complete the questionnaire. Once completed, RTI will be notified and will schedule a cognitive interview by phone with the POC. The POC and the interviewer will both have access to the completed instrument to use as a reference while the interviewer follows the cognitive testing protocol to complete the interview. Separate cognitive testing protocols, including the informed consent, have been developed for CPFCCL participants (Attachment C) and non-CPFCCL participants (Attachment D). Participants will be asked to take note of any aspects of the instruments that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. Participants will not receive any compensation for the interview. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary.

Concepts and topics for LEMAS-FSS questionnaires were developed and prioritized through ongoing discussions between the LEMAS project team, NIJ, NCFS, and other key stakeholders. NCFS formed a subcommittee of experts to take the lead on survey development. The BJS and RTI worked closely with the subcommittee to identify core estimates and items that could answer key research questions. The questionnaire was further refined into the current forms by the BJS, RTI and NIJ personnel.

The survey topics include agency characteristics, formalization of forensic services in the agency, scope of forensic activities undertaken and outsourced, and evidence processing and screening. For seven key forensic science disciplines (fingerprints, digital and multimedia evidence, firearms & ballistics/tool marks, traffic accident reconstruction, fire investigation, crime scene (general), and controlled substances) additional questions will be asked to determine the number of sworn and non-sworn personnel, certification and training, and agency accreditation. The CPFFCL version differs from the non-CPFFCL version in three ways. First, after confirming the CPFCCL agencies have a forensic laboratory, we will not need to ask them about their options for using other forensic laboratories. Secondly, we will not need to ask about accreditation status, caseload or backlogs of CPFCCL agencies as this was obtained in the 2014 CPFCCL. Finally, the CPFFCL version also contains some additional wording to remind respondents to consider only those personnel who are not primarily assigned to a forensic laboratory.

Cognitive testing will also provide an opportunity for the BJS to calculate more reliable burden estimates for law enforcement agencies to complete the questionnaires. This information will be taken into consideration as the BJS continues the development and design of the LEMAS-FSS. All information gathered from the LEMAS-FSS testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB by June 2017. The BJS plans to fully implement the LEMAS-FSS by October 2017. All of the dates noted above are planned and pending final approval.

Burden Hours

The burden hour estimates are provided in the following table. Estimated burden hours are the same for all 40 respondents regardless if they receive the CPFFCL or non-CPFFCL version of the surveys.

	Average burden hours per respondent	Total respondents	Est. burden hours
Initial contact and scheduling	0.17	40	6.8
Complete questionnaire	0.50	40	20.0
Complete cognitive interview	1.00	40	40.0
TOTAL HOURS			66.8

Institutional Review Board

The project team obtained approval from RTI’s IRB to ensure the cognitive testing protocol is compliant with informed consent and data confidentiality standards.

Contact Information

Questions regarding any aspect of this project can be directed to:

Shelley Hyland

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 305-5552

E-mail: Shelley.Hyland@usdoj.gov

Anthony Whyde

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 307-0711

E-mail: Anthony.Whyde@usdoj.gov

Attachments

Attachment A: LEMAS-FSS: CPFFCL respondent survey

Attachment B: LEMAS-FSS: Non-CPFFCL respondent survey

Attachment C: CPFFCL informed consent and cognitive testing protocol

Attachment D: Non-CPFFCL informed consent and cognitive testing protocol

6