Memo to OMB on CPFCL Generic Clearance

MEMORANDUM

TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Jeffrey H. Anderson

Director

Bureau of Justice Statistics

Allen Beck

Senior Statistical Advisor

Bureau of Justice Statistics

Devon Adams

Acting Deputy Director

Bureau of Justice Statistics

Kevin Scott

Chief, Law Enforcement Statistics Unit

Bureau of Justice Statistics

FROM: Connor Brooks

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: July 13, 2020

SUBJECT: BJS request for Generic Clearance to conduct: 1) verification of reporting units for the 2019 Census of Publicly Funded Forensic Crime Laboratories (CPFFCL) and 2) cognitive interviews to finalize CPFFCL survey instrument, under the generic clearance agreement OMB Number 1121-0339

Introduction

In late 2020 or early 2021, the Bureau of Justice Statistics (BJS) is planning to conduct the Census of Publicly Funded Forensic Crime Laboratories as part of a series which began in 2002 and was most recently conducted in 2015. This data collection series provides national statistics on personnel, budgets, workloads, backlogs, and quality assurance practices of crime laboratories. The goal of this iteration of the CPFFCL is 1) to continue to collect the same information as previous studies in order to report on the current state of the field and assess trends, and 2) introduce new questions that provide a more complete picture of the workload of crime laboratories.

This generic clearance is to request approval to conduct two activities in preparation for the 2020 administration of the CPFFCL under BJS’s generic clearance agreement (OMB Number 1121-0339). First, BJS plans a frame verification effort with telephone outreach to confirm or collect contact information for a portion of the frame. Second, BJS plans to conduct cognitive interviews to test the survey instrument for clarity and interpretation of the questions and answer options. More details about each of these requests follow.

Request for Frame Verification Telephone Outreach Effort

In the 2014 CPFFCL, BJS included 409 federal, state, and local publicly funded forensic crime laboratories in its frame. The inclusion criteria for a laboratory that has been used in the CPFFCL series is that a laboratory employs one or more full-time scientists 1) with a minimum of a bachelor’s degree in chemistry, physics, biology, criminalistics, or a closely related forensic science field, and 2) whose principal function is examining physical evidence in criminal matters and providing reports and testimony to courts of law regarding such evidence.

BJS has permission from the Drug Enforcement Administration (DEA) to use the survey frame from the National Forensic Laboratory Information Management System (NFLIS) (DEA; OMB Clearance #1117-0034, Expiration Date: May 31, 2022) to augment existing information about laboratories eligible for the CPFFCL. Of the 409 laboratories currently on the CPFFCL frame, 172 were surveyed as part of the NFLIS-Drug Survey of Crime Laboratory Drug Chemistry Sections survey in 2019. All but seven of those laboratories had at least some interaction with the NFLIS study team through prompting calls, data quality follow up calls, or nonresponse follow-up calls.

BJS requests clearance from OMB to collect or confirm contact information from the seven laboratories who were not contacted for or did not respond to the 2019 NFLIS survey and the 237 forensic laboratories that were not part of the 2019 NFLIS survey effort because they lacked drug chemistry sections. As part of obtaining this information, the project team will conduct internet searches and search for these offices on professional lists (e.g., American Society of Crime Laboratory Directors) and using the National Institute of Justice’s recent list of Paul Coverdell grantees. Because this is a community that is regularly surveyed, it is not anticipated that internet searches and list comparisons will yield more than 50 new laboratories. Thus, we anticipate no more than 300 laboratories will require verification calling efforts. BJS expects to begin the verification effort in the late summer or early fall.

Appendix A provides the list of the proposed twelve verification questions, which include skip patterns. The verification call effort will involve project staff calling the laboratories to confirm information currently on record with respect to the laboratory’s official name, address, and the name and contact information for the laboratory director. The verification call effort will also help identify a survey point of contact (and that person’s contact information) best suited to answer the survey questions.

Burden Hours for the Frame Verification Telephone Outreach Effort

Table 1 provides a summary of the burden estimate for this effort. The project team will contact up to 300 crime laboratories via telephone. Respondents will need an average of 5 minutes to identify an appropriate person within the office to answer the questions if the gatekeeper is unable to answer the questions. BJS estimates that, once contact is made, gatekeepers will be able to provide the needed information in 5 minutes.

Table 1. Burden Estimate for the Frame Verification Telephone Outreach Effort

Task	Average burden per laboratory	Total estimated burden hours
Identify appropriate respondent for verification questions	5 minutes	300 respondents x 5 minutes = 25 hours
Confirm information	5 minutes	300 respondents x 5 minutes = 25 hours
Total respondent burden for all respondents = 50 hours

Request for Cognitive Interviewing

BJS also requests clearance to conduct one round of cognitive testing of the 2019 CPFFCL survey form. The project team will conduct up to 24 cognitive interviews over the telephone. To prepare the 2019 CPFFCL form, the project team reviewed the 2014 CPFFCL and conducted a data quality assessment of the 2014 responses to identify questions with high nonresponse. Questions with high item missingness were identified as candidates for deletion or revision. The project team also met with an expert panel for two days to review the 2014 survey form for clarity, to ensure questions are still relevant to the field, and to suggest new questions that will help address gaps in knowledge. This feedback was compiled, resulting in the addition, deletion, and revision of questions. BJS reviewed changes and additions to questions.

The cognitive interview protocol is designed to assess the survey form for general understanding, question and response wording, and survey design, all of which will help minimize survey burden and improve data quality. These interviews will also let BJS assess if the changes and additional questions are performing as intended. The goal of this effort is to understand how well the questions work when administered to a purposive subset of the survey’s target population, which will include a mix of federal, single state laboratories (e.g., the West Virginia State Police Forensic Laboratory), states with multiple laboratories (e.g., Texas Department of Public Safety), regional laboratories, and local laboratories that serve large and small jurisdictions and across geographical regions.

Given the ongoing COVID-19 pandemic, and BJS’s desire to be sensitive to the needs of the community, BJS, through the project team, has reached out the American Society of Crime Laboratory Directors (ASCLD) leadership to assess the field’s openness to and capacity for participating in a cognitive interview effort. From these communications, as of May 15, 2020, BJS has determined that depending on the area of the country, the status of crime laboratory operations differs. There are 3 local laboratories and 1 state system that have suspended their operations altogether; 1 local laboratory and 4 state systems are operating at reduced capacity; and 3 local laboratories are busier because of COVID-19. BJS expects that the crime laboratory community may be taxed upon return to work or potentially have less to do, depending on the volume of cases that await them.

ASCLD leadership has suggested that there are many laboratory staff, including laboratory directors, who are working from home, but it is difficult to ascertain whether the extenuating circumstances mean that they have more or less time (e.g., they are working from home but are caring for dependents). Given the varying statuses of the community and the unprecedented circumstances posed by COVID-19, the usual approach of BJS and its data collection agent reaching out to prospective participants may not be the most effective recruitment strategy. Thus, BJS plans to coordinate outreach with ASCLD, a subcontractor to the data collection agent.

To identify laboratory directors available to participate in the cognitive interviews, ASCLD leadership will send a mass email to their list of laboratory directors that acknowledges the difficulties associated with COVID-19 and respectfully requests any volunteer laboratory directors who have time to assist with the study (see Appendix B-1). The BJS letter explaining the survey will be attached to the ASCLD outreach email (see Appendix B-2). Any interested parties will be asked to contact the project team directly via email.

From the emailed responses, project staff will use the 2014 CPFFCL data and an Internet search to get basic information about the responding laboratory, including full laboratory name, where it is located, government level of laboratory (i.e., federal, state, county, or municipal), and size of laboratory based on known information about the number of full-time personnel. These preliminary results will be compared with the other scheduled or completed interviews to see how the prospective laboratory’s characteristics would support diversity in respondents. Next, the project team staff will call the prospective respondents within two business days to confirm this information through a series of screening questions (see Appendix B-3) to determine final eligibility (see more details about this on the next page).

BJS intends to include 24 laboratories in these interviews stratified on 1) government-level administration of the laboratory (i.e., federal, state, county, or municipal) and 2) size of laboratory based on known information about the number of full-time personnel. Screening potential participants allows BJS the possibility of obtaining laboratory diversity within this opt- in convenience sample while minimizing burden since the team can draw on existing information about the laboratory prior to the call that can be quickly confirmed over the telephone. BJS expects that this approach will yield enough respondents to allow the project team to purposively select laboratories that handle different types of forensic examinations, differing caseloads, and with varied levels of resources across different parts of the U.S. The team will ask ASCLD to send out one more reminder if this process does not yield a diverse set of respondents (see reminder message in Appendix B-4).

Sample Stratification Characteristic	Characteristic Values
Laboratory government level	Federal
	State
	County
	Municipal or Local
Size of laboratory	Large (25 or more full-time employees)
	Small (fewer than 25 full-time employees)

Using the texts included in Appendix B, the project team will contact screen and confirm the laboratory information with prospective CPFFCL cognitive interview respondents. If eligible, respondents will be invited participate in 60-minute telephone cognitive interviews (see screener questions in Appendix B-3). If ineligible, respondents will be thanked for their time, consideration, and willingness to participate in cognitive interviews. The project team will either schedule the cognitive interview during the screening call, or if it is a bad time to do that activity, will call back at another time to schedule the cognitive interview. Upon confirming the laboratory’s eligibility and willingness to participate in the cognitive testing, the project team will email the participant a packet including the consent form (Appendix C-1) and the CPFFCL form (Appendix D). Because BJS is interested in how respondents understand and interpret questions rather than the actual answers to the questions, respondents will not have to complete the questionnaire and return it. Participants will receive instructions that clearly state they do not have to complete the form but are encouraged to review it ahead of their scheduled interview.

Laboratories that express interest but are not eligible or represent a stratum from which BJS has met its interview quota will be politely thanked for their interest.

Participation in the cognitive test will be completely voluntary. All participants will be 18 years of age or older and will be either laboratory directors or deputy directors. Prior to the scheduled interview, the participants will be provided information that describes the interview, why they were chosen, what will happen during the discussion, the risks and benefits of participation, and details ensuring confidentiality (see Appendix C-1). At the time of the scheduled interview, project staff will review these materials with the respondents, including the consent form. They will have the option to refuse to answer a question at any time during the interview. Since these are telephone interviews, the interviewer will read the consent form to the participant and obtain verbal consent. After receiving verbal consent, the interviewer will check and sign the hard copy informed consent to have on record, certifying they have obtained the participant’s permission to continue with the interview (see Appendix C-2). Interviews will only be conducted if consent is obtained. If a participant refuses, they will be thanked for their time and consideration.

If the participant consents to audio recording, the interviews will be recorded to allow for accurate capture of responses and account of answers during the analysis stage. Audio recordings will be saved on the project team’s secured project share site and, if transferred to BJS, will be transferred via secure FTP and stored on BJS’s secure servers. The files will be deleted once the cognitive testing report is finalized. Audio recordings will not be transferred outside of BJS or RTI. No personally identifiable information will be included in the recordings. If a participant consents to an interview, but does not consent to being recorded, the interview will be conducted with the interviewer taking notes of the participant’s responses. A thank you letter will be sent via email to participants within 48 hours of the completion of the cognitive interview via email (see Appendix B-4).

The survey topics in the questionnaire include—

• Section A: Organization includes questions on the type of laboratory, position in a multi-lab system, jurisdiction served, and functions and procedures performed

• Section B: Budget includes questions on total annual budget, budget categories, cost of outsourcing, and funding sources

• Section C: Staffing includes questions on the number of employees in various roles, certification of employees, salaries, hires, and separations

• Section D: Workload includes questions on the number of requests and items for examination received by the laboratory, archiving of digital data

• Section E: Outsourcing includes questions on reasons for outsourcing and types of evidence being outsourced

• Section F: Quality Assurance includes questions on accreditation, research, proficiency testing, and review processes

• Section G: Feedback & Submission provides a free-text area in which respondents can provide feedback or any other relevant information.

Because the CPFFCL form has performed well in the past, BJS intends the cognitive interviews to focus only on the substantially revised or new questions to see if respondents are able to understand and answer the question. However, participants will see the full instrument so that they may see all questions in context. The new or substantially revised questions include—

• Section A: Questions A2, A6, A7

• Section B: Questions B1, B2

• Section C: Questions C2, C3

• Section D: Expert panel feedback indicated that analysis of items may be a better metric by which to measure the workload of laboratory than number of requests received and completed. BJS added sub-items to questions D1–D17 to obtain the number of items for examination received by labs. Questions D18 and D19 are also new.

• Section E: Questions E2, E4, and E5

• Section F: Questions F1, F2, F3, F4, F8, F9, F11– F13.

The testing protocol and example probes are presented in Appendix E. The respondents will see only the survey items (Appendix D) and not the probing questions (items in blue in Appendix E). Each cognitive interview, including probes, will take a maximum of 60 minutes. The cognitive testing protocol will introduce the participants to BJS’s goals for the 2019 CPFFCL, goals and guide participants through the questionnaire.

Interviewers will guide participants through the form and ask them to note any aspects of the instruments that are unclear, any questions or topics omitted, or any answer choices or response categories missing or insufficient, or questions that should be struck or revised, or identify places where the question flow should be modified. Interviewers will instruct participants to answer only those questions that they can readily answer; participants are not expected to conduct research to answer any questions. For those questions that are not readily answered without researching information, we will ask participants how they would find the information necessary to answer the question and who would be the person best suited to answer the question. For the new or substantially revised questions discussed above, interviewers will also ask probing questions to check participants’ understanding of the questions and whether the questions are answerable as intended. To gauge burden, we will also ask participants how long they think it would take them to complete the form on their own, including any time necessary for research or involving other staff members.

Participants will not receive any compensation for the interview but will receive a thank you email. The project team will review the feedback from the cognitive interviews and revise the survey instruments as necessary. Cognitive interviews will take place in summer 2020. All information gathered from the CPFFCL cognitive testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB in fall 2020.

Burden Hours for the Cognitive Interviews

The burden hour estimates are:

Table 2. Burden Estimate for Cognitive Interviews

Task	Average burden per laboratory	Total estimated burden hours
Initial contact and scheduling	10 minutes	24 respondents x 10 minutes = 4 hours
Complete cognitive interview	60 minutes	24 respondents x 60 minutes = 24 hours
Total respondent burden for all respondents = 28 hours

Institutional Review Board

The project team has obtained approval from RTI’s IRB to ensure the testing protocols are compliant with informed consent and data confidentiality standards (Appendix F).

Contact Information

Questions regarding any aspect of this project can be directed to:

Connor Brooks

CPFFCL Program Manager

Bureau of Justice Statistics

Office of Justice Programs

U.S. Department of Justice

810 7th Street NW

Washington, DC 20531

Office Phone: 202-514-8633

E-Mail: Connor.Brooks@usdoj.gov

Appendices

Appendix A:	2019 Census of Publicly Funded Forensic Crime Laboratories Verification Call Script
Appendix B:	2019 Census of Publicly Funded Forensic Crime Laboratories Cognitive Testing Recruitment Materials
Appendix C:	2019 Census of Publicly Funded Forensic Crime Laboratories Cognitive Testing Informed Consent
Appendix D:	2019 Census of Publicly Funded Forensic Crime Laboratories Instrument
Appendix E:	2019 Census of Publicly Funded Forensic Crime Laboratories Cognitive Interviewing Protocol
Appendix F:	IRB Submission Confirmation

9