SSCIA_PartB_FINAL

Part B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The Survey of State Criminal Investigative Agencies on Law Enforcement Use of Force (SSCIA) will collect data on the role state criminal investigative agencies (SCIAs) have in use of force investigations. SCIAs are state agencies “having responsibility for the division, unit, or bureau primarily created for the purpose of general criminal investigation having statewide jurisdiction and whose personnel have full peace officer power” (ASCIA, 2016). There are currently 49 SCIAs in the United States (Table 5).¹ All 49 SCIAs will be asked to participate in the SSCIA.

Table 5. List of State Criminal Investigative Agencies

Agency	Agency (cont.)
Alabama Bureau of Investigation	Nebraska State Patrol
Alaska Bureau of Criminal Investigation	Nevada Department of Public Safety
Arizona Department of Public Safety	New Hampshire State Police
Arkansas State Police	New Jersey State Police
California State Bureau of Investigation	New Mexico State Police
Colorado Bureau of Investigation	New York State Police
Connecticut State Police	North Carolina Bureau of Investigation
Delaware State Police	North Dakota Bureau of Criminal Investigation
Florida Department of Law Enforcement	Ohio Bureau of Criminal Investigation
Georgia Bureau of Investigation	Oklahoma State Bureau of Investigation
Idaho State Police	Oregon State Police
Illinois State Police	Pennsylvania State Police
Indiana State Police	Rhode Island State Police
Iowa Department of Public Safety	South Carolina Law Enforcement Division
Kansas Bureau of Investigation	South Dakota Division of Criminal Investigation
Kentucky State Police	Tennessee Bureau of Investigation
Louisiana State Police	Texas Department of Public Safety
Maine State Police	Utah Department of Public Safety
Maryland State Police	Vermont State Police
Massachusetts State Police	Virginia State Police
Michigan State Police	Washington State Patrol
Minnesota Bureau of Criminal Apprehension	West Virginia State Police
Mississippi Bureau of Investigation	Wisconsin Division of Criminal Investigation
Missouri Highway Patrol	Wyoming Division of Criminal Investigation
Montana Division of Criminal Investigation

2. Procedures for Collecting Information

Data Collection Procedures. BJS has established a partnership with the Association of State Criminal Investigative Agencies (ASCIA) to facilitate data collection from SCIAs. Currently, 41 SCIAs are members of ASCIA. The President of ASCIA, Mark Gwyn, and the Chair of the ASCIA Use of Force Committee, Vernon Keenan, support the SSCIA and have composed a letter to be included with the initial survey invitation to encourage responding (Attachment 2). Additionally, ASCIA has a contact list including those representatives who are on the Use of Force Committee and knowledgeable on the topics covered in the SSCIA. ASCIA leadership also has contact information for key personnel in the eight SCIAs that are not members. Furthermore, Vernon Keenan has offered to send out a non-response reminder email, discussed below. BJS will work with RTI International for the data collection through the Analytic Resource Center.

The SSCIA will involve six touch points (Table 6). Data collection will begin with a survey invitation letter emailed to the point of contact (POC) for each SCIA to inform them about the survey. The email will be signed by the Director of BJS and will explain the purpose and significance of the survey. It will also include the survey web address and agency-specific log-in credentials (Attachment 8). The survey invitation letter will also provide a toll-free telephone number and project-specific e-mail address for the survey should the POC have any questions. Attached to the survey invitation email will be a letter of support from the ASCIA leadership (Attachment 2).

Within two weeks of submitting the survey, the respondent will receive a thank you email (Attachment 9). The thank you will formally acknowledge receipt of the survey and state that the agency may be contacted for clarification once their survey responses are processed.

Table 6: Agency Outreach Schedule

Activity	Format	Recipient	Timeline
Invitation letter	Email	All	Day 1
1st Reminder	Email	Non-respondents	Day 15
2nd Reminder	Email	Non-respondents	Day 29
Telephone prompting	Phone	Non-respondents	Day 43
End of study	Email	Non-respondent	Day 71
Thank you email	Email	Respondents	Within 2 weeks of survey submission

Two weeks after emailing the survey invitation letters, a reminder email will be sent to all POCs (Attachment 10). These emails, signed by the BJS Project Manager, will express the importance of the SSCIA and encourage response via the online survey (or paper copy, if preferred). The reminder email will also include a copy of the .PDF questionnaire, so that respondents may submit via email, fax or mail rather than online.

Two weeks after the reminder emails are sent, a second reminder email from Director Vernon Keenan will be sent to the POCs who have not responded (Attachment 11) that will include the website information and instructions on how to download another paper copy of the questionnaire if needed. Two weeks after sending the second reminder, we will begin telephone follow-up with all non-responding SCIAs. The procedures for this activity and the final nonresponse follow-up contact are discussed in Section 3.

Upon receipt of a completed survey (web or paper copy), data will be reviewed and edited, and if needed, the respondent will be contacted to clarify answers or provide missing information. RTI will enter data from all paper copy questionnaires into the web survey application. This will ensure that the same data quality review procedures are applied to all survey data, regardless of response mode. Prior to contacting the respondent, RTI staff will address data inconsistencies via BJS-approved editing specifications.

RTI also will ensure that responses fall within the proper coding schemes specified by BJS. The following is a summary of the data quality assurance steps that RTI will observe during the data collection and processing period:

Within Survey Checks. The web system will have built-in checks to validate data fields to the extent possible. In order to balance data quality with respondent burden, RTI will implement a series of soft checks on key items. Soft checks prompt the user to provide a response but do not prevent the user from advancing to the next series of questions. Hard checks, by comparison, do not let the user move to the next question if the current question does not meet specifications. In general we will implement soft checks to improve data quality without unduly increasing burden.

Data Editing. RTI will attempt to reconcile missing or erroneous data through automated and manual edits of each questionnaire. In collaboration with BJS, RTI will develop a list of edits that can be completed by referring to other data provided by the respondent on the survey instrument. Through this process, RTI can quickly identify which cases require follow-up and indicate the items that need clarification or retrieval from the respondent.

Data Retrieval. When it is determined that additional data retrieval is needed, an Agency Liaison (AL) will contact the respondent for clarification. Throughout the data retrieval process, RTI will document the questions needing retrieval (e.g., missing or inconsistent data elements), request clarification, obtain values for missing data elements, and examine any other issues related to the respondent’s submission.

Data Quality Review. To confirm that editing rules are being followed, RTI will review frequencies for the entered data after the first 10 percent of cases are received. Any issues will be investigated and resolved. Throughout the remainder of the data collection period, RTI staff will conduct regular data frequency reviews to evaluate the quality and completeness of data collected in both the web and paper copy modes.

3. Methods to Maximize Response Rates

We expect the SSCIA to have a very high response rate given the coordination with and marketing by ASCIA leadership, its member’s interests and state agency responding to the Census of State and Local Law Enforcement Agencies (CSLLEA) and Law Enforcement Management and Administrative Statistics (LEMAS). The 2013 LEMAS and 2014 CSLLEA had 100% response rate for state agencies. Approximately half of the SCIAs reside in state police departments which are included in the LEMAS and CSLLLEA. All of these agencies participated in both waves. The other SCIAs participated in the 2014 CSLLEA. Therefore it is expected we will obtain a nearly 100% response rate to the SSCIA.

The project team has begun marketing the survey to the ASCIA members through committee conference calls and recently spoke about the testing at the in-person meeting in Charleston, SC on May 1. Director Keenan has marketed the survey as a priority to ASCIA members. Additionally, the content of the survey has a direct benefit to not only ASCIA members but the other SCIAs as well. Law enforcement use of force investigations is a prominent topic. Since there is no available information on what is being done by SCIAs nationwide, the SCIAs are also interested in knowing what is occurring in other states. As part of the marketing effort, BJS will provide the final published report on the findings to the SCIAs that participate.

BJS will use a web-based instrument supported by several online help functions to maximize response rates. For convenience respondents will receive the survey link in an email invitation. A toll-free number will be available to provide both substantive and technical assistance. In addition, the web survey interface is user-friendly, which encourages response and ensures more accurate responses. Because online submission is such an important response method, close attention has been paid to the formatting of the web survey instrument. The online application is flexible and can adapt to meet the needs of multiple device types (e.g., desktop computer and tablet), browser types (e.g., Internet Explorer and Google Chrome), and screen sizes. Other features in the instrument include the following:

• Respondents’ answers will be saved automatically, and they will have the option to leave the survey partway through and return later to finish.

• The online instrument will be programmed with data consistency checks and automatic prompts to ensure inter-item consistency and reduce the likelihood of “don’t know” and out-of-range responses, thereby reducing the need for follow-up with the respondent after survey submission.

• Upon submission, respondents will receive a message that confirms receipt of their survey.

• Respondents may also download and print a paper version of the survey from the website.

In order to obtain high response rates and to ensure unbiased estimates, follow-up procedures have been incorporated into BJS’s response plans. Ensuring adequate response (not just agency response rates, but also item responses) begins with introducing SCIAs to the survey. This will be accomplished initially through the initial invitation letter and accompanying documents (Attachments 2, 8). Resources available to help the respondent complete the survey (e.g. telephone- or email-based support) will be described in detail. The survey instrument will include the name of the individual who completed the survey so that it can be used for potential quality control follow-up later. Within the survey invitation letter, we will provide the respondent with a summary of the information that they will need to provide. This will allow them to gather this information before attempting to complete the online instrument.

The data collection schedule is designed to include several follow-up communications to allow the respondent to complete the survey at a time most convenient for them and to ensure a response rate in excess of 95%. Approximately two weeks after emailing the survey invitation letter, BJS will send an e-mail reminder (Attachment 10). The email will express the importance of the SSCIA and encourage response via the online survey. This reminder will also include a PDF of the questionnaire, so that respondents may submit the survey via email, fax or mail. Within two weeks of receiving a survey, the respondent will receive a thank you email or letter depending on the completion mode (Attachment 9). The thank you will formally acknowledge receipt of the survey and indicate that the agency may be contacted for clarification once their survey responses are processed.

Two weeks after the reminder emails are sent, a second reminder email from Director Vernon Keenan will be sent to the POCs who have not responded (Attachment 11) that will include the website information and instructions on how to download another paper copy of the questionnaire if needed.

Two weeks after sending the second reminder, we will begin telephone follow-up with all non-responding SCIAs (Attachment 12). In preparation for this outreach, an Agency Liaison (AL) will be trained on the study protocol and procedures for contacting nonresponding agencies. Most notably, the AL will receive training on how to ask agencies to complete the web survey, administering the web survey by telephone, arranging for respondents to receive paper copy questionnaires, and tracking cases (including contact attempts). After nonresponse telephone calls, the AL will make targeted attempts with nonresponding agencies to capture critical items.

Three weeks after telephone calls begin, a final email will be sent to the POC which will include an end-of-study message. This message will go to any to-date nonrespondents to announce the forthcoming closure of the study and make a final appeal to participate (Attachment 13).

In order to promote 100% item completion, RTI will monitor item responses on all surveys as they are completed and submitted. RTI will have a management system linked to the web-based data collection system that will be designed to flag non-response items and invalid responses as surveys are completed. RTI will also flag non-responses on hard copy submissions on a rolling basis. The data collection manager will oversee personal telephone or e-mail contacts with individual respondents to clarify missing or invalid responses and to take corrective action. These changes will also be tracked for follow up, if necessary. The RTI project manager will have additional staff members to ensure response follow-up. They will be supplemented on an as-needed basis by one senior RTI researcher and the BJS project manager.

4. Final Testing of Procedures

Pilot Test Methodology

Working with ASCIA, BJS and RTI participated in the ASCIA Use of Force Committee Conference call on February 10, 2016. At this time, BJS provided a summary on the purposes of the survey and the type of information it sought to collect. Once the meeting had commenced, six agencies volunteered to be involved in the pilot testing: Georgia Bureau of Investigation, Louisiana State Police, Michigan State Police, Pennsylvania State Police, Tennessee Bureau of Investigation, and South Carolina Law Enforcement Division. BJS kept the pilot to no more than five agencies since the universe (N=49) is small. BJS chose the five sites that were most diverse in practices and based on ASCIA input who would likely provide active feedback. All state agencies except South Carolina were included in the pilot testing.

The pilot survey was sent on March 11, 2016. The ASCIA members were given two weeks to complete the survey and provide feedback. Eight reviewers from the five ASCIA agencies were given an electronic draft of the survey instrument and asked to comment on question wording, and response categories, as well as overall structure and layout. Respondents were also advised to time themselves as they went through the survey. Responses were primarily received as written annotations within the document. All five pilot sites provided feedback via email on the survey.

Once survey responses were obtained, a 60-minute debriefing call was held on April 1, 2016 with all five pilot sites. During the call, BJS and RTI facilitated discussion on all the instrument items and reviewed recommended changes. Additional feedback was solicited on the introduction, question ordering, mode of data collection and reference periods. The pilot sites unanimously agreed the primary mode should be web survey as they generally use this mode to collect information from ASCIA members. A number of changes to the survey were made based on the feedback received (see Attachment 3) and skip patterns were placed in such a way to ease programming of the web survey. Once the survey was revised, it was sent back out to the Georgia Bureau of Investigation and Pennsylvania State Police for review as they provided the most feedback on the initial survey. No other comments were provided.

5. Contacts for Statistical Aspects and Data Collection

1. BJS contacts include:

Shelley Hyland, PhD

202-616-1706

Shelley.Hyland@usdoj.gov

2. Persons consulted on statistical methodology, data collection and analysis:

Travis Taniguchi, PhD

RTI International

Azot Derecho

RTI International

References

ASCIA (2016). About the ASCIA. Retrieved from: https://www.ascia.org/about.php

Hickman, Matthew & Poore, Jane, “Assessing the Validity and Reliability of National Data on Citizen Complaints about Police Use of Force,” Washington, DC: National Institute of Justice, January 2015 [Draft].

President’s Task Force on 21st Century Policing. (2015). Final report of the President’s Task Force on 21st Century Policing. Washington, DC: Office of Community Oriented Policing Services.

1 The State of Hawaii does not have a state police agency. In past law enforcement surveys, the Hawaii Department of Public Safety (DPS) is typically classified as the state law enforcement agency. However, the scope of the Hawaii DPS is quite limited. We found no evidence that the Hawaii DPS has a state criminal investigative division. ASCIA also confirmed this information.

8