Report of Cognitive Test

Hate Crime Cognitive Testing Labs

June 26-27, 2014

WVU Police Department and FBI CJIS Division

Findings from Cognitive Interviews

The new Hate Crime Incident Report component of the Excel Workbook was pretested with 20 participants. Fifteen of the participants were law enforcement officers, while the remaining five were civilian law enforcement employees. The twenty participants were also comprised of four representatives from a small city police department, two representatives from a large city police department, two participants from a county sheriff’s office, five representatives from a campus police agency, four representatives from the state police, and three representatives from a federal police force. There were two separate versions of the collection instrument to test the presentation of the bias motivation codes in two ways. The twenty participants were split equally between each version to be tested.

The purpose of the interviews was to test cognitive and usability elements of the redesigned collection. The interviews found the following general observations:

• Participants found that having to scroll back and forth to remind themselves of which information is to be reported for particular offenses in multiple offense incidents was frustrating.

• The lengthy instructions at the top of the form created difficulty for the participants to get started.

• Certain portions of the instructions that were replicated from the paper-based collection were non-functional and irrelevant causing confusion for the participants.

• Participants expressed a desire for certain features to be automated such as the identification of which fields were in error or calculating ages.

• Participants were frustrated by inconsistent requirements of “zeroes” versus “blank fields.”

• Participants asked for improved functionality of the tabbing through the form in order to quickly move from one field to another.

• Participants would often not realize that there were lengthy lists in the dropdown menus and would not initially scroll through all the choices. However, with some exploration, they all eventually realized that there were many more options to choose from other than what initially appeared in the list.

The findings related to the testing of two separate formats for the display of bias motivation codes related to race and ethnicity:

• No participants expressed confusions about why anti-Arab bias would be included with the anti-ethnicity bias motivation codes.

• One participant initially looked for anti-Hispanic bias motivation under the anti-race bias motivation codes, but quickly found it under the anti-ethnicity codes.

• While most participants were able to correctly identify the proper use of the anti-multiple races, group bias motivation, they questioned why it was collected in this manner.

• Results from the cognitive testing did not seem to indicate preference for one display over the other.

• In general, participants indicated a need for better explanation about the purpose of the reference guide, which provided the definitions for each of the bias motivations on a separate tab in the Excel workbook.

Finally, when preliminary information was asked about the use of newer bias motivation codes of anti-Sikh, anti-Hindu, and anti-Arab, the following observations can be made:

• If law enforcement personnel work in an environment of limited diversity, they tend to be uncertain about what signs or signifiers would assist them in correctly identifying members of these communities. Most of the incidents were classified as anti-Muslim or anti-Arab if there was any indication of ethnic headwear or symbols.

• When asked how a particular scenario was identified as anti-Arab, often participants mentioned that the victim was speaking Arabic.

• Interestingly, those law enforcement personnel that had recent military experience expressed that military training had familiarized them with information about these communities regardless of how much diversity existed in the locations that they serve.

• Many participants mentioned that training would have helped both in understanding the form and in understanding the definitions of each of the bias motivations.

Revisions to Data Collection in Response to Findings

The following changes will be made to the data collection instrument:

• The technical issues that came to light during the testing will be addressed within the functionality of Excel. Several suggestion made by participants (e.g., automatically advancing from one field to another) are not possible in Excel. These suggestions will be incorporated into future versions that will be HTML-based web forms.

• Any vestiges of the prior paper-based collection that are no longer needed will be eliminated from the form.

• To eliminate much of the scrolling through the form,

  • Lists associated with the reference guide (e.g., the lists of bias motivations) will be display in landscape mode to eliminate having to scroll down to see the entire list, and

  • All the fields that pertain to the offense will be moved closer to the offense field.

• Instructions will be improved on the form to aid the respondent in understanding the fields and their proper use.

• Improved scenarios will be developed for future testing that will have greater realism for law enforcement.

• Training materials will need to address the issue of educating the law enforcement community about ethnic and religious minority groups.

• Given that presenting the bias motivations for race and ethnicity in a combined list would have mitigated one user’s difficulty in finding anti-Hispanic bias, these particular codes will be presented in a combined list (i.e., Race/Ethnicity/Ancestry). However, current methods of collecting the offender(s) race and ethnicity will still be collected separately.