Measuring Accuracy Facial Forensics Comparisons

OMB Control No. 0693-0043 – NIST Generic Clearance for Usability Data Collections

Measuring the accuracy of facial forensics comparisons

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

In the paper Perceptual expertise in forensic facial image comparison (uploaded into ROCIS) by White, Phillips, Hahn, Hill, and O’Toole, the perceptual accuracy of facial forensic experts was measured, and examiners were shown to be more accurate than the general public. However, the study did not explain why. A follow-up study will be conducted to measure the accuracy of facial forensic examiners. The study will have two parts. The first part of the collection will measure the accuracy of forensic examiners using their tools and laboratory methods. The second part of the collection will survey participants on their background, experience, and education. In addition to facial examiners, there will be three control groups: face super-recognizers, fingerprint examiners, and non-examiner face experts. Participants for the three control groups will be selected because they have backgrounds and experiences similar to facial examiners.

As shown in the paper Face Recognition by Metropolitan Police Super-Recognisers (uploaded into ROCIS) by Robertson, Noyes, Dowsett, Jenkins, and Burton, face super-recognizers are better at the face recognition task than students or police trainees.

There are three collection instruments for this survey. The first instrument is the Screening Questionnaire that will be completed by all that chose to participate, which is estimated to be 225. The second collection instrument is the Background Questionnaire. There are four different background questionnaires, and based on answers to the screening questionnaire, each member of the four control groups will be asked to complete the appropriate Background Questionnaire. The third collection instrument is the Image-Pair Comparison Questionnaire. Additional details about the instruments are given in the answers to Questions 3 and 4. All participants will complete the Screening Questionnaire verbally, responses will be documented or recorded by the interviewer and will be used to determine if a volunteer is eligible to participant in the study. Responses received from the Background Questionnaire will be used to determine which factors contribute to greater accuracy in facial comparison. Responses for the Background Questionnaire will be saved by the web-based survey.

All eligible participants in the study will be asked to compare 20-30 pairs of face-images. After comparing each pair of face-images, participants will be asked to answer two questions that are contained in the Comparison Questionnaire. The responses to the Comparison Questionnaire will be used to measure the accuracy of the participants in performing facial comparisons.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

The Background Questionnaire(s) to be administered are based on previous studies as well as consulting with experts in facial forensics and experts in survey design at NIST. These questionnaires are based on a survey used to assess the backgrounds of forensic fingerprint examiners described in the paper Accuracy and reliability of forensic latent fingerprint decisions (uploaded into ROCIS) by Ulery, Hicklin, Buscaglia, and Roberts. The Background Questionnaires were designed to help answer questions that arose from the results in the paper Perceptual expertise in forensic facial image comparison by White, Phillips, Hahn, Hill, and O’Toole.

The Comparison Questionnaire is based on the paper by White, Phillips, Hahn, Hill, and O’Toole, and consulting with experts in facial forensics.

The Screening Questionnaire was based on consultations with experts in forensics and the previous study described in the paper, Accuracy and reliability of forensic latent fingerprint decisions by Ulery, Hicklin, Buscaglia, and Roberts.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

Participants will come from the facial forensic, face recognition, and fingerprint communities. Emails asking for potential participants (recruitment announcement) will go to professional groups such as the Facial Identification Scientific Working Group (FISW1G), the Facial Identification subcommittee of the Organization of Scientific Area Committees (OSAC), the European Network of Forensic Science Institutes (ENSFI), and the Scientific Working Group on Friction Ridge Analysis, Study and Technology (SWGFAST).

The questionnaires are part of a study to measure the accuracy of four groups that will be performing facial forensic comparisons. After reviewing the study and consent form (uploaded into ROCIS) , volunteers who choose to participate will sign the consent form and proceed to the screening questionnaire. Participants will be asked (verbally) the 13 screening questions over the phone. fFfour of these questions will collect the respondent’s contact information in order to send materials (such as the background questionnaire and images) and to be sure appropriate precautions are in place when sending or receiving information via fax. As stated in the Screening Questionnaire, a postal address is needed to send a participant’s accuracy score to them if requested. However, the participant may choose to defer providing this information until they decide if they wish to request their score. The answers to the Screening Questions will be used for contact information and to determine if the volunteer fits into one of the four groups.

After it is determined if the participant fits into one of the four groups, the participant will be given a link to the group-appropriate Background Questionnaire, which will be administered online where the answers will be saved. An individualized link to the appropriate Background Questionnaire will be emailed to the participant. Every participant will complete the same background survey as every other participant in their group, but the individualized URL will allow linking of responses to participant. Participants will be given a link to a web-based survey for completing the Comparison Questionnaire. For the Comparison Questionnaire, an individualized URL will allow linking of responses to participants.

We estimate that we will receive up to 225 responses from the four groups described above. We estimate we will receive 120 responses from facial forensic examiners, 20 responses from face super-recognizers, 75 responses from fingerprint examiners, and 10 responses from non-examiner face experts.

Facial forensics examiners are people who perform forensic facial comparisons as part of their jobs and have been trained to perform facial forensic comparisons.

Face super-recognizers are people who have been acknowledged as one by at least one of two ways: has taken a test establishing them as one or was recruited to work using their skills as a face super-recognizer.

Fingerprint examiners are people who perform forensic fingerprint comparisons as part of their jobs and have been trained to perform fingerprint comparisons. These examiners must not perform or have performed facial forensic comparisons as part of their jobs. They must also have been trained to perform facial forensic comparisons.

Non-examiner face experts are people who are knowledgeable about face recognition. These people neither are nor have been forensic examiners. They also have not been trained in how to perform forensic comparisons. However, they have a combination of experience conducting research in face recognition, managing face recognition programs, or addressing policy issues related to face recognition.

All respondents participating, estimated to be 225, are expected to complete the Screening Questionnaire as they have agreed to take part in the study. We expect all participating respondents to complete the appropriate Background Questionnaire.

PII will be collected for administering the study.  The respondents’ personally identifiable information (PII) will be protected in the following manner.  Each subject enrolled in this study will be assigned a study number and this number will identify all data, including results of facial comparisons.  A master list linking the PII to the study number will be kept on an encrypted disk located in a locked office or a locked cabinet at NIST.  Responses to the screening and background questionnaires will be treated as PII and will not be shared outside of NIST.  Personal privacy information would be withheld from release to the public under the Freedom of Information Act, 5 U.S.C. 552, exemption (b)(6):  personnel and medical files and similar files the disclosure of which would constitute a clearly unwarranted invasion of personal privacy.

 

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

This study will address concerns raised in the National Academy of Sciences (NAS) forensic study Strengthening Forensic Science is the United States: A Path Forward. The NAS study called for more scientific rigor and the need to establish an underlying forensics science, particularly in forensic pattern analysis such as comparing two faces. Facial forensic examiners are trained to compare faces that appear in images and videos to determine if the two faces are the same person or not, providing justification for their conclusions. They are prepared to testify in court and to defend their conclusions.

The first aim of the study is to measure the accuracy of the four groups at comparing faces. The answers from the comparison questions will allow us to estimate the accuracy of the four groups.

Another aim of this study is to understand which factors in an examiner’s background contribute to greater accuracy. Answers to the Background Questionnaires will be used to characterize the backgrounds of facial forensic examiners, face super-recognizers, fingerprint examiners, and non-examiner face experts. Analyzing the responses from all four groups will assist in identifying factors that contribute to the accuracy of facial examiners.

The resulting understanding of the sources of facial examiner accuracy will help in establishing a forensic science. In turn, this will help in meeting the Daubert standard for facial examinations in U.S. Courts. The Daubert standard is used to determine if expert evidence is admissible.

Using methods from signal detection theory, analysis of variance, and correlation theory, the backgrounds of these three groups will be compared with results from the accompanying experiment measuring face recognition accuracy. This analysis will be the first to study the background of facial examiners and their accuracy.

National Research Council. (2009). Strengthening Forensic Science in the United States: A Path Forward. Washington, DC: The National Academies Press.

Ulery, B., Hicklin, R., Buscaglia, J., & Roberts, M. (2011). Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences of the United States of America, 108(19), 7733-7738. doi:10.1073/pnas.1018707108

White, D. Phillips, P.J., Hahn, C.A., Hill, M., & O’Toole A.J. (2015). Perceptual expertise in forensic facial image comparison. Proceedings of the Royal Society of London B, 282(1814). doi:10.1098/rspb.2015.1292

Robertson, D.J., Noyes, E., Dowsett, A.J., Jenkins, R., & Burton, A.M. (2016). Face Recognition by Metropolitan Police Super-Recognisers. PLoS ONE, 11(2). doi:10.1371/journal.pone.0150036