Supporting Statement A - NGI BB 06292022

Supporting Statement for Paperwork Reduction Act Submissions

New Collection

Latent Print Examiner Black Box Study 2022

OMB Control # 1110-xxxx

Part A. Justification

1. Necessity of Information:

This collection is needed as part of a black box¹ research study of latent print examiners (LPEs)² to assess if the accuracy and reproducibility of LPEs’ conclusions have changed in the decade since the 2009-2011 FBI-Noblis Latent Print Black Box Study (hereafter “BB 1”).³

Latent fingerprints are transferred fingerprint impressions left on property and locations. This is in contrast to exemplar fingerprints, which are collected under controlled conditions from known subjects, such as during arrests or background checks. The FBI’s Next Generation Identification system (NGI)⁴ uses automated algorithms to search latent fingerprints against biometric repositories and return potential matching candidates to LPEs using the service for authorized investigative purposes. LPEs must then compare each latent to the candidates returned by NGI and make a comparison conclusion of identification, exclusion, or inconclusive.

The previous BB 1 study focused on LPE accuracy based on candidates returned by the FBI’s Integrated Automated Fingerprint Identification System (IAFIS), which has since been replaced by NGI. The accuracy of automated latent fingerprint algorithms has improved significantly since IAFIS. Searches in NGI are now much more likely to return the correct candidates, and poor-quality latents are also more likely to be successfully searched. In addition, the FBI’s fingerprint repository has significantly increased the number of records that it contains. One side effect of these changes is that, when searches are run using latents from subjects who are not in the NGI repository, LPEs can expect the resulting nonmated candidates returned by NGI to be much more similar to one another than those returned by IAFIS. This raises the potential concern that latent print examiners using NGI are being asked to conduct more challenging comparisons, which in turn may have an effect on the accuracy or reproducibility of their conclusions.

This study will build on BB 1 but will be designed to assess an error rate specific to NGI searches, and the study design will incorporate a variety of lessons learned regarding how to design, conduct, and perform analyses in black box studies.

2. Needs and Uses:

The information collected is part of a research study supporting the forensic science and legal communities. The study is being conducted by the FBI Laboratory with support from a contractor, Noblis (hereafter together referred to as the “Research Team”). There are two phases of this research study that are pertinent to this question:

• Participation Phase — Participants (volunteers, limited to practicing LPEs working as employees or contractors for federal, state, and local U.S. law enforcement agencies within the last 2 years) register for the study and conduct fingerprint comparisons on the study website.

• Analysis Phase — The Research Team extracts data from the website and analyzes the results.

The Participation Phase of the study will be conducted on an unclassified website created by a FBI contractor, Noblis, and hosted on Amazon Web Services (AWS). Noblis will open registration to eligible LPEs, who are required to complete data use agreements and IRB informed consent forms. Prior to starting the study, participants will be asked a series of eligibility questions to ensure that they meet the eligibility requirements (such as being a latent print examiner who has performed casework for one of the eligible agencies with NGI access within the last 2 years). Eligible participants must complete a background questionnaire related to their training, employment, and experience as a latent print examiner. Participants will then be able to login to the website and conduct comparisons of the latent prints. Two-factor authentication will be required for login (i.e., requiring a code texted to a cell phone). Each Participant will be assigned a series of image pairs on the website, and an associated set of multiple-choice questions, such as the comparison conclusion and the difficulty of the comparison. Each image pair contains one latent and one exemplar; the Participants are not informed which latent-exemplar pairs are mated (from the same subjects) or nonmated (from different subjects). At the conclusion of the Participation Phase of the study, the personal information for Participants will be destroyed and the Participants’ responses will be anonymized.

In the Analysis Phase of the study, the Research Team will extract the de-identified Participant responses and analyze the results. This analysis will take place outside of the website. The findings will be published in a peer-reviewed journal and presented at scientific and legal conferences.

3. Use of Technology:

The study collection will be conducted using a website developed for this study, which was developed using Amazon Web Services (AWS). Both Participants and the Research Team will access the study’s website using a standard web browser. Participants will have no access to any information on the study website other than the assigned fingerprint pairs, background questionnaire and study instructions. During the registration process, Participants will be provided a URL to complete data use agreements and informed consent forms. Login will require a system-assigned user ID, a password set by the Participant, and 2-factor authentication (a code texted to a cell phone). After login, Participants will be presented with a series of fingerprint image pairs and asked to provide multiple choice comparison conclusions. Participants will have access to only one fingerprint pair at a time: after submitting a response for a fingerprint pair, those images will no longer be accessible. Participants will have access to the study website for a period of weeks or months; it is anticipated that each Participant will spend 8-12 hours conducting the assigned comparisons. Developers will have a separate authentication process to access the website.

At the conclusion of the Participation Phase, a de-identification procedure will be conducted to remove all identifying information from the Participants’ responses, after which tables of the responses will be downloaded by the Research Team.

4. Efforts to Identify Duplication:

There is no comparable collection that can be used or modified for use for the purposes of this research. This research employs the NGI system, which belongs to the FBI. No other such studies have been requested, nor has other such use been authorized by the FBI – there is no duplication. Because of the nature of this follow-on study and the need for comparison to the first Latent Print Examiner Black Box study, the NGI system must be utilized for generating the comparison prints (which is already completed) and only experienced latent print examiners with legitimate agency access to NGI can participate in the study (due to NGI access privacy restrictions). Therefore, a new collection is required for completing the research goals of this study.

An FBI Privacy Threshold Assessment (PTA) has been completed and authorization has been granted. Please note that the PTA focused primarily on the use of the NGI system for finding highly similar nonmated fingerprints for use in this study; however, all aspects of the study, including Participation Phase and Analysis Phase were included in that review. A copy of the approved PTA is included for reference.

5. Methods to Minimize Burden on Small Businesses:

Participation is limited to LPEs working as employees or contractors for federal, state, and local U.S. law enforcement agencies. The only effect this study could have on small businesses would be for those who provide LPEs under contract to law enforcement agencies. Small law enforcement agencies may choose not to volunteer to be participants in this research study if they deem the burden is too high. There are no negative consequences or costs to agencies with personnel that opt not to participate.

6. Consequences of Less Frequent Collection:

This is a one-time collection. If it is not collected, then the US Government will not have current information about the accuracy and reliability of latent fingerprint comparisons using modern automated fingerprint identification system technology comparable to that used in casework. This information is needed to support the continued acceptance of latent print evidence and expert testimony in court.

The burden has been minimized to the extent possible, without compromising the ability to yield meaningful results. In order to be relevant, the study must be conducted by practicing latent print examiners. Some of the rates that we are attempting to measure are very small (<1%), which requires a large sample for confidence in the measured rates. We have scaled the study with these issues in mind and have balanced the test so that we have enough data while minimizing the time spent on the study by the participants.

7. Special Circumstances Influencing Collection:

There are no special circumstances influencing the collection. However, we are requesting consideration for a rapid turnaround by OMB. The study design and requisite approvals for start-up, use of NGI, PTA, IRB approval, PII handling, secure transfers and review of sensitive data have already significantly impacted the timeline and budget for the research study. The current contractor, Noblis, has received all of the approvals to participate in this research, host the AWS platform for the collection, and have access to the NGI fingerprint images, anonymize the data, and assist with the data analysis; however, their existing contract and budget for this project will expire on Sept. 30, 2022. Absent contract action to extend the contract vehicle (currently awaiting Contracting Officer to determine if an extension is possible), the collection needs OMB approval to proceed NLT August 1, 2022 in order for the participants to complete the study within the contract period and within budget. In addition, if not completed within the contract period and if the contract cannot be extended, we will lose all of our remaining funds for this research (>$300K). Loss of the funds without competing the research collection would be detrimental, as there are no funds budgeted in FY2023 to replace the funds lost.

8. Public Comments and Consultations:

No public comments were received during the 60-day comment period.

9. Payment of Gift to Claimants:

No payment or gift to respondents will be provided by the Research Team. Agencies may authorize remuneration of employees, contractors, or grantees for completion of this study as part of their normal work at the agency’s discretion to include base pay, overtime, compensatory time, or other incentives.

10. Assurance of Confidentiality:

This is what we tell participants in the Instructions about confidentiality:

Results will be anonymous. Efforts will be made to keep your information in the study records confidential. The research results may be published, but anonymity of both participants and attribution of results to participants will be maintained. Personally Identifiable Information (PII) will be used only for the purpose of conducting the study and will not be used or released for other purposes. Your study results will not be linked to your PII. No reference will be made in oral or written reports, publications or in the databases in which results may be stored that could link your name to the study. A blind coding system will ensure anonymity. The subject ID numbers associated with your name, email address, and affiliation will be anonymized so that the analysis team will not be able to associate your conclusions or the responses to the questionnaire with any personal information. Cross-references between the subject IDs and individual results will be destroyed prior to the publication or public presentation of any results. Therefore, the identities of participants will not be associated with the results at any point during analysis, and such association will not be possible subsequently, such as for discovery.

The researchers will not disclose which individuals did or did not take the test. In reporting, results will be aggregated across multiple examiners, based on categories of experience established in the background questionnaire. Care will be taken so that the results are not aggregated in a way that compromises anonymity.

The study, including its methods for assuring and maintaining confidentiality, has been reviewed and approved by the FBI’s Institutional Review Board for Human Subject Research (IRB #618-21).

Additionally, this Privacy Act statement will be included:

Privacy Act Statement:

Pursuant to the Privacy Act of 1974, 5 U.S.C. 552a, we are providing the following information regarding this collection of information. The authority under which this information is being collected is 28 CFR §§ 0.85, 20.31. The principal purpose for which the information will be used is to evaluate the abilities of latent fingerprint examiners to reach reliable and/or accurate conclusions when comparing friction ridge images. Routine Uses: This information may only be disclosed as permitted by the Privacy Act of 1974, 5 U.S.C. 552a(b), and all applicable routine uses established in Blanket Routine Uses, 66 Fed. Reg. 33558 (June 22, 2001), as amended, as well as the following FBI System of Records Notice: FBI-009, Next Generation Identification, 84 Fed. Reg. 54, 182 (October 9, 2019).

If you choose not to provide the mandatory information and decline to participate in the study, there will be no negative consequences.

11. Justification for Sensitive Questions:

There are no questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private.

12. Estimate of Hour Burden:

This research study has been scaled for an estimated 250 respondents; however, it is open to all volunteers who meet eligibility requirements.

This is a one-time collection, so the frequency of response is one, and the total number of annual responses is calculated as the number of respondents time one = 250.

An estimated 720 minutes per respondent (to complete background questionnaire, fingerprint comparisons and post-comparison survey questions) was calculated based on historical data from prior research conducted by this Research Team and casework by latent print examiners.

Therefore, the annual hour burden to the public is estimated to be 3000 hrs (250 participants x 720 min).

13. Estimate of Cost Burden:

There is no additional cost burden to respondents resulting from the collection of information beyond what is included in Items 12 and 14.

14. Estimated Annualized Costs to Federal Government:

Total annualized capital/start-up costs are estimated at $200,000. This cost is estimated from the cost of contract and USG labor in designing the study, acquiring the requisite approvals, issuing the contract task, searching NGI, and selecting the fingerprint images, assembling the study materials and reviewing, characterizing, and verifying the selected data. Because this work is being done as a Research Team, the contractor and USG employee labor is included together here rather than separately, as both are costs to the Federal Government. There was no new capital equipment acquired for this collection.

The total annual costs for O&M are estimated at $25,000 and include maintaining the website, responding to participant questions, and reporting to IRB. Cost of maintaining IT systems are part of the overhead on the contract, which is included in the hourly rates of the staff (thereby already included in the overall estimate for O&M).

This is a one-time collection that will be completed within one year. However, the data generated are expected to be used over a period of approximately 3-5 years.

15. Reasons for Change in Burden:

This is a new ICR request with no previous burden.

16. Plans for Publication:

Complex analytical techniques to be used:

Various statistical analyses will be performed using commercial software and freeware such JMP and R. Various regression/machine learning techniques will also be implemented including but not limited to linear regression, logistic regression, Variable Importance Analysis (VIA), Kruskal-Wallis Analyses, Random Forest, and Benjamini-Hochberg.

Schedule for entire project:

Task/Milestone	Original Planned Completion Date	Current Planned Completion Date	Actual Completion Date
IRB, IT Security, OGC, and CJIS approvals	Prior to test start	--	9/16/2021 (all)
Develop Test Plan	10/25/2021	--	8/30/2021
Recruit participants	2/14/2022	--	2/7/2022 (Started)
Select/collect fingerprint data	1/3/2022	--	1/06/2022
Software development	2/14/2022	6/1/2022	In progress
Alpha test (internal testers)	2/14/2022	6/1/2022
Beta test (external LPEs)	3/14/2022	7/1/2022	conduct only with federal labs
Conduct test (LPEs)	7/4/2022	Start: 8/1/2022 End: 11/1/2022	awaiting OMB PRA
Analysis of results	12/12/2022	Initial analyses complete: 2/1/2023
Preparing presentations	1/30/2023	4/1/2023
Preparing manuscript/submit to journal	7/3/2023	9/1/2023
Publication process	1/1/2024	3/1/2024

17. Expiration Date Approval:

We are not seeking approval to not display the expiration date for OMB approval of the information collection.

18. Exceptions to the Certification Statement:

There are no exceptions to the certification statement.

1 A black box study measures the reliability of methods that rely mainly on human judgment, as opposed to those that rely on laboratory instruments.

2 Participation will be open to practicing LPEs from federal, state, and local United States (U.S.) law enforcement agencies.

3 Ulery BT, Hicklin RA, Buscaglia J, Roberts MA (2011). Accuracy and reliability of forensic latent fingerprint decisions. Proceedings of the National Academy of Sciences of the United States of America. 108. 7733-8. DOI: 10.1073/pnas.1018707108 (https://www.pnas.org/content/108/19/7733).

4 NGI is the FBI’s biometric and criminal history records system and is managed by the FBI’s Criminal Justice Information Services (CJIS) Division. It is addressed in separate privacy documentation.

7