NIST, Information Technology Laboratory, MEMEX Data Collection

OMB Control No. – 0693-0043 – NIST Generic Clearance for Usability Data Collections

Professional User Feedback Survey of Memex Search Systems

FOUR STANDARD SURVEY QUESTIONS

1. Explain who will be surveyed and why the group is appropriate to survey.

Professionals in the domain of counter-human trafficking, primarily from law enforcement and nongovernmental agencies. This is the target group of users for the Memex search systems, who would use these systems to conduct investigations of leads by searching data from the internet that can indicate human trafficking activity. These search systems are specially designed to identify, collect, and extract the most important information that law enforcement investigators use when pursuing leads in this domain, and so the systems should help investigators perform their tasks more effectively and efficiently.

2. Explain how the survey was developed including consultation with interested parties, pre-testing, and responses to suggestions for improvement.

This project is being conducted as part of the DARPA Memex program, which seeks to develop advanced search engine capabilities for people in specialized domains. This survey was developed by a focus team of usability experts, search system experts, and individuals with experience both in the counter-human trafficking domain and with the Memex search systems. We then ran a pilot with counter-human trafficking specialists that did not know the search systems. The pilot allowed us to refine the survey.

3. Explain how the survey will be conducted, how customers will be sampled if fewer than all customers will be surveyed, expected response rate, and actions your agency plans to take to improve the response rate.

The survey will be completed by volunteers with domain expertise (experience in counter-human trafficking) who are being recruited by DARPA. These volunteers are an “convenience sample” obtained from parter organizations with whom DARPA hopes to transition the software prototypes. We expect all volunteers to complete the survey. To ensure this, we will be facilitating the survey collection process by working in person with the volunteers to complete the surveys.

4. Describe how the results of the survey will be analyzed and used to generalize the results to the entire customer population.

The demographics will be summarized using averages and medians as appropriate. The answers on scales will be analyzed against search task type and search system. The textual responses will be anonymized and reported as anecdotal experiences with the search systems.

We do not expect these results to generalize to the entire customer population because resources do not support extending the survey over a target population sufficient to balance all the factors we expect to influence the outcome. Nevertheless we hope that the results will be informative for DARPA and for the designers of the search systems.

1