1110-0057 Supporting Statement Part A

SUPPORTING STATEMENT A

1110-0057

Instrument Pretesting and Burden Estimation – Generic Clearance

The Federal Bureau of Investigation (FBI) Uniform Crime Reporting (UCR) Program is requesting a 3-year extension of this generic clearance for the FBI to develop, test, and improve its data collections, survey instruments, and methodologies. The procedures include, but are not limited to, potential tests of data collection and survey operations, focus groups, cognitive laboratory activities, pilot testing, exploratory interviews, experiments with questionnaire design, and usability testing of electronic data collection instruments.

The FBI UCR Program is requesting the renewal of this generic clearance in order to pretest and determine burden assessments for the UCR collections and initiatives to include the Summary Reporting System (SRS), the National Incident-Based Reporting System (NIBRS), the Crime Data Explorer (CDE), the Law Enforcement Officers Killed and Assaulted (LEOKA), the Hate Crime Data Collection, and the National Use-of-Force Data Collection. Further, this generic clearance request includes any pretesting conducted for new data collections initiated by the Criminal Justice Information Services (CJIS) Advisory Policy Board (APB) or recommendations provided by the joint National Academy of Sciences study of Bureau of Justice Statistics (BJS) and the FBI UCR Program’s data collections.

The FBI UCR Program has added more statistical rigor and pretesting functions to its data collections to improve Paperwork Reduction Act (PRA) approval requests. The generic testing clearance allows the FBI to take advantage of a variety of methods that are useful for identifying questionnaire/assessment and procedural problems, suggesting solutions, and measuring the relative effectiveness of alternative solutions. Through the use of these techniques, questionnaires and assessments can be simplified for respondents, respondent burden can be reduced, procedures for the collection of administrative data can be streamlined, and the quality of the questionnaires and assessments used in continuing, and one-time surveys and assessments can be improved. This results in an increase in the quality of the data collected when employed routinely in the testing phase of the UCR data collections.

This clearance package serves as a request for generic clearance. This document contains a description of the scope of possible activities that might be covered under this clearance. The requested clearance is important to the FBI UCR Program’s use of pretesting activities, because of the length of time required to plan the activities. This generic clearance will go through two Federal Register periods. Following these review periods, the FBI UCR Program requests the Office of Management and Budget (OMB) review, comment on, or clear the proposed studies in a two-week period with no additional Federal Register notice period required under the generic clearance. This clearance is similar to the testing clearances held by the Census Bureau, the Bureau of Labor Statistics, the BJS, the National Center for Education Statistics, and the National Center for Science and Engineering Statistics.

The specific methods proposed for coverage by this clearance are described below. The procedures are outlined according to the FBI UCR Program’s plans for keeping OMB informed about the identity of the surveys and the nature of the research activities being conducted.

The methods proposed for use in the questionnaire and assessment development are as follows:

Pilot testing. For the purposes of this clearance, pilot tests are defined as data collection efforts conducted among either purposive or statistically representative samples. The main objectives for these pilot studies are the evaluation of questionnaires, procedures, and testing the feasibility of new data collection modes, such as evaluating web forms on the internet. The FBI UCR Program will only publish research and development and methodological reports on the results of these pilot tests, but will not publish statistical reports or data sets, based on the findings. Pilot tests are an essential component of this clearance package because they serve as the vehicle for investigating basic item properties for new or redesigned data collection efforts, such as reliability, validity, and difficulty, as well as feasibility of methods for standardized administration of forms. Under this clearance, a variety of surveys will be pretested, and the exact nature of the surveys and the samples are undetermined at present. However, due to the smaller nature of the tests, it is expected some will not involve representative samples. In these cases, samples will be treated as convenience samples. The needs of the particular sample will vary based on the content of the survey being tested, but the selection of sample cases will not be completely arbitrary in any instance. Where applicable, pilot testing will include sample administrative data for determining burden estimates, allowing the FBI UCR Program to test procedures regarding data procurement and comparability of data. Such burden estimates will include only a few questions asking how much it costs for agencies to make changes to their software programs and records management systems (RMS). For example, a question may ask for costs and labor burdens experienced by state and local law enforcement agencies when the FBI UCR Program changes data specifications by adding new offense categories.

Behavior coding. This method involves applying a standardized coding scheme to the completion of an interview or questionnaire, either by a coder using a tape-recording of the interview or by an in-person observer at the time of the interview. The coding scheme is designed to identify situations that occur during the interview which reflect problems with the questionnaire. For example, if respondents frequently skip mandatory data fields before the question is completed, the questionnaire may be too long. If respondents frequently give inadequate answers, this suggests there are some other problems with the question. Quantitative data derived from this type of standardized coding scheme can provide valuable information to identify problem areas in a questionnaire and can be used as a substitute for or as a complement to the traditional interviewer debriefing.

Exploratory interviews. These may be conducted with individuals to understand a topical area and may be used in the very early stages of developing a new survey. It may cover discussions related to administrative records (e.g. what types of records, where, and in what format), subject matter, definitions, etc. Exploratory interviews may also be used to investigate whether sufficient issues are present related to an existing data collection to consider a redesign. During the exploration phase, law enforcement agencies and RMS capabilities can be evaluated along with establishing the perceived need and value of the data collection among the law enforcement community.

Respondent debriefing. The debriefing form is administered at the end of the questionnaire being tested, and contains probing questions to determine how respondents interpret the questions and whether they have problems in completing the survey/questionnaire. This structured approach to debriefing enables quantitative analysis of data from a representative sample of respondents, to learn whether respondents can answer the questions and whether they interpret them in the manner intended by the questionnaire designers.

Follow-up interviews or re-interviews. This involves reinterviewing or reassessing a sample of respondents after the completion of a survey or assessment. Responses given in the re-interview are compared with the respondents’ initial responses for consistency between responses. In this way, re-interviews provide data for studies of test–retest reliability and other measures of data quality. In turn, this information aids in the development of improved, more reliable measures.

Cognitive interviews, usability testing, and other field techniques. This method involves intensive, one-on-one interviews in which the respondent is typically asked to "think aloud" as he or she answers survey questions. A number of different techniques may be involved, including asking respondents to paraphrase questions, probing questions asked to determine how respondents came up with their answers, and so on. The objective of cognitive interviewing is to identify problems of ambiguity or misunderstanding, identify potential improvements on form appearance, flow, and instructions, or highlight other difficulties respondents have answering questions. This is frequently one of the early stages of revising a questionnaire. Usability testing is more typically associated with assessing physical features of a data collection, such as navigation.

Focus groups. This qualitative method involves group sessions guided by a moderator, who follow a topical outline containing questions or topics focused on a particular issue, rather than adhering to a standardized questionnaire. Focus groups are useful for surfacing and exploring issues (e.g., confidentiality concerns) which people may feel some hesitation about discussing.

1. Justification

1. Necessity of Information Collection

Under the authority of:

• Uniform Federal Crime Reporting Act of 1988, Title 34, United States Code (U.S.C), § 41303

• Acquisition, Preservation, and Exchange of Identification Records; Appointment of Officials, 28 U.S.C., § 534(a) and (c), (1930)

• Anti-Arson Act of 1982, 18 U.S.C § 841

• 34 U.S.C. § 41305, Hate Crime Statistics Act of 1990

• Violent Crime Control and Law Enforcement Act of 1993 Conference Report on H.R. 3355, Section 15006 Gang Coordination and Information (b) Data Collection

• USA Patriot Improvement and Reauthorization Act of 2005, Public Law 109-177 (March 9, 2006) H.R. 3199: Section 307 (e)

• William Wilberforce Trafficking Victims Protection Reauthorization Act of 2008, Public Law 110-457, Title II, Section 237 (a).

The FBI was designated by the U.S. Attorney General to acquire, collect, classify, and preserve national data on law enforcement data as part of the Uniform Crime Reports.

During the three-year generic clearance term, the FBI UCR Program will provide periodic reports on pretesting activities. The pretests conducted under this generic clearance will be for development work only, not for estimation or publication purposes. These development activities will include investigations of item types, research on the availability and quality of administrative data from state and local justice agencies, state and local burden assessments, small scale tests to test appropriate access and retrieval methods for various types of administrative data, research about mode of administration, methodology of questionnaires and assessments, and testing of items.

Prior to each test, the FBI will provide OMB with a memo describing the study to be conducted and a copy of instrumentation and debriefing materials that will be used. Depending on the stage of instrumentation development, this may be a printed questionnaire, a set of prototype items showing each item type to be used and the range of topics to be covered by the questionnaire, or an interview script. The FBI UCR Program requests that OMB raise comments on substantive issues within 10 working days of receipt.

2. Needs and Uses

The information collected as a result of developing and testing questionnaires and other data collection protocols will be used by staff from the FBI UCR Program to evaluate and improve the quality of the data in the surveys and assessments that are ultimately conducted. None of the data collected under this clearance will be published for reporting on the nature of crime in the United States.

Because the questionnaires being tested under this clearance are still in the process of development, the data that result from these collections are not considered official statistics. Data will not be made public except when results are prepared for presentations related to survey methodology at professional meetings or publications in professional journals.

Information quality is essential to the pre‑dissemination review of the information disseminated by the FBI UCR Program. Information quality is also integral to the information collections conducted by the FBI UCR Program and is incorporated into the clearance process required by the PRA. Specifically, the FBI UCR Program will use this generic clearance to develop a set of valid and reliable statistical sampling methods to develop national estimates and agency imputation data models.

The FBI UCR Program has undertaken an effort to transition to a NIBRS-only data collection by January 1, 2021. With this transition, the FBI UCR Program will retire the SRS collections. Therefore, under this generic clearance the following data collections/subject areas may undergo testing:

1. NIBRS: The FBI UCR Program NIBRS data collection changes often require software programming changes to agencies’ records management systems. NIBRS does not have a paper collection form. As such changes to NIBRS often require burden assessments not from front line data entry points (law enforcement officer incident reports) but rather from either law enforcement information technology employees or contracted vendors. Testing for NIBRS purposes focus on exploratory interviews and cognitive testing to ensure changes or additions to the NIBRS data collection are properly understood and do not introduce error through unclear documentation or instruction and to assess burden costs. Further, NIBRS testing will help evaluate new technological approaches for reducing burden, such as the capability of operating on a machine-to-machine platform by utilizing an Extensible Markup Language (XML) submission platform. The XML will allow the individual agencies that report FBI UCR Program data to submit information directly to and receive data directly from CJIS data collection servers, which will increase the efficiency of the data and decrease the time it takes to accurately process and store the data.

2. LEOKA: The LEOKA data collection has three unique data collections intended to understand the number and characteristics of violence perpetrated against the nation’s law enforcement officers. LEOKA intends to perform a review of the LEOKA long forms to evaluate the data being collected and the long-term goal of the collection.

In June 2020, the President signed into law the Law Enforcement Suicide Data Collection Act. This law mandates the U.S. Attorney General, acting through the FBI Director, to establish a data collection to collect, store, and disseminate data related to suicides within the law enforcement community. The FBI UCR Program has been tasked with developing this new collection. The FBI UCR Program anticipates additional testing and interviews may be needed.

LEOKA will also be collecting new health-related data elements. These data elements will include health-related deaths of law enforcement officers where the death can be attributed to factors related to the officer’s line of duty. Examples of such instances are the September 11, 2001, attacks and COVID-19. LEOKA anticipates additional testing and possible interviews.

3. CDE: The CDE publishes nationwide crime data collected by the FBI UCR Program in a digital format. The tool allows users to view trends and download bulk data in order to better understand crime across the country. As more requirements for building the application are established, additional testing or surveys will be required.

4. Number of Full-Time Law Enforcement Employees: The FBI UCR Program anticipates adding the ability to capture part-time and reserve/auxiliary/other law enforcement officers; race and ethnicity categories; and information on recorded police contacts with citizens. The FBI UCR Program anticipates extensive cognitive testing, data collection instrument testing, and other pretests to verify validity and reliability of data submissions will be needed over the three years covered by this generic clearance. Prior to a pretest, the FBI UCR Program will notify the OMB of the scope of the study and anticipated burden law enforcement agencies will experience.

5. Lawful Access: The FBI UCR Program will be evaluating the possibility of collecting data related to the lawful access of encrypted information on cell phone and other devices by law enforcement, when such devices are involved in a criminal incident. This collection is designed to track the frequency of incidents where encrypted information concerning a criminal incident is requested. The FBI UCR Program anticipates extensive cognitive testing, data collection instrument testing, and other pretests to verify validity and reliability of data submissions that will be needed over the three years covered by this generic clearance. Prior to a pretest, the FBI UCR Program will notify the OMB of the scope of the study and anticipated burden law enforcement agencies will experience.

6. Estimation: The FBI will continue to research new imputation techniques, as well as implement nonresponse bias studies, that will allow for the FBI UCR Program to publish national estimates for its data collections.

7. Beyond 2021: The FBI UCR Program will be pursuing several changes to its collections as part of an initiative to evaluate the future of the FBI UCR Program. Some of these initiatives will include the expansion of collection of firearm data, expansion of victim injury information, creation of a NIBRS Offense Repository, expansion of the NIBRS drug information, expansion of the NIBRS assault offenses, addition of a "Reckless Endangerment" category, addition of a "corruption" category within NIBRS, addition of "wildlife/Hunting violations" to NIBRS, addition of "Violation of Tribal Law" to NIBRS, addition of the collection of geolocation codes within NIBRS, and expansion/modification of sex offense definitions. The FBI UCR Program anticipates additional evaluation and changes over the duration of this
   three-year clearance and will increase the burden estimation accordingly.

3. Use of Information Technology

When the survey or assessment being pretested employs automated methods for its data collection, the research conducted under this submission will also use automated data collection techniques. This clearance offers the FBI UCR Program an opportunity to try innovative technologies that can reduce burden, improve data quality and reliability, and increase the use of information technology.

4. Efforts to Identify Duplication

This research does not duplicate any other questionnaire design work being done by the FBI or other federal agencies. The purpose of this clearance is to stimulate additional research and collaboration with other federal agencies, which would not be done under other circumstances due to time constraints or without shared resources. The FBI and the BJS are continuing their partnership through the National Crime Statistics Exchange (NCS-X) effort to increase participation in the NIBRS data collection by at least 400 agencies, to include the 72 largest agencies to ensure data coverage to generate statistically-sound national estimates of crime known to law enforcement agencies (LEAs). The NCS-X is providing assistance to select state, local, and tribal agencies as well as to select state UCR programs to enable them to report the NIBRS data to the FBI. The CJIS APB also recommended, and the FBI Director approved, an effort for the FBI UCR Program to transition to a NIBRS-only data collection by January 1, 2021. All efforts would be collaborative in nature, and no duplication in this area is anticipated.

To the maximum extent possible, the FBI UCR Program will make use of previous information, reviewing results of previous evaluations of survey data before we attempt to revise questionnaires. However, this information is not sufficient to refine our survey questionnaires and assessments without conducting additional research.

5. Minimizing Burden

This research will be designed as relatively small-scale data collection efforts. This will minimize the amount of burden required to improve questionnaires, data collection instruments, and procedures, to test new ideas, and refine or improve upon positive or unclear results from other tests. The results of the research conducted under this clearance are expected to improve the methods and instruments utilized in full scale studies and thereby improve information quality, while minimizing burden to respondents.

6. Consequences of Less Frequent Collection

This clearance involves one-time questionnaire and data collection research and development activities for each survey or data collection connected with the clearance. Without adequate testing, data quality will be affected. In addition, activities covered under this clearance will allow for more specific and precise calculation of burden hours, costs associated with data collection, and survey efforts conducted by the FBI UCR Program.

7. Special Circumstances

All the guidelines listed in the OMB guidelines are met. There are no special circumstances.

8. Public Comments and Consultations

The 60-day and 30-day Federal Register notices have been submitted and published in the Federal Register with no public comments received.

9. Provision of Payments or Gifts to Respondents

No currently proposed projects involve the use of incentives. Incentives may be offered to cover travel and incidental expenses for persons participating in focus groups or cognitive testing of questions.

10. Assurance of Confidentiality

The FBI UCR Program typically does not offer any confidentiality assurances, as most collections will be from LEA staff acting in their official capacity and will be information largely in the public domain. However, when warranted, the FBI UCR Program may offer some confidentiality protections and will indicate the FBI UCR Program will protect the privacy of information provided to the extent provided by law and will take procedural safeguards such as limiting access to those on the project team, destroying or encrypting identifiers once the project is complete and other physical security measures. Further, when working with other federal agencies, such as BJS, the FBI UCR Program will work with its partners to ensure confidentiality and privacy protections are agreed upon and enforced equally by all parties.

11. Justification for Sensitive Questions

It is possible some potentially sensitive questions may be included in questionnaires that are tested under this clearance. One of the purposes of the testing is to identify such questions, determine sources of sensitivity, and address concerns related to those questions, insofar as possible, before the actual survey is administered. Justification for any sensitive questions included in a project, covered by this generic clearance, will be included in the individual project submissions.

12. Estimate of Respondent’s Burden

The FBI UCR Program estimates the hour burden for exploratory, pilot, cognitive, and focus group work will average about 500 burden hours per year. Primarily, pilot tests, cognitive labs, and focus groups are anticipated to need 30 respondents for each of the seven programs that may require pretesting within the FBI UCR program (see section 2, Needs and Uses). Three hundred respondents for focus group and cognitive testing are anticipated annually over the next three years. Secondarily, we anticipate conducting UCR change burden surveys, which ask how much it will cost law enforcement agencies to implement hardware and software upgrades to report UCR data in the most current formats. These assessments will be sent to a sample population of approximately 500 LEAs reporting UCR data, and these burden estimates will only take 15 minutes to complete and are conducted via emails and other electronic communications means, primarily through www.cjis.gov. In many cases, the FBI UCR Program office will only contact state UCR program offices and direct contributors to conduct change burden studies; however, many of the UCR State program offices will forward the survey to their contributing agencies. As the FBI UCR Program office is in essence affecting the UCR State program offices to collect burden data from contributing agencies, we feel it necessary to include their burden in our estimates according to the guidelines of the PRA.

Table 1: Burden Estimate

	Testing			Burden Assessments
Year	Participants	Hours/Participant	Hours	Participants	Hours/Participant	Hours	Total Hours
2020	400	1	400	500	0.25	125	525
2021	400	1	400	500	0.25	125	525
2022	400	1	400	500	0.25	125	525
3-Year	1,200	3	1,200	1,500	0.75	375	1,575

By combining 500 respondents for hour-long pretesting sessions (500 hours) and 500 fifteen-minute burden assessment interviews (125 hours), the estimated annual burden hours across all proposed project activities will be 525 hours. The total estimated respondent burden is approximately 1,575 hours for the period from August 2020 through July 2023.

A variety of forms will be used in conducting the research under this clearance, and the exact number of different forms, length of each form, and number of subjects/respondents per form are not thoroughly known at this time. However, it is projected these activities will include testing items and data collection modes, and conducting pilot tests, cognitive labs or interviews, exploratory interviews, re-interviews, behavior coding, and focus groups, among other developmental research approaches.

13. Estimate of Cost Burden

There is typically no cost to respondents for participating in the research being conducted under this clearance, except for their time to complete the questionnaire or participate in an interview or focus group.

14. Cost to Federal Government

It is very difficult to anticipate the actual number of participants, length of interview, and/or mode of data collection for the work to be conducted under this clearance over the entire 3-year clearance period. Without that information, it is not possible to estimate in advance the cost of the work under this generic clearance to the federal government. Costs associated with each individual project will be covered by the statistical unit conducting the research and will come from their data collection budgets. Information about costs in the individual submissions, to include staff time and travel expenses, will be provided to OMB prior to pretesting.

15. Reason for Change in Burden

There will be an increase in burden as the number of respondents needed for UCR projects increased from 300 to 400 for testing and from 400 to 500 for burden assessments. This adjustment, from 1,200 hours to 1,575 hours is an increase of 375 hours.

16. Project Schedule and Analysis Plans

This research program is for questionnaire and procedure development purposes. Data tabulations will be used to evaluate the results of questionnaire testing. The information collected in this effort will not be the subject of estimates or other statistics in the FBI UCR Program reports, however, it may be published in research and development reports or be included as a methodological appendix or footnote in a report containing data from a larger data collection effort. The results of this research may be prepared for presentation at professional meetings or publication in professional journals. Due to the nature of this clearance, there is no definite or tentative time schedule at this point. It is expected that work will be conducted continuously throughout the duration of the clearance.

17. Request to Not Display Expiration Date

No exemption is requested.

18. Exceptions to the Certification

The FBI CJIS Division does not request an exception to the certification.

1