NCVSP Cognitive Testing Memo

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Robert Houser

Clearance Officer

Justice Management Division

Alexis R. Piquero, Ph.D.

Director

Bureau of Justice Statistics

Kevin M. Scott, Ph.D.

Deputy Director, Statistical Operations Division

Bureau of Justice Statistics

FROM: Heather Brotsos and Rachel E. Morgan, Ph.D.

Victimization Statistics Unit

Bureau of Justice Statistics

DATE: December 6, 2022

SUBJECT: Request to cognitively test and start frame development work for the 2023 National Census of Victim Service Providers (NCVSP) under the OMB generic clearance agreement (OMB Number 1121-0339).

1. Introduction

The Bureau of Justice Statistics (BJS) through a cooperative agreement with the Justice Research and Statistics Association (JRSA), Westat, and the National Organization for Victim Assistance (NOVA), is preparing to collect data for the 2023 National Census of Victim Service Providers (NCVSP). The NCVSP aims to collect information about organizations serving victims of crime and abuse. This data collection was fielded previously in 2017. In planning with the project team and subject matter experts (SMEs), BJS has identified several questions on the survey that require adjustments to language and response options. BJS is requesting clearance to conduct two rounds of cognitive interviews with service providers to inform and refine the proposed changes, and to conduct usability testing of the instrument prior to the 2023 administration.

BJS is also requesting clearance to begin development work to update the NCVSP frame. Based on BJS’ experience conducting the 2017 NCVSP and 2019 National Survey of Victim Service Providers (NSVSP), which used the 2017 NCVSP frame for sampling, we know organizations may have updated contact information, ceased operations, changed/added services, or changed structure since 2017. New organizations have also been formed. Given these changes, frame development for the 2023 NCVSP begins with obtaining new lists of providers from federal, state and local agencies as well as national associations representing victim service providers.

2. History of NCVSP

Prior to the 2017 NCVSP, there had been no comprehensive collection of information about the landscape of the victim services field as a whole. Attempts to conduct more detailed research into the services for victims of crime were impeded by the lack of a frame, or roster, of service providers across the United States. Under the Victim Services Statistical Research Program (VSSRP), BJS initiated work to fill this gap through a project to conduct a census of victim service providers (VSPs). The goal of this effort was to produce essential information about the size and characteristics of the field. Additionally, this work would serve as the frame for further data collection and survey research efforts. The 2017 NCVSP captured information about organizations that actively provide services to crime or abuse victims as the primary function of the organization or through dedicated personnel or specific programs.

This second administration of the census requires assembling an updated frame to capture changes to the field, including large-scale efforts to expand victim services. In the years since the 2017 NCVSP, funding from the Victims of Crime Act (VOCA) has fluctuated, with an initial expansion of funding—including a set aside of funding for tribal organizations—and a more recent decrease in funding. Together with the economic and societal changes related to the COVID pandemic and increased awareness of the need for equity in service provision, the number and scope of victim service programs has likely changed. Therefore, the 2023 NCVSP will provide crucial information to planners, funders, researchers, and others regarding the changing landscape of services for victims of crime.

During a review of the 2017 NCVSP data file and subsequent data analysis, BJS and the project team determined that some questions did not produce desired information and had high missingness due to question-wording or response options. Some of these questions were interpreted differently by respondents. Questions related to agency structure, including agencies that may be co-located or have multiple offices, proved difficult for some agencies. Additionally, questions in the 2017 instrument about funding and about the specific populations or types of victimizations that organizations served demonstrated poor validity. The review of 2017 survey results also identified the need for more clarity in the unit of reporting level.

3. Proposed survey instrument revisions

The project team compiled documents relating to the 2017 NCVSP, including internal memos, published reports, and general instrument feedback from data users. The team reviewed these documents and cataloged general instrument issues, instances of missing data, survey disposition, and questions that produced data of uncertain validity. In response to this review, the instrument has been revised (Attachment A). These changes include new questions designed to better capture the unit of analysis—particularly those organizations that operate through chapters or satellite locations—and a revised section designed to gather data about funding for victim services. It also includes a number of changes to improve respondent understanding and ease of participation.

The following sections of the NCVSP instrument have been revised for 2023:

• Screening questions

• Organization/agency structure

• Types of services provided during the past 12 months

• Victimization types and victim populations served

• Number of paid staff who left direct victim service positions in the previous year

• Funding sources

• Issues of concern to the VSP field

BJS proposes testing these revised items to ensure clarity and response validity before seeking OMB approval to implement and administer a final version for the 2023 NCVSP. The testing will include cognitive interviews and usability tests, as described in the sections below.

4. Updating the NCVSP frame

The project team will combine any eligibility and/or contact information from the 2017 NCVSP and 2019 NSVSP. JRSA will also work with BJS to access data from recent BJS-funded surveys that ask about dedicated victim services staff with prosecutors, police, and hospitals; updated federal grantee lists (e.g., OVC, OVW, HHS); and other known victim service provider lists from states, national organizations, and other sources. Contacts may include managers at state victim coalitions or service provider networks, lead policy advocates, state victim assistance academy directors, or similar individuals. The team will pay particular attention to obtaining lists of providers working with underserved populations, for example victims of human trafficking and victims in underserved populations.

5. Testing procedures

5.1 Cognitive interviews

Cognitive interviews are important for evaluating respondent understanding and the ability to accurately answer survey questions. Cognitive interviews involve an interviewer administering the survey questions to a respondent and probing that respondent on how they interpreted the question, how difficult it was to answer, and their process for formulating an answer. Cognitive interviews are generally conducted prior to fielding BJS survey instruments that are new or have been substantively revised. Without such testing, the survey could yield a large quantity of incomplete information (because respondents will not be able to finish it or will not respond due to confusion or burden) or responses that are not uniform from one respondent to the next (e.g., the types of funding organizations receive).

5.1.1 Recruitment

Table 1, below, provides the breakdown of VSPs to be recruited for cognitive testing. We will recruit twice as many VSPs as we plan to cognitively interview to anticipate nonresponse from some VSPs. The 2017 NCVSP confirmed that government or nonprofit organizations make up the two largest types of VSPs; thus, more testers will be drawn from those types. Subject matter expert advisors to the project have already agreed to recruit up to seven VSPs within their networks, as directed. Project partner NOVA will recruit additional VSPs as needed. See Exhibit 1 for details on recruitment. A standard recruitment email is provided (Attachment B).

VSPs who agree to participate in cognitive testing will be connected to a staff member at JRSA, who will schedule the interview. Participants will be informed that they should allow approximately one hour to participate in the interview via video call.

Exhibit 1. NCVSP Cognitive Interview Recruitment Protocol

Table 1. Recruitment Targets for Cognitive Testing Round 1 and Round 2

Organization Type	Organization Category Tribal (T), Campus (C), Hospital (H), Government (G), Nonprofit (N), Informal (I)	Recruitment Target in Round 1	Recruitment Target in Round 2 (selection will be adjusted depending on the questions revised for Round 2)
State VOCA Assistance Funding Administrator - no direct services (testing screen out)	G	2
State VOCA Assistance Funding Administrator - direct services	G	2	2
State victim compensation program	G	2	2
Law enforcement agency without victim assistance program (testing screen out)	G	4
Law enforcement-based victim assistance program	G	2	2
Corrections-based victim service program	G	4	2
Nonprofit social service organization without victim assistance program (testing screen out)	N	2
Nonprofit VS TTA provider without direct service program (testing screen out)	N	2
Campus-based – large university	C	4	2
Campus-based - community college	C	4	2
Nonprofit with chapters, headquarters	N	4
Nonprofit, local chapter	N	2	2
Nonprofit with satellite locations, main office	N	4	2
Nonprofit, satellite location	N	2	2
Co-located program (one program operated at the site of another organization)	N or G	4	2
Tribal coalition	T	2
Tribal program, large Tribe	T	2	2
Tribal program, small Tribe or Alaska Native program	T	4	2
Program serving formerly incarcerated	N or G	4	4
Hospital-based program	H	4	2
Trafficking program, culturally-specific	N	2	2
Trafficking program, start up	N	4	2
Trafficking program, established	N	4	2
Domestic violence coalition	N	2	2
Domestic violence program, established	N	2	2
Family Justice Center	N or G	2
Domestic violence program, start-up	N	2	2
Sexual assault coalition	N	2	2
Sexual assault program, established	N	2	2
Sexual assault program, start-up	N	2
Children's Advocacy Centers (CACs), stand-alone	N or G	2	2
CAC, hospital-based	H	2
Informal program for homicide survivors	I	2	2
Total		90	50

5.1.2 Consent procedures

The email invitation (Attachment B), the introduction to the survey, and the cognitive interview script read to respondents at the beginning of the interview provide the elements of informed consent (Attachment C). The initial invitation provides the purpose of the NCVSP, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study.

5.1.3 Cognitive interviews

Cognitive interviews will be conducted in two rounds during January and February of 2023. Round 1 cognitive interviews will explore only those questions that have been changed or added since the 2017 NCVSP, unless a precursor question is required to promote understanding. Round 2 cognitive interviews will explore only those questions that were further added or adjusted following review of the results of Round 1 testing. Project staff at JRSA who have been trained in cognitive interviewing techniques will conduct the interviews.

Interviews will be conducted virtually using a video conferencing platform and in English. The interviewer will display one question at a time on the screen. Respondents will be asked to read the question and provide a response. The cognitive interviewer will follow a specific script which will explore such elements as the respondent’s grasp of the meaning of each question, the ability to answer each question, the degree of burden, and the appropriateness of data provided in response to specific questions (Attachment C). Interviews will be recorded with the respondents’ permission to ensure answers are accurately captured.

The goals of this cognitive interviewing are to:

• Identify any items that cause excessive burden for the respondents;

• Assess comprehension issues associated with any of the questions, including whether respondents interpret questions consistently; and

• Refine the wording of questions so that respondents to the final survey receive the clearest instructions possible concerning how victims and services should be defined.

Following the first round of interviews, the team will determine which changes should be made to the instrument and conduct a second round of interviews with a different group of VSP respondents and compile findings.

5.2 Usability testing

Once the instrument is finalized and formatted and before the 2023 NCVSP is launched, the project team will prepare materials for and conduct usability tests with 15 VSPs to ensure the self-administered, online and Computer-Assisted Telephone Interviewing (CATI) instruments function as intended.

5.2.1 Recruitment

VSPs who participate in the cognitive interview will be asked whether they would be willing to test the usability of the self-administered, online instrument. Fifteen VSPs who agree will be selected, in line with the needed categories of VSPs indicated in Table 2. In addition, BJS and the project team will pretest the CATI instrument to make sure it is functioning as intended and correctly programmed.

Table 2. Target Usability Testers by VSP Type

Organization Type	Organization Category Tribal (T), Campus (C), Hospital (H), Government (G), Nonprofit (N), Informal (I)	Usability Testers
Law enforcement-based victim assistance program	G	1
Campus - large	C	1
Nonprofit with chapters, headquarters	N	1
Nonprofit, local chapter	N	1
Nonprofit with satellite locations, main office	N	1
Nonprofit, satellite location	N	1
Co-located program (one program operated at the site of another organization)	G or N	1
Tribal program, large Tribe	T	1
Tribal program, small Tribe or Alaska Native program	T	1
Trafficking program, culturally-specific	N	1
Trafficking program, established	N	1
Domestic violence coalition	N	1
Domestic violence program, start-up)	N	1
Sexual assault program, established	N	1
Children's Advocacy Centers (CACs), stand alone	G or N	1
Total		15

5.2.2 Consent procedures

The email invitation to usability testing provides the elements of informed consent including the purpose of the NCVSP, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study (Attachment B).

5.2.3 Usability tests

Usability testing will be conducted in early May of 2023. Fifteen VSPs will be asked to complete the online instrument and respond to a short online questionnaire about their experience immediately after completion of the survey (Attachment D).

Because we expect a portion of VSPs to complete the survey by phone, BJS and the project team will pretest the CATI script and verify it is programmed as described in the draft NCVSP instrument. The CATI script is designed to establish type of VSP and eligibility status early and to ensure that conducting the NCVSP by phone results in high-quality, complete data. After this test, BJS and the project team will discuss any issues.

5.3 Burden

As shown in Table 3, BJS estimated the respondent burden to be 94.55 hours. This burden comprises three components: initial contact and scheduling, cognitive interviews, and usability testing. Up to 140 VSPs may be contacted and asked to participate in the cognitive interviewing, however, a maximum of 70 VSPs will complete an interview. The current request for approval, under the OMB Generic Clearance (OMB Number 1121-0339), is for two rounds of cognitive testing of the revised sections of the NCVSP instrument with up to 45 VSPs in the first round and 25 in the second round. Because some of the categories of needed testers will overlap (e.g., a domestic violence service provider may operate through a satellite location), the final number of testers may be less than 70. The second round of cognitive testing will use an instrument updated based on feedback from the first round of interviews.

Table 3. Burden Associated with Planned NCVSP Cognitive Testing and Usability Testing

Test	Maximum # of Respondents	Average administration time (minutes)	Burden (hours)
Initial contact and scheduling for round 1 + round 2 cognitive interviews	140	10	23.3
Cognitive testing
Round 1	45	60	45
Round 2	25	45	18.75
Total	70	-	63.75
Usability testing	15	30	7.5
TOTAL	225	-	94.55

Additionally, as shown in Table 4, BJS estimated the respondent burden for the frame development work to be 100 hours. This estimate includes the burden associated with organizations accessing and electronically sharing lists of victim service providers with BJS.

Table 4. Burden Associated with NCVSP Frame Development Work

	Maximum # of Respondents	Average administration time (minutes)	Burden (hours)
Organization sharing with BJS	100	60	100
TOTAL	100		100

The total respondent burden requested in this generic clearance agreement is 194.55 hours.

5.4 Language

The online testing and cognitive interviews will be conducted in English.

6. Timeline

Milestone	Start Date	End Date
Obtain OMB generic clearance	12/05/22	12/19/22
Recruitment and testing period	12/20/22	02/10/23
Analyze data and develop instrument recommendations	02/13/23	02/27/23
Draft final report	02/28/23	03/14/23

7. Informed consent

The introduction to the census and the script read to respondents at the beginning of the virtual cognitive interview will provide information about the purpose of the NCVSP, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study. It will announce the estimated length of the interview in advance, allowing the participant an opportunity to decline if the burden would be unacceptable. Cognitive interviewees will be asked their permission to record the interview.

Usability testers will also receive an initial email invitation that provides information about the purpose of the NCVSP, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study. It will also include the estimated length of the usability test and feedback provision in advance, allowing the participant an opportunity to decline if the burden would be unacceptable.

8. Data confidentiality and security

BJS is authorized to conduct this data collection under 34 U.S.C. § 10132. The testing will collect identifying information including the name of the testing organization, address, email, and telephone number. All information related to the cognitive interviews and usability tests, including the recordings of interviews, will be stored on a secure drive at JRSA with restricted access to those staff members who are directly involved in cognitive or usability testing. To protect the identities of the respondents, including the cognitive interviewees and usability testers, no identifying information will be kept in the final data file. In addition, the recorded conversations of the interviews will be erased upon completion of the cognitive testing report. We estimate this to be one month after the pretest has ended. Once the instrument is revised and the summary report completed, all copies of the cognitive interview data will be destroyed.

All project staff are required to sign a pledge of confidentiality and privacy certificate which confirms the maintaining of data and following the procedures outlined above. Furthermore, all cognitive interviews will be conducted by project staff at JRSA, who must complete official security training.

9. Cost

VSP respondents will not be compensated for their participation.

10. Reporting

Upon completion of the cognitive interviewing and usability testing, JRSA will provide BJS with a report describing the findings and including final recommendations. It will also include any suggested changes to the instrument based on the cognitive interviewing. The report will provide detailed information on the testing methodology, respondent characteristics, and findings related to the new or revised questions tested.

11. Protection of Human Subjects

JRSA’s Institutional Review Board (IRB), which has Federal-wide assurance, has reviewed the planned testing activities, and designated these activities as ‘not human research’ (Attachment E).

Contact information

Questions regarding any aspect of this project can be directed to:

Rachel E. Morgan, Ph.D.

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7^th Street NW

Washington, DC 20531

Phone: (202) 598-9237

Email: Rachel.Morgan@usdoj.gov

Attachments

Attachment A: Draft 2023 NCVSP survey instrument

Attachment B: NCVSP cognitive & usability test invitation emails

Attachment C: NCVSP cognitive testing protocol

Attachment D: NCVSP usability test debrief

Attachment E: JRSA IRB approval

5