Memo to OMB on Cog Test for NIS-4

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Robert G. Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

Through: Jeffrey H. Anderson, Ph.D.

Director

Bureau of Justice Statistics

From: Jessica Stroop

Statistician, Project Manager

Allen Beck, Ph.D.

Senior Statistician

Bureau of Justice Statistics

Date: January 21, 2021

Re: BJS request for OMB Generic Clearance for cognitive testing for the National Inmate Survey (NIS-4), under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) seeks generic clearance approval for cognitive interviewing activities related to the National Inmate Survey (NIS-4) program. The NIS supports BJS in meeting the statistical, data and research requirements of the Prison Rape Elimination Act (PREA; P.L. 108-79, Sec.4; see Attachment 1).

The act requires BJS to “carry out, for each calendar year, a comprehensive statistical review and analysis of the incidence and effects of prison rape.” The act further instructs BJS to conduct surveys and other statistical studies of current and former inmates. To implement the act, BJS developed the National Prison Rape Statistics Program (NPRS), which includes four separate data collection efforts: the Survey of Sexual Victimization (SSV), the National Survey of Youth in Custody (NSYC), the National Former Prisoner Survey (NFPS), and the National Inmate Survey (NIS).

These collections are independent, and while not directly comparable, they provide separate measures of the prevalence of sexual victimization and detailed information on the characteristics and circumstances surrounding that victimization. The SSV collects information about incidents of sexual violence reported to adult and juvenile correctional authorities, as well as characteristics of incidents that are substantiated upon investigation. The NSYC gathers self-reported sexual assault data from youth held in state-operated juvenile facilities and locally or privately operated facilities that hold youth under state contract. The NFPS collects data on sexual victimization from former state prisoners, during the totality of their prior term of incarceration, including any time in a local jail, state prison, or community correctional facility prior to final discharge. The NIS collects data on allegations of sexual victimization from inmates held in prisons and jails.

Between April and December 2007, BJS completed the first NIS (NIS-1) in 282 local jails and 146 state and federal prisons. Between October 2008 and December 2009, BJS conducted the second NIS (NIS-2) in 286 local jails and 167 state and federal prisons. Between February 2011 and May 2012, BJS conducted the third NIS (NIS-3) in 233 state and federal prisons and 358 local jails.

In all three rounds of the survey the data were collected by RTI International (RTI) under a cooperative agreement with BJS. In each round, inmates completed an audio computer-assisted self-interview (ACASI), in which they interacted with a computer-administered questionnaire, heard questions and instructions delivered via headphones, and responded via a touch-screen enabled laptop computer. A total of 63,817 inmates participated in NIS-1; 81,566 inmates participated in NIS-2; and 92,449 inmates in NIS-3.

Request for Cognitive Testing

This submission is to seek clearance for English language cognitive testing of selected items in preparation for the fourth National Inmate Survey (NIS-4). This request is part of the cooperative agreement between BJS and RTI (award 2008-RP-BX-K001.) As one of the remaining tasks under the agreement, BJS and RTI have modified items in NIS-3 and developed new items for inclusion in NIS-4. Cognitively testing these items will enable us to determine how respondents interpret the questions and response options, and to evaluate their appropriateness for inclusion in the next NIS.

The cognitive testing activities will include a maximum of 50 interviews from adult respondents who are either currently held in a local detention facility or recently released from prison or jail. RTI will seek approval to conduct interviews with inmates from one detention facility in South Carolina. RTI will also contact one or more local non-profit organizations in North Carolina that provide aid to recently released inmates to recruit respondents for the survey.

Participation in the cognitive test will be completely voluntary. Only adults (persons age 18 or older) will be invited to participate. They will be given the option to refuse at any time during the test should they wish. During the consent process, volunteers will be provided information that describes the interview, why they were chosen, what will happen during the discussion, the risks and benefits of participation, and details ensuring confidentiality. (See Attachment 2.). Inmates who consent to the test will be offered a snack to eat while they complete the interview. Former inmates will be offered $40 as an incentive and to compensate for their time and effort.

The cognitive testing procedures will follow the same procedures used for the NIS-3. The main focus of the cognitive process is on four cognitive tasks required to answer a survey question: interpretation, retrieval/recall of information, judgment formation (sorting through information in order to formulate and identify a response), and response editing (deciding which response to report). The NIS-3 procedures were previously approved by the RTI IRB, which has federal wide assurance. (See Attachment 3.) After drafting the new items for NIS-4, RTI submitted the items and a description of the activity to its IRB. The IRB chair determined that a full IRB review was not necessary, based on the determination that the items are to be administered in a cognitive test, which is not designed to contribute to generalizable knowledge and therefore does not constitute research as defined by 45 CFR 46.102. This determination is in the attached letter from the RTI IRB. (See Attachment 4.)

However, following the cognitive testing activities, BJS will submit the full survey for an IRB review and approval. A full cognitive test activity report will accompany the IRB submission and all future OMB clearance requests for pilot and full scale field testing.

Cognitive Testing Overview

The purpose of this memo is to seek approval to conduct cognitive interviews with a maximum of 50 adults (age 18 or older) who are either currently incarcerated in a jail or were previously incarcerated in prison or jail facilities. There are a total of 49 questions that will be tested along with their corresponding response options. The instrument to be tested and the testing protocol are presented in Attachment 5. The questions in the interview have been time tested to confirm that they can be completed within 18 minutes; however, it is estimated that the full cognitive interview, including probes, will take a maximum of 60 minutes.

Items being tested are those that are new content and have not been used in previous NIS collections or tested with correctional populations in other studies. They include -

• U.S. citizenship (1 item)

• Sexual orientation and gender identity (4 items)

• Inmate-on-inmate incident details (15 items)

• Staff-on-inmate incident details (9 items)

• Life prior to incarceration experience (1 item)

• Disability (1 item)

• Restrictive housing (18 items)

Respondents will not be asked to provide answers to the survey questions, rather they will be asked to review the question while the interviewer reads it aloud, and then asked for their interpretation of what is meant by the question and any terms that may be potentially confusing or misunderstood.

Goals of the Cognitive Testing

The goal of the cognitive testing is to assess the respondents’ general comprehension of the new survey items. This will be accomplished through the use of traditional cognitive interview probing techniques. Interviewers will hand a paper version of the ACASI items to the respondent and will read the questions aloud as the respondent reads along. Respondents will be asked to comment on their interpretation of each question being tested. The interviewer reading the questions aloud most closely mimics the ACASI administration, which is designed to address the low literacy levels often found among current and former inmates.

Eligibility and Selection of Respondents for Cognitive Interviews

The goal is to recruit 15 inmates and 35 former inmates to participate in the cognitive testing. Both males and females will be eligible for the interviews, and all respondents must be at least 18 years of age. Staff at the non-profit organizations and the detention facilities will not be involved in the screening process nor in the scheduling of interviews. RTI staff will coordinate with staff at the detention facility and at the non-profit organizations to ensure that appointments are scheduled only during times when private interviewing space (e.g., a conference room or office) will be available.

Former Inmates. Among the former inmates, RTI will attempt to recruit a total of 15 females and 20 males. RTI will provide flyers (Attachment 6) to the staff at the non-profit organizations where former inmates receive assistance. Staff will be asked to post the flyers in areas where former inmates will be likely to see them when they visit the organizations to receive services. Staff will also have copies they can hand out to former inmates with whom they meet. The flyer will provide a brief description of the cognitive interviewing activity and will include a phone number for interested individuals to call to receive further information and determine whether they are eligible for an interview. All staff at the non-profit organizations will be informed of the interviewing so that they can direct individuals to review the flyers.

Upon contacting RTI, individuals will be asked when they were released from jail. To be eligible for the cognitive interview, the individuals must have been released within the previous 6 months

Current Inmates. For current inmates, RTI will attempt to recruit five females and 10 males. RTI will work with staff at the detention facilities to determine the best times of day to be at the facility in order to maximize the number of inmates who may be available to hear about the interviews and when private interviewing space will be available. A correctional officer will make an announcement in a housing unit that a researcher from RTI is onsite and is interested in talking one-on-one with inmates for about an hour. The correctional officer will not provide any detailed description about the nature of the interview; that explanation will be provided by the RTI interviewer. Once an inmate comes to meet with the RTI interviewer, the interviewer will verify the inmate’s age (to ensure only inmates 18 or older are interviewed) and then complete the informed consent procedures. Any inmate who does not wish to participate will be excused, and the interviewer will ask the correctional officer to bring the next interested inmate. When an inmate agrees to be interviewed, the interviewer will conduct the interview and then excuse that inmate before screening the next inmate.

Consent for Cognitive Interviews

Prior to the start of an interview, respondents will be informed about the purpose of NIS and the specific purpose of the cognitive interviews. All consent procedures specified for the national study will be communicated and followed as well. Interviews will only be conducted if consent is obtained for those ages 18 or older. The interviews will be recorded and respondents will receive an incentive for participating.

Interviewing incarcerated individuals presents distinct methodological challenges. The NIS is completely voluntary even though it is administered to inmates. Because of the sensitive nature of the survey, it’s imperative that the items be designed at a level that is cognitively appropriate for those who may have limited literacy. The items must also display a level of familiarity for the structure, design and function of today’s correctional facilities.

Use of Incentives

Based on past experience with NIS, it is helpful to offer an incentive to this hard-to-reach population, to encourage participation and reduce any potential nonresponse bias. To encourage participation by former inmates, a cash incentive of $40 will be offered. The purpose of this incentive is to offset the cost of participation. Eligible participants are recently released jail or prison inmates who must travel to the facility to participate in an interview. The cognitive test cannot be conducted through a web-based interview or via a face-to-face interview at their home, as many of the potential participants have not yet secured housing following their release. Moreover, RTI must ensure that the interview setting is private. For these reasons, we need to offer a cash incentive to compensate respondents who are willing to travel to the designated facility and consent to be interviewed.

For the adults currently held in a detention facility, a small bag of cookies or chips (value less than $1.00) will be offered. This is consistent with prior administrations of the NIS in which inmates consenting to the interview were offered and could consume a small snack before returning to their housing unit. The purpose of this incentive is to encourage participation. Jail inmates are frequently presented with education, job training, treatment or visitation opportunities that can be more attractive than voluntary participation in a survey. The incentive is being offered to make survey participation more attractive and reduce bias.

Past NIS collections have demonstrated that such incentives result in higher response rates. Similar large-scale federal surveys that ask about sensitive behaviors have successfully used incentives in their process, including the National Survey on Drug Use and Health (NSDUH), the National Survey of Family Growth (NSFG), and the National Health and Nutrition Examination Survey (NHANES).^{1, 2}

Cognitive Interview Procedures

All interviews will be conducted by a senior member of the RTI project staff who has been specifically trained for this study. RTI staff will not obtain the name of any participants. During the interviews, prior inmates will be asked to comment on their interpretation of the draft questions. They will be instructed to read along as the interviewer reads each question aloud. After each question, the interviewer will invite the respondent to comment on the question intent or wording. After specific questions, as defined in the cognitive interview protocol, the interviewer will probe for the respondent’s interpretation of particular terms or phrases or ask them to discuss the appropriateness of the question. (See Attachment 5 for the protocols and items that will be tested with all respondents.)

Considerations and Assurances for All Cognitive Test Activities

Protection of Human Subjects

There is some risk of emotional distress for the respondents, given the sensitive nature of the topic of sexual victimization, particularly since some of the questions are explicit and of a personal nature. However, since respondents are not being asked to answer the sensitive questions about themselves, but are only being asked to comment on their comprehension of the items, we believe the risk will be minimal. There were no human subject concerns raised by the RTI board that provided the waiver.

Language

All cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

The burden for this test consists of (1) the facilities arranging for the respondents to participate in the interviews and (2) conducting cognitive interviews. The burden associated with these activities is presented in Table 1.

The expected burden for the respondent to complete the interview is approximately 60 minutes. This estimate is based on past experience with cognitive testing in NIS-1, NIS-2 and NIS-3, and preliminary time testing by RTI staff of the new items. A maximum of 50 respondents will be involved in the cognitive testing, and 3 facility staff, for a maximum total burden of 81 hours.

For both facilities, it is anticipated that agency staff will be engaged in the following activities: 1) arranging for the data collection visit; 2) identifying and securing participation of respondents; 3) preparing appropriate interviewing space; and 4) arranging for counseling services. The total estimated staff burden for these activities is 5 hours per facility for each staff member.

In the jail facility, the goal is to complete 15 interviews. It is estimated that completing 15 interviews will take approximately 4 days, spending 4 hours per day in the facility for a total of 16 hours with a staff person available to monitor the interview during these hours, plus 5 hours of burden for the staff liaison to arrange for the visit and associated logistics for a total of 36 burden hours.

For the non-profit facility, the goal is to complete 35 interviews, or 35 hours, plus the estimated staff burden for these activities of 5 hours per staff, or 10 hours, for a total of 45 burden hours. In this facility, it is not necessary to have security staff available to monitor the interviews.

Table 1

Estimated Burden Hours Associated with

Planned NIS Cognitive Testing Activities

	Maximum # of Respondents	Jail Burden Hours Per Respondent (1 facility)	Non Profit Facility Burden Hours Per Respondent (1 facility)	Maximum Burden (hours)
Facility liaison	3 respondents, 5 hours per respondent for logistics, 1 staff at jail for interview monitoring	1 staff member = 5 hours for logistics, 1 staff person for 16 hours of monitoring during interviews	2 staff members = 10 hours for logistics only - staff for monitoring not necessary	31 hours
In-person Cognitive Interviewing at Facility	50	15	35	50
TOTAL BURDEN	53 participants	36 hours	45 hours	81 hours

The number of interviews with participants in the facility will provide a sufficient number of interviews to detect any important issues regarding the questions and response categories. It will also permit the sample to be spread across male and female and varied age participants as described above.

Costs to the Federal Government

The total cost of conducting the cognitive interviews will be approximately $45,065 under the cooperative agreement with RTI (Award 2008-RP-BX-K001) for the NIS-3. The breakdown of costs are as follows:

35 interviews with former inmates: 35 x $40.00 incentive = $ 1,400.00

15 interviews with current inmates: 15 x $1.00 snack = $ 15.00

Labor for recruiting/scheduling/interviewing and summarizing results by RTI: $43,650.00

$45,065.00

Actual costs may be lower if fewer than 50 interviews are conducted. This estimate excludes the cost of BJS staff working on the project team.

Use of Information Technology to Reduce Burden

The cognitive testing will utilize technology (primarily email and telephone communication) to facilitate recruitment and the scheduling process to reduce participant burden and control study costs. Where feasible and appropriate, we will use email to communicate with involved parties. Due to the small scale of this cognitive test, all interviews will be conducted face to face with paper versions of the survey read aloud to mimic ACASI and interviewers will utilize laptops to take detailed and organized notes.

Data Confidentiality and Security

BJS’s pledge of confidentiality is based on its governing statutes Title 34 USC, Section 10231, which establishes the allowable use of data collected by BJS. Under this section, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 10134). BJS staff, other federal employees, and RTI staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 34 USC. Sec. 10231, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). To protect the identity of the respondents, no identifying information will be kept on the final data file. The survey will not be collecting the name of any of the respondents.

Data Analysis

Cognitive interviewers will summarize the findings from each completed interview based on the completed questionnaire, and notes taken during the interview. The summaries will be analyzed to help identify common themes organized by overall questionnaire issues, individual questionnaire items and sections, and participants’ overall reactions to the questionnaire.

The cognitive interviewing analysis will assess and identify problems such as comprehension issues, difficulties understanding the tasks, and any concerns related to the translation of items. These issues will be assessed qualitatively, based on the interviewers’ assessments of their own experience.

Upon completion of all the cognitive testing, a draft cognitive interviewing report will be delivered to BJS that will include recommendations for the final revision to the survey for the pilot test. These recommendations will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, average time needed to complete the survey instruments, and any issues with question comprehension. The report will also document changes made to the NIS-4 survey instruments that are being recommended for use in the pilot and field test.

Contact Information

Jessica Stroop

Statistician, Project Manager

Bureau of Justice Statistics

Phone: 202-598-7610

Email: jessica.stroop@usdoj.gov

Allen Beck, Ph.D.

Senior Statistical Advisor

Bureau of Justice Statistics

Phone: 202-616-3277

Email: allen.beck@usdoj.gov

Rachel Caspar, M.A.

Director, Center for Survey Methodology

RTI International

Phone: 919-541-6376

Email: caspar@rti.org

1 Center for Behavioral Health Statistics and Quality. “Behavioral Health Trends in the United States: Results from the 2014 National Survey on Drug Use and Health.” 2015. Available at http:// www.samhsa.gov/data/sites/default/files/NSDUHFRR1-2014/NSDUH-FRR1-2014.pdf. Accessed September 6, 2016.

2 Centers for Disease Control and Prevention. “Evaluation Briefs: Using Incentives to Boost Response Rates.” July 2010. Available at https://www.cdc.gov/ healthyyouth/evaluation/pdf/brief22.pdf. Accessed September 6, 2016.

7