Memo to OMB for Spanish Version of NSYC-3 Cognitive Testing

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Jennifer E. Park

Office of Statistical and Science Policy

Office of Management and Budget

Through: Melody Braswell

Department Clearance Officer

Justice Management Division

Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

From: Jessica Stroop

Statistician, Project Manager

Allen Beck, Ph.D.

Statistician

Bureau of Justice Statistics

Date: January 22, 2021

Re: BJS Request for OMB Clearance for Spanish Cognitive Testing for the National Survey of Youth in Custody (NSYC-3) under the BJS Generic Clearance Agreement (OMB# 1121-0339)

Introduction

The Bureau of Justice Statistics (BJS) seeks generic clearance approval for Spanish language cognitive interviewing activities related to the National Survey of Youth in Custody (NSYC) program. The NSYC supports a statistical collection mandated by Congress (Public Law 108-79; see Attachment 1).

Purpose of the Research

On September 4, 2003, the Prison Rape Elimination Act of 2003 (PREA or the Act) was signed by President George W. Bush (Public Law 108-79). The Act requires BJS to “carry out, for each calendar year, a comprehensive statistical review and analysis of the incidence and effects of prison rape.” The Act further instructs BJS to conduct surveys and other statistical studies of current and former inmates. To implement the Act, BJS developed the National Prison Rape Statistics Program (NPRS), which includes four separate data collection efforts: the Survey of Sexual Victimization (SSV), the National Inmate Survey (NIS), the National Survey of Youth in Custody (NSYC), and the National Former Prisoner Survey (NFPS).

These collections are independent, and while not directly comparable, they provide various measures of the prevalence and characteristics of sexual assault in correctional facilities. The SSV series collects information about incidents of sexual violence reported to and investigated by adult and juvenile correctional authorities, as well as characteristics of substantiated incidents. The NIS collects allegations of sexual assault self-reported by adult and juvenile inmates in adult correctional facilities. NSYC gathers self-reported sexual assault data from youth in juvenile correctional facilities. The NFPS measures allegations of sexual assault experienced during the respondent’s last incarceration, as reported by former state inmates under active post-custody supervision.

Between June 2008 and April 2009, BJS completed the first NSYC (NSYC-1) in 166 state-owned or operated facilities and 29 locally or privately operated facilities, and between February 2012 and September 2012, BJS completed the second NSYC (NSYC-2) in 273 state-owned or operated facilities and 53 locally or privately operated facilities. Both rounds of the survey were conducted by Westat. In both rounds, youth completed the survey using an audio computer-assisted self-interview (ACASI), whereby they heard questions and instructions via headphones and responded via a touch-screen. For NSYC-1, a total of 10,263 youth participated in the survey; for NSYC-2, 9,703 youth participated.

Request for Pilot Testing

This submission is to seek clearance for the cognitive testing of the Spanish language version of the instrument prepared for the third NSYC (NSYC-3). This request falls under a cooperative agreement BJS has with Westat to collect data for NSYC-2, (award 2009-RP-BX-K001). As one of the remaining tasks under the agreement, BJS and Westat have developed and are testing new items for inclusion in NSYC-3. Cognitive testing of the items in English was conducted in August 2016 with 20 youth residing at three in loco parentis (ILP) facilities in two states. As a result of the testing, a number of edits were made to new items for NSYC-3. Cognitively testing these new items in Spanish will enable us to better understand the way Spanish-speaking youth are interpreting the questions and response options, and to evaluate the appropriateness of the questions for youth in correctional facilities.

The cognitive testing activities will include 16 to 20 interviews with youth whose primary language is Spanish, or who are fluent in Spanish, and who are living in juvenile residential correctional facilities to test new items being considered for NSYC-3. The interviews will be conducted at facilities that provide consent in loco parentis for the minors in their custody. Youth invited to participate in the survey will have the choice to refuse or participate in the cognitive test.¹ Youth assent materials for participation (Attachment 3) are attached in English (for OMB reference) and Spanish.

The NSYC-3 Spanish cognitive testing procedures and designs have been approved by the Westat IRB which has federal wide assurance and had previously approved the testing of the English materials for NSYC-3 in 2016. A copy of the approval notice for the NSYC-3 Spanish cognitive testing effort is attached (Attachment 4).

Cognitive Testing Overview

The purpose of this memo is to seek approval to conduct cognitive interviews with 16 to 20 Spanish-speaking youth in correctional facilities who will test new items being considered for NSYC-3. There are a total of 118 question items being tested. This is a comparable number of items to what was tested among English speaking youth in the 2016 cognitive test. The questions have been divided into two separate surveys (by topic area) and each version will be tested with 8-10 youth. The two versions of the instrument and testing protocols are presented in Attachment 2a in English, and in Attachment 2b in Spanish. Each version of the protocol and questionnaire have been pretested to confirm that each can be completed within 60 minutes. Items being tested are those that are new content and have not already been tested with other youth populations in other studies. These include:

• Version 1 (60 items)

  • Staff treatment of youth (3 items)

  • Youth misconduct in the facility (1 item)

  • Youth education on PREA (Prison Rape Elimination Act) (7 items)

  • Mental health of youth in the facility (7 items)

  • Youth incident details (19 items)

  • Disability (11 items)

  • Sexual orientation and gender identity (4 items)

  • Facility living conditions (if time permits) (8 items)

• Version 2 (58 items)

  • Legal counsel & institutional experiences (3 items)

  • Solitary confinement (9 items)

  • Youth/gang involvement (4 items)

  • Treatment programs (3 items)

  • Staff grooming (4 items)

  • Staff incident details (27 items)

  • History of victimization (3 items)

  • Youth education and aspirations (1 item)

  • Sexual orientation and gender identity (if time permits) (4 items)

For the more sensitive items being tested (regarding youth incident details, staff incident details, grooming behaviors, drug use, and history of victimization), respondents will not be asked to provide answers to the survey questions, rather the youth will be asked to review the question while the interviewer reads it aloud, and then the interviewer will immediately ask the respondent for their interpretation of what the question is asking and any terms that may be potentially confusing or misunderstood. Likewise, if particular items do not apply to youth (such as receiving counseling services or being placed in solitary confinement), respondents will only be asked to provide their interpretation of the question, rather than be asked to provide an answer.

Goals of the Cognitive Testing

The goal of the interviews is to test the respondents’ general comprehension of the Spanish language version of the survey questions. This will be accomplished through the use of traditional cognitive interview probing techniques. Spanish-speaking interviewers will hand a paper version of the ACASI items to the respondent and will read the questions aloud as the respondent reads along. For most items (other than those mentioned above), respondents will answer the question and then will be asked to comment on their interpretation of each question being tested. For the more sensitive items mentioned above, respondents will be asked only to comment on the question, not to answer it.

Translation Process

Translation of documents from English into Spanish for the National Study of Youth in Custody was performed by Westat’s in-house English-Spanish translation team. This team is composed of bilingual native Spanish-speaking and native English-speaking linguists who have been working in the field of translation for a minimum of 8 years.

The first step in the translation process was the creation of a draft Spanish translation of the documents by one of the native Spanish linguists. Next, the documents were thoroughly proofread and reviewed by a second native Spanish-speaking linguist. The recommended edits resulting from that review were sent back to the translator, who reviewed each proposed change carefully and implemented them accordingly, based on principles of consistency, appropriate register and tone, rules of grammar and syntax, and cultural appropriateness for Spanish speakers living in the United States. The third step consisted of review by a third Spanish-speaking linguist, an English-language native, with a view to ensure the highest level of equivalency between the source and target texts. Again, these recommended edits were sent back to the translator who reviewed each proposed change and implemented them accordingly. Lastly, a final quality control review was conducted to ensure the texts were free of typographical errors.

Eligibility and Selection of Youth for Cognitive Interviews

Two juvenile facilities will be identified where administrators will permit cognitive interviews with their residents. See Attachment 5 for materials that will be used to recruit state agencies and specific facilities within those states. A total of 8 to 10 adjudicated youth whose primary language is Spanish or who are fluent in Spanish will be selected by each facility. Some of the youth may be over 18, however, we expect some to be minors. These facilities (or their state) will provide consent in loco parentis to interview the minors. Only those selected for the cognitive test who are over 18 years of age, or for whom the facility provides consent, will be eligible to participate in the cognitive test.

Youth Assent for Cognitive Interviews

Prior to the start of an interview, the youth will be informed about the purpose of NSYC and the specific purpose of the cognitive interviews. All mandatory reporting and counseling procedures specified for the national study will be communicated and followed as well. Interviews will only be conducted if youth assent is obtained for those under age 18, and if consent is provided for those ages 18 or older. The interviews will not be recorded and youth will not receive any incentive for participating. The youth assent form is provided in Attachment 3. Interviews will only be conducted if the youth assents.

Cognitive Interview Procedures

All interviews will be conducted by a senior member of the Westat project staff who has been specifically trained for this study and is a native Spanish-speaker. Westat staff will never ask for the name of any youth. During the interviews, youth will be asked to comment on their interpretation of the draft questions. The youth will be instructed to read along as the interviewer reads each question aloud. After each question, the interviewer will invite the youth to comment on the question intent or wording. After specific questions, as defined in the cognitive interview protocol, the interviewer will probe for the youth’s interpretation of particular terms or phrases or ask the youth to discuss the appropriateness of the question. See Attachment 2 for the English version of the protocols that have been translated to Spanish, and the items that will be tested with Spanish-speaking youth.

Considerations and Assurances for All Cognitive Test Activities

Protection of Human Subjects

There is some risk of emotional distress for the facility youth respondents, given the sensitive nature of the topic of sexual victimization, particularly since some of the questions are explicit and of a personal nature. However, since respondents are not being asked to answer the sensitive questions about themselves, but are only being asked to comment on their comprehension of the items, we believe the risk will be minimal. There were no human subject concerns raised in the English round of testing conducted in 2016.

The interviews will be stopped if requested by the respondent or if the respondent shows signs of emotional distress. All respondents will be provided with an opportunity to speak with a counselor following the interview. Facility youth will be provided with an onsite resource if they wish to speak to a counselor, and we will also arrange for an offsite counselor for those who wish to speak to someone outside the facility.

Language

All cognitive interviews will be conducted in Spanish.

Burden Hours for Cognitive Testing

The burden for this task consists of facilities arranging for the youth to participate in the interviews, and the youth participating in cognitive interviews. The burden associated with these activities is presented in Table 1.

The expected burden for each facility to complete this testing is approximately 17 hours, for a total of 34 hours to complete the test in two facilities (17 hours per facility x 2 facilities = 34 hours). . This estimate is based on past experience with NSYC-1 and NSYC-2, and the NSYC-3 English cognitive testing. Thirteen people per facility will be involved in the cognitive testing for a total of 26 people to complete the test in two facilities. (1 state liaison + 2 staff to escort youth + 10 respondents = 13 x 2 facilities = 26 total).

The 6 hours of burden for working with the state and facility liaisons consists of 1) contacting 1-2 facility superintendents/administrators to determine willingness and availability to participate; 2) providing authorization for Westat to conduct the research; 3) communicating informed consent and mandatory reporting procedures to facility staff; 4) specifying security clearance requirements for Westat staff; and 5) identifying external counseling resources for youth who request such information. These activities will take an estimated 3 hours per state liaison or 6 hours total.

It is anticipated that facility staff will be engaged in the following activities: 1) arranging for the data collection visit; 2) identifying and securing participation of 8-10 youth; 3) preparing appropriate interviewing space; 4) obtaining consent for youth through in loco parentis; 5) arranging for counseling services; 6) outlining mandatory reporting procedures for the state; and 7) escorting youth to and from interviews. The total estimated staff burden for these activities is 4 hours per facility security staff, or 8 hours total.

Expected burden placed on youth for this data collection averages 1 hour per respondent for the facility interviews. We estimate between 16 to 20 facility youth will participate, resulting in a maximum estimated facility youth burden of 20 hours.

Table 1

Estimated Burden Hours Associated with

Planned NSYC Spanish Cognitive Testing Activities

	Maximum # of Respondents	Average Administration Time (minutes)	Maximum Burden (hours)
State and facility liaison	2 state liaisons	180	6
Facility communications and logistics with staff	4 staff	120	8
In-person Cognitive Interviewing at Facilities	20 youth	60	20
FACILITY BURDEN TOTAL	13 people		17 hours
TOTAL	26 people		34 hours

The number of interviews with youth in facilities is based on the desire to complete 8-10 interviews for each of the two sets of questions. This should be sufficient to detect any important issues regarding the questions in each group. This will also permit the sample to be spread across the 12-21 age range, as well as across both male and female youth.

Costs to the Federal Government

The total cost of conducting the cognitive interviews, will be approximately $20,000 under the cooperative agreement with Westat (Award 2009-RP-BX-K001) for the National Survey of Youth in Custody (NSYC-2). This excludes the cost of BJS staff working on the project team.

Use of Information Technology to Reduce Burden

The cognitive testing study will utilize technology to facilitate recruitment and the scheduling process to reduce participant burden and control study costs. Where feasible and appropriate, we will use email to communicate with states and facilities.

Data Confidentiality and Security

BJS’s pledge of confidentiality is based on its governing statutes Title 42 USC, Section 3735 and 3789g, which establish the allowable use of data collected by BJS. Under these sections, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 3735). BJS staff, other federal employees, and Westat staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 42 U.S.C. Sec. 3789g, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). To protect the identity of the respondents, no identifying information will be kept on the final data file. The survey will not be collecting the name of any of the respondents.

Data Analysis

Cognitive interviewers will summarize the findings from each cognitive interview. Summaries will be written in English but will use Spanish to highlight comments and recommendations. Interviewers will prepare summary findings on each completed interview based on the completed questionnaire, and notes taken during the interview. The summaries will be analyzed to help identify common themes organized by overall questionnaire issues, individual questionnaire items and sections, and participants’ overall reactions to the questionnaire.

The cognitive interviewing analysis will assess and identify problems such as comprehension issues, difficulties understanding the tasks, and any concerns related to the translation of items. These issues will be assessed qualitatively, based on the interviewers’ assessments of their own experience.

Upon completion of all the cognitive testing, a draft cognitive interviewing report will be delivered to BJS that will include recommendations for the final revision to the Spanish survey for the field test. These recommendations will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, average time needed to complete the survey instruments, and any issues with question comprehension. The report will also document changes made to the Spanish NSYC-3 survey instruments that are being recommended for use in the pilot and field test.

Contact Information

Jessica Stroop

Statistician, Project Manager

Bureau of Justice Statistics

Phone: 202-598-7610

Email: jessica.stroop@usdoj.gov

Allen Beck, Ph.D.

Statistician

Bureau of Justice Statistics

Phone: 202-616-3277

Email: allen.beck@usdoj.gov

1 Throughout this document, we refer to youth “assent” to include both assent by minors and consent by youth who have reached the age of majority.

9