Revised OMB Memo for Cognitive Testing for NSYC-3

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Shelly Wilkie Martinez

Office of Statistical and Science Policy

Office of Management and Budget

Through: Lynn Murray

Department Clearance Officer

Justice Management Division

Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

From: Jessica Stroop

Statistician, Project Manager

Allen Beck, Ph.D.

Statistician

Bureau of Justice Statistics

Date: July 6, 2016

Re: BJS Request for OMB Clearance for Cognitive Testing for the National Survey of Youth in Custody under the BJS Generic Clearance Agreement

(OMB# 1121-0339)

The Bureau of Justice Statistics (BJS) seeks generic clearance approval for cognitive interviewing tasks for activities related to the National Survey of Youth in Custody (NSYC) program. This package supports a collection mandated by Congress (Public Law 108-79; see Attachment 1). BJS is submitting to OMB for approval of developmental materials associated with NSYC-3. This initial clearance will be followed by a request to conduct a pilot test later in 2016.

Purpose of the Research

On September 4, 2003, the Prison Rape Elimination Act of 2003 (PREA or the Act) was signed by President George W. Bush (Public Law 108-79; see Attachment 1). The Act requires BJS to “carry out, for each calendar year, a comprehensive statistical review and analysis of the incidence and effects of prison rape.” The Act further instructs BJS to collect survey data: “…the Bureau shall…use surveys and other statistical studies of current and former inmates…” The law was passed in part to overcome a shortage of available research on the incidence and prevalence of sexual violence within correctional facilities.

To implement the Act, BJS has developed the National Prison Rape Statistics Program (NPRS), which includes four separate data collection efforts: the Survey on Sexual Violence (SSV), the National Inmate Survey (NIS), the National Survey of Youth in Custody (NSYC), and the National Former Prisoner Survey (NFPS).

These collections are independent and, while not directly comparable, they provide various measures of the prevalence and characteristics of sexual assault in correctional facilities. The SSV series collects information about incidents of sexual violence reported to and investigated by adult and juvenile correctional authorities, as well as characteristics of substantiated incidents. The NIS collects allegations of sexual assault self-reported by adult and juvenile inmates in adult correctional facilities. NSYC gathers self-reported sexual assault data from juveniles in juvenile correctional facilities. The NFPS measures allegations of sexual assault experienced during the respondent’s last incarceration, as reported by former inmates on active supervision.

Between June 2008 and April 2009, BJS completed the first NSYC (NSYC-1) of 166 state-owned or operated facilities and 29 locally or privately operated facilities, and between February 2012 and September 2012, BJS completed the second NSYC (NSYC-2) of 273 state-owned or operated facilities and 53 locally or privately operated facilities. Both rounds of the survey were conducted by Westat, and included at least one facility in all 50 states and the District of Columbia. In both rounds, youth completed the survey using an audio computer-assisted self-interview (ACASI), whereby they heard questions and instructions via headphones and responded via a touch-screen. For NSYC-1, the sample represented 26,551 adjudicated youth held nationwide, with a total of 10,263 youth participating in the survey. For NSYC-2, the sample represented 22,944 adjudicated youth, with a total of 9,703 youth participating in the survey.

BJS published survey findings from NSYC-2 in June 2013, including the following:

• An estimated 9.5 percent of youth in state juvenile facilities and privately operated facilities (representing 1,720 youth nationwide) reported one or more incidents of sexual victimization by another youth or facility staff in the past 12 months or since admission, if less than 12 months.

• About 2.5 percent of youth (450 nationwide) reported an incident involving another youth, and 7.7 percent (1,390) reported an incident involving facility staff.

• Approximately 3.5 percent of youth reported having sex or other sexual contact with facility staff as a result of force, threats, or other explicit form of coercion; 4.7 percent of youth reported sexual contact with facility staff without any type of force.

This submission is to seek clearance for the cognitive testing of the instrument prepared for the third NSYC (NSYC-3). This request falls under a cooperative agreement BJS has with Westat to collect data for NSYC-2, (award 2009-RP-BX-K001) and as one of the remaining tasks under the agreement, Westat is developing and testing new items for inclusion in NSYC-3. Cognitive testing will enable us to better understand the way youth are interpreting the questions and response options, and to evaluate the appropriateness of the questions for youth in correctional facilities.

The cognitive testing activities consist of two parts. Part One will include 16 to 20 interviews with youth in juvenile residential correctional facilities to test new items being considered for NSYC-3. We will conduct the interviews at facilities that provide consent in loco parentis (ILP) for the minors in their custody. Youth invited to participate in the survey will have the choice to refuse or comply with the request.¹ Youth assent materials (for those ages 12 to 17) and consent materials (for those ages 18 or older) (Attachment 3) are attached.

Part Two will include up to 20 interviews with youth and young adults (ages 12 to 21) who are not living in correctional facilities in order to test new items regarding sexual orientation, gender identity, and disability. The youth and young adults recruited for this testing will be those who identify themselves as lesbian, gay, bisexual or transgender (LGBT), are unsure of their sexual orientation or gender identity, or who have a close family member (such as a parent or sibling) or close friend who is LGBT.

The NSYC-3 cognitive testing design and procedures are under review by the Westat IRB which has federal wide assurance. For NSYC-2, the IRB determined that the study involved minimal risks to research subjects given the safeguards built into the study procedures. A copy of the NSYC-2 approval notice is attached (Attachment 4).

PART ONE: Cognitive Testing with Youth in Juvenile Residential Correctional Facilities

Cognitive Testing Overview

We are seeking approval to conduct cognitive interviews with 16 to 20 youth in correctional facilities who will test new items being considered for NSYC-3. The instrument and testing protocols are presented in Attachment 2. There are a total of 70 questions being tested, some with multiple sub-items. We anticipate in a 60 minute interview we can gather feedback on approximately 35 questions. We have split the questions by topic grouping into two versions, such that each version will be tested with 8-10 youth. We have pre-tested each version of the protocol and confirmed that each can be completed within 60 minutes. Items being tested are those that are new content and have not already been tested with other youth populations in other studies. These include:

• Version 1

  • Facility living conditions (9 questions)

  • Staff treatment (6 questions)

  • Misconduct/victimization (1 question)

  • Education on PREA (3 questions/8 items)

  • Grooming behaviors (4 questions)

  • Mental health (4 questions/15 items)

  • Youth incident details (7 questions/18 items)

  • Disability items (if time permits) (3 questions)

  • Sexual orientation/gender identity (if time permits) (4 questions)

• Version 2

  • Legal counsel & institutional experiences (4 questions)

  • Solitary confinement (5 questions/9 items)

  • Youth/gang involvement (4 questions)

  • Drug use (1 question, 2 items)

  • Staff incident details (9 questions/25 items)

  • History of victimization (3 questions)

  • Treatment program and services (if time permits) (6 questions, 11 items)

  • Sexual orientation/gender identity (if time permits) (4 questions)

For the more sensitive items being tested (regarding youth incident details, staff incident details, grooming behaviors, drug use, and history of victimization), we will not be asking respondents to answer the survey questions, rather we will ask the youth to review the question while the interviewer reads it aloud, and then the interviewer will immediately ask the respondent for their interpretation of what the question is asking and any terms that may be potentially confusing or misunderstood. Likewise, if particular items do not apply to youth (such as receiving counseling services or being placed in solitary confinement), respondents will only be asked to provide their interpretation of the question, rather than answering it.

Goals of the Cognitive Testing

The overall goal of the interviews is to test the respondents’ general comprehension of the survey questions. This will be accomplished through the use of traditional cognitive interview probing techniques. Interviewers will hand a paper version of the ACASI items to the respondent and will read the questions aloud as the respondent reads along. For most items (other than those mentioned above), respondents will answer the question and then will be asked to comment on their interpretation of each question being tested. For the more sensitive items mentioned above, respondents will be asked only to comment on the question, not to answer it.

Eligibility and Selection of Youth for Cognitive Interviews

We will identify two juvenile facilities in which administrators will permit cognitive interviews with their residents. See Attachment 6 for materials that will be used to solicit cooperation from state agencies and specific facilities within those states. A total of 8 to 10 adjudicated youth will be selected by each facility. Some of the youth may be adults, however, we expect some to be minors. These facilities (or their state) will provide consent in loco parentis to interview the youth. Only those youth who are adult, or for whom the facility provides consent, will be eligible to participate in the cognitive testing.

Youth Assent for Cognitive Interviews

Prior to the start of an interview, the youth will be informed about the purpose of NSYC and the specific purpose of the cognitive interviews. All mandatory reporting and counseling procedures specified for the national study will be communicated and followed as well. The interviews will not be recorded and youth will not receive any incentive for participating. The youth assent form is provided in Attachment 3. Interviews will only be conducted if the youth assents.

Cognitive Interview Procedures

All interviews will be conducted by a senior member of the Westat project staff who has been specifically trained for this study. Westat staff will never ask for the name of any youth. During the interviews, youth will be asked to comment on their interpretation of the draft questions. The youth will be instructed to read along as the interviewer reads each question aloud. After each question, the interviewer will invite the youth to comment on the question intent or wording. After specific questions, as defined in the cognitive interview protocol (see Attachment 2), the interviewer will probe for the youth’s interpretation of particular terms or phrases or ask the youth to discuss the appropriateness of the question.

PART TWO: Cognitive Testing with LGBT Youth Not in Juvenile Residential Correctional Facilities

Cognitive Testing Overview

We are seeking approval to conduct cognitive interviews with 20 youth and young adults (ages 12 to 21) who will test sexual orientation, gender identity, and disability items being considered for NSYC-3. The instrument and testing protocol are presented in Attachment 2. There are a total of 7 questions being tested. We anticipate the interview will last 30 minutes.

Gender Identity

Current best practice for measuring gender identity in surveys is to use a 2-step process. For NSYC-3, we will be testing two questions. Both items have been approved by OMB for the National Crime Victimization Survey (NCVS) for respondents age 16 and older. We will be testing the addition of a response option of “not sure” for adolescents who may feel conflicted about their gender identity, and will be probing to explore the extent to which youth and young adults understand the term “transgender,” and whether the proposed definition is easily understood.

GI1. What sex were you assigned at birth, on your original birth certificate? (NCVS)

a. Male

b. Female

GI2. Do you currently describe yourself as male, female or transgender? (NCVS)

[Click here for a definition of transgender. “Some people describe themselves as transgender when they experience a different gender identity from their sex at birth. For example, a person born into a male body, but who feels female or lives as a woman.]

a. Male

b. Female

c. Transgender

d. None of these

e. Not sure

Sexual Orientation

Few national surveys of adolescents ask about their sexual orientation. One exception is the Youth Risk Behavior Survey, which added a question about sexual orientation to the 2015 high school survey. We will be testing the OMB-approved NCVS questions on sexual orientation with the addition of “not sure” as appropriate for adolescents (particularly younger adolescents) who may still be questioning their sexual orientation. In addition, we will be testing the addition of questions about sexual attraction which may help identify youth who have gay, lesbian, or bisexual feelings but no sexual experience and do not yet identify with the gay/straight/bisexual labels or who select “something else.” The Williams Institute recommends including sexual orientation questions for adolescents that are based on attraction rather than behavior. Research has found that most children are aware of sexual attraction by age 10, well before most become sexually active.² Question 4 is designed to assess attraction rather than behavior for youth who may not be sexually active.³ This additional question may help measure same-sex attraction more accurately than question 3 alone.

The BJS study team will work with the Federal Interagency Working Group on Sexual Orientation and Gender Identity to evaluate the functioning of the question 3 “not sure” response category and question 4.

GI3. Which of the following best represents how you think of yourself? (NCVS)

(Males) (Females)

a. Gay⁴ a. Lesbian or gay

b. Straight, that is, not gay b. Straight, that is, not lesbian or gay

c. Bisexual c. Bisexual

d. Something else d. Something else

e. Not sure e. Not sure

GI4. People are different in their sexual attraction to other people. Which best describes your feelings?

Only attracted to males 1

Mostly attracted to males 2

Equally attracted to females and males 3

Mostly attracted to females 4

Only attracted to females 5

Not sure 6

Disability Items

Most national surveys identify youth with disabilities either through a parent report (or report of another adult in the household) (National Health Interview Survey, American Community Survey, American Housing Survey, Current Population Survey) or by school report. Consequently while many of the measures discussed below are included on national surveys, many are collected from the parent or other adult who knows the child. Cognitive testing will help shed light on whether these questions can reasonably be understood and answered by youth. We will ask all respondents to only review the questions, rather than answer them. This will allow us to collect feedback on all of the items, whether or not they apply to the particular respondent. The BJS study team will consult with National Health Interview Survey staff at the National Center for Health Statistics regarding the testing of these items.

DI1. Because of a physical, mental or emotional problem, do you have serious difficulty concentrating, remembering, or making decisions?

YES 1

NO 2

DK/REF

DI2. Is the difficulty you experience because of a physical problem?

YES 1

NO 2

DK/REF

DI3. Is the difficulty you experience because of a mental or emotional problem?

YES 1

NO 2

DK/REF

Recruitment and Selection of Youth for Non-Facility Cognitive Interviews

Westat will use two primary methods to recruit youth/young adults for these interviews. First, we will advertise internally to Westat staff (over 2,000 employees) to encourage those who have LGBT family members or friends to contact Westat regarding the study. Secondly, we will reach out to local LGBT organizations, such as The DC Center, Youth Pride Alliance, and the Wanda Alson Foundation and ask them to circulate flyers and advertising content to their constituents. Advertising language is provided in Attachment 5. All interviews will take place at Westat’s facilities in Rockville, MD.

Youth and young adults interested in participating will complete a screener questionnaire by calling Westat’s 800 recruitment number. The screener will determine if the potential respondent considers himself or herself to be LGBT, or if they have a close relative (such as a parent, sibling, or grandparent) who is LGBT, or a close friend. This will allow the questions to be tested either directly with LGBT youth, or for indirect probes to be asked for the youth’s interpretation of how the family member or relative might answer the questions.

Parental Consent for Cognitive Interviews

We will obtain written parental consent for all non-facility youth who are under age 18. A copy of the parental consent form is found in Attachment 3. Youth under age 12 are not eligible for this study.

Youth Assent for Cognitive Interviews

Prior to the start of an interview, the youth will be informed about the purpose of NSYC and the specific purpose of the cognitive interviews. All mandatory reporting and counseling procedures specified for the national study will be communicated and followed as well. Non-facility participants will receive $25 for participating. Interviews will only be conducted if parental consent and youth assent are provided for those under age 18, and if consent is provided for those ages 18 or older. Youth under age 12 are not eligible for this study. Interviews will be recorded, but if the parent or youth refuses the recording, the interview will proceed with manual note-taking. All consent and assent forms are provided in Attachment 3.

Cognitive interview procedures

All interviews will be done by a member of the Westat project staff. During the interviews, youth will be asked to comment on their interpretation of the draft questions, and in some cases provide responses to further the discussion. The youth will be instructed to read along as the interviewer reads each question aloud. After each question or set of questions the interviewer will invite the youth to comment on the question intent or wording. After specific questions, as defined in the cognitive interview protocol, the interviewer will probe for the youth’s interpretation of particular terms or phrases or ask the youth to discuss the appropriateness of the question. All participants will be provided a list of resources for youth and LGBT issues on the back of the assent and consent forms.

PARTS ONE AND TWO: Considerations and assurances for all cognitive testing activities

Protection of Human Subjects

There is some risk of emotional distress for the facility youth respondents, given the sensitive nature of the topic of sexual victimization, particularly since some of the questions are explicit and of a personal nature. However, since respondents are not being asked to answer the sensitive questions about themselves, but are only being asked to comment on their comprehension of the items, we believe the risk will be minimal. Likewise, for the non-facility youth, we believe the risk will be minimal in asking youth to comment on their gender identification and sexual orientation, and to comment on disability questions.

The Interviews will be stopped if requested by the respondent or if the respondent shows signs of emotional distress. All respondents will be provided with an opportunity to speak with a counselor following the interview. Facility youth will be provided with an onsite resource if they wish to speak to a counselor, and we will also arrange for an offsite counselor for those who wish to speak to someone outside the facility. Non-facility participants will be provided with a list of resources targeted for youth and LGBT youth.

Language

All cognitive interviews will be conducted in English. Spanish language versions of the instruments will be developed and submitted to OMB for approval before being tested at a later time.

Burden Hours for Cognitive Testing

The burden for this task consists of facilities arranging for the youth to participate in the interviews, and the youth participating in cognitive interviews. The burden associated with these activities is presented in the following table.

We request a total of 36 hours (state and facility liaisons 6 + facility youth 20 + non-facility youth 10). This estimate is based on our experience with NSYC-1 and NSYC-2. As per Attachment 6, we anticipate that state liaisons will contact 1-2 facility superintendents/administrators to determine willingness to participate; provide authorization for Westat to conduct the research; communicate informed consent and mandatory reporting procedures to facility staff; specify security clearance requirements for Westat staff; and identify external counseling resources for youth who request such information.

We anticipate that facility staff will be engaged in the following activities: arranging for the data collection visit; identifying and securing participation of 8-10 youth; preparing appropriate interviewing space, obtaining consent for youth through in loco parentis; arranging for counseling services; defining mandatory reporting procedures; and escorting youth to and from interviews. The total estimated staff burden for these activities is 6 hours.

Expected burden placed on youth for this data collection averages 1 hour per respondent for the facility interviews, and 30 minutes per respondent for the non-facility interviews. We estimate between 16 to 20 facility youth will participate, resulting in a maximum estimated facility youth burden of 20 hours. We estimate that 10 to 20 non-facility youth will participate, resulting in a maximum non-facility youth burden of 10 hours.

Minimum and Maximum Burden Associated with

Planned NSYC Cognitive Testing Activities

	Maximum # of Respondents	Average Administration Time (minutes)	Maximum Burden (hours)
Facility communications and logistics	2	120	6
In-person Cognitive Interviewing at Facilities	20	60	20.0
In-person Cognitive Interviewing at Westat	20	30	10.0
TOTAL			36.0

The number of interviews with youth in facilities is based on the desire to complete 8-10 interviews for each of the two sets of questions. This should be sufficient to detect any important issues regarding the questions in each group. This will also permit the sample to be spread across the 12-21 age range, as well as across both male and female youth.

The number of interviews outside the facility is based on the desire to interview youth across a wide range of ages (12-21). We anticipate that it may be hard to find youth who self-identify in a non-traditional gender identity category. If this is the case, the sample size allows us to recruit youth who know other youth who identify in a non-traditional category to fully test the questions.

Costs to the Federal Government

The total cost of conducting the cognitive interviews, including incentives for the in-person interviews in Part Two, will be approximately $42,100 under the cooperative agreement with Westat (Award 2009-RP-BX-K001) for the National Survey of Youth in Custody (NSYC-2). This excludes the cost of BJS staff working on the project team.

Use of Information Technology to Reduce Burden

The cognitive testing study will utilize technology to facilitate recruitment and the scheduling process to reduce participant burden and control study costs. Where feasible and appropriate for Parts One and Two, recruitment efforts will use email communications when possible, participants increasingly prefer to communicate via email so they can respond when it is convenient. Using email for recruitment and scheduling can help to reduce participant burden and save time and money that would otherwise be spent conducting telephone calls, leaving voice messages and making call-backs.

Data Confidentiality and Security

BJS’s pledge of confidentiality is based on its governing statutes Title 42 USC, Section 3735 and 3789g, which establish the allowable use of data collected by BJS. Under these sections, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 3735). BJS staff, other federal employees, and Westat staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 42 U.S.C. Sec. 3789g, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). To protect the identity of the respondents, no identifying information will be kept on the final data file. The survey will not be collecting the name of any of the respondents.

Data Analysis

Cognitive interviewers will summarize the findings from each cognitive interview. Interviewers will prepare summary findings on each completed interview based on the complete questionnaire, recording of interview, and notes taken during the interview. The summaries will be analyzed to help identify common themes organized by overall questionnaire issues, individual questionnaire items and sections, and participants’ overall reactions to the questionnaire.

The cognitive interviewing analysis will assess and identify problems such as comprehension issues and difficulties understanding the tasks. These issues will be assessed qualitatively, based on the interviewers’ assessments of their own experience.

Upon completion of all the cognitive testing, a draft cognitive interviewing report will be delivered to BJS that will include recommendations for the final revision to the survey for the field test. These recommendations will be based upon the findings of both parts one and two of the cognitive testing, and will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, average time needed to complete the survey instruments, and any issues with question comprehension. The report will also document changes made to the initial draft NSYC-3 survey instruments that are being recommended for use in the pilot and field test.

1 Throughout this document, we refer to youth “assent” to include both assent by minors and consent by youth who have reached the age of majority.

2 Wilber, S. (2015). Lesbian, Gay, Bisexual and Transgender Youth in the Juvenile Justice System. Baltimore, MD: Annie E. Casey Foundation.

3Austin, S.B., Conron, K.J., Patel, A., Freedner, N. Making Sense of Sexual Orientation Measures: Findings from a Cognitive Processing Study with Adolescents on Health Survey Questions. Journal of LGBT Health Research 
Vol. 3, Iss. 1, 2007.

4 NCVS is conducting an experiment to test order of response options, for whether straight or gay should come first. We may ultimately change the order of our response options for NSYC-3 depending on the outcome of this research.

5