Campus Climate Survey Memo to OMB

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Shelly Wilkie Martinez

Official of Statistical and Science Policy

Office of Management and Budget

Through: Lynn Murray

Clearance Officer

Justice Management Division

William J. Sabol

BJS Acting Director

From: Michael Planty, Lynn Langton, Jessica Stroop

Date: December 8, 2014

Re: BJS Request for OMB Clearance for Cognitive Testing of the Campus Climate Survey Validation Study under the BJS Generic Clearance Agreement (OMB Number 1121-0339)

The Bureau of Justice Statistics (BJS) is requesting clearance for cognitive testing of an instrument to capture data about rape and sexual assault as well as attitudes and perceptions of safety on campus among undergraduate university students. This instrumentation effort is part of BJS’s Campus Climate Survey Validation Study (CCSVS), a study that BJS is developing through a cooperative agreement with Research Triangle Institute (RTI) (Award 2011-NV-CX-K068) and with funding from the Office of Violence Against Women (OVW). The primary purpose of the CCSVS is to develop and test instrumentation and methods for conducting a survey of college and university students that measures the ‘climate’ of the school and student body. Climate is defined as a compilation of indicators related to the percentage of students experiencing rape and sexual assault victimization, victim reporting and interaction with school officials and law enforcement, perceptions of school policies and responsiveness, and bystander intervention. The results from the cognitive interviews will be used to inform the implementation of a pilot test to be conducted in the Spring of 2015 with a representative sample of students in approximately 10 to 15 schools to explore methodological issues related to data collection and content validity. Key outcomes and performance indicators for the pilot test will focus on response rates, item nonresponse, inter-item reliability, and use of incentives to gain cooperation, as well as across school variation in survey implementation and results.

The Campus Climate Survey Validation Study (CCSVS) is a component of a broader effort by BJS to improve estimation of the rate and prevalence of rape and sexual assault (RSA). At present, BJS is involved in a number of projects that are aimed at collecting estimates of RSA through different approaches and methodologies including a redesign of our victimization omnibus survey and also developing and piloting two additional alternatives to collection methodologies (Rape and Sexual Assault Pilot Test). The CCSVS survey instrument development and testing will help inform BJS efforts associated with the redesign of the National Crime Victimization Survey (NCVS) instrument, and improved survey methodologies, including the use of web-based self-administered survey modes for measuring sensitive topics related to criminal victimization and offending, and the development of innovative and cost-effective approaches to cognitively testing new survey instrument items. The CCSVS aims to fill important gaps in knowledge about the prevalence of RSA among a potentially high risk population, the impact of the context of the survey and use of behaviorally-specific questions on prevalence and incidence rates, and information on victim help-seeking behaviors and reasons for not reporting RSA to formal and informal networks and authority figures.

The CCSVS instrument contains items that align purposefully with the NCVS, specifically the reference period and the open ended narrative which will allow BJS to supplement the results obtained through this project with NCVS data, producing more valid and reliable estimates on RSA.

For this clearance, and prior to the administration of the full CCSVS pilot test, we plan to conduct in-person cognitive testing of the full CCSVS instrument with up to 60 college students, and also plan to test targeted sections of the instrument using crowdsourcing techniques (described in more detail below) with up to 600 online respondents. Together these cognitive testing tasks will require up to 115 burden hours and will help to identify issues with the wording or content of the instrument and to estimate the burden associated with its completion.

Once the instrument has been finalized through cognitive testing, a pilot test will be conducted to address issues related to the administration of the survey with a representative sample of students at universities, such as mode, response rates, number of follow-up contacts needed, and successful approaches for gaining university and student participation. OMB approval for the pilot test will be sought after completion of the cognitive testing in early 2015.

Description and Purpose of Overall Project

In January of 2014, the White House Task Force to Protect Students from Sexual Assault was established with goals of developing recommendations for how colleges and universities could better respond to reports of sexual assault and for increasing the transparency of enforcement efforts. As the principal statistical agency of the Department of Justice and the sponsor of the NCVS and other projects that seek to reliably measure rape and sexual assault, BJS agreed to develop and test a survey and methods that can be shared with schools and used to conduct reliable and valid climate surveys.

The White House Task Force has developed a "core" set of items for a climate survey, designed to capture key aspects of the problem of sexual assault for campuses using the best available and promising instruments from the research literature (https://www.notalone.gov/assets/ovw-climate-survey.pdf). The key conceptual areas that were mentioned in the instrument were: 1) general climate of the university, 2) perceptions of leadership, policies and reporting, 3) the prevalence of sexual violence, 4) the context around the incidents of sexual violence, 5) bystander confidence and readiness to help, 6) perceptions of sexual assault, 7) rape myth acceptance, and 8) prevalence of interpersonal violence. The proposed instrument to be cognitively tested for the CCSVS will validate the utility and reliability of items that reflect the conceptual areas outlined in the task force report, though the questions themselves differ from those originally developed by the White House Task Force. The current instrument being used for the CCSVS will allow us to be able to produce university-level key estimates for RSA, will not capture attempted incidents, and ask confirmation questions to detect false negatives and false positives. Also, rather than focus on ‘most serious’ incident, the CCSVS allows respondents to report detailed information on up to 5 incidents which research suggests will capture approximately 98% of all incidents experienced by victims. There are also items that capture information on the number of perpetrators, month in which the incident occurred, and the victim’s help-seeking and reporting behavior and experiences. It will also assess methodological issues for consideration in conducting a campus climate assessment, including administration and sampling considerations, and considerations related to the sensitive nature of the topic and the challenges of obtaining cooperation from a student population and providing incentives to survey respondents. The CCSVS pilot study will be used to answer key questions such as:

• How does the use and amount of incentives impact response rates?

• How does the length of the interview impact response rates?

• How can the length of the survey be reduced without reducing its utility?

• Do the survey measures produce variations in responses across universities that are instructive?

• Do the survey items produce estimates that have external validity?

• Do any of the survey items create undo stress or burden for respondents?

Design and CCSVS instrument

RTI and BJS began the development of the CCSVS instrument by examining items on the White House’s ‘toolkit’ that had been comprised of best practices from the body of academic literature that sought to measure elements of both RSA, but also climate perceptions of safety, harassment items, interpersonal violence, and bystander behaviors among university students. Many of these items were quickly identified as being too complicated, burdensome, or unclear, such that they had the potential to result in low response rates and in the collection of invalid data. The instrument was thus revised to focus on the elements that were of paramount interest but crafted using best practices from the research literature such that the likelihood of collecting high quality data would be improved. To the extent possible we modeled questions after prior studies in an effort to reduce burden on respondents and increase validity, particularly for respondents who may have experienced victimization. However, the CCSVS is also designed to measure perceptions and attitudes of RSA and safety on a campus, which will be the majority of the respondents.

For these reasons, the project team also engaged in a series of discussions with a variety of stakeholders and practitioners in the field to assist with the development of the instrument. Early and continuing discussions were held with OVW and other agencies and stakeholders to better understand the current state of research on RSA, understand what other collections were in development, and to focus and eliminate duplication of efforts wherever possible. The team also relied on the advice and guidance of a variety of experts and practitioners. Drafts of the instrument have also been provided to experts who have provided feedback. These conversations and meetings were instrumental in making determinations about the structure of the survey to ensure its utility for adequately measuring RSA among a specific high-risk population. Additionally, in an effort to assess if and how to include certain topics in the CCSVS instrument, the project team sought guidance on an as-needed basis from individuals identified as key researchers on a range of different types of related content areas, such as perpetration, bystander intervention, rape myths, stalking, and interpersonal violence. We also then reviewed our ongoing work on RSA with Westat to ensure that we are aligning the key concepts of RSA research informed by the White House areas of interest, but also in a way that is consistent with BJS practices and can be used to supplement the NCVS going forward as it pertains to measuring incidence and prevalence in a reliable fashion. However, incidence and prevalence of rape/sexual assault and help-seeking behaviors related to college campus resources are primary concerns unique to this study. There are also key mode differences that require different approaches to measurement (RDD, ACASI as opposed to web-based self-administered).

The resulting CCSVS instrument is designed to capture detailed essential information about RSA incidents that will be used to describe the scope of the issue among university students, and inform efforts at BJS to improve the measurement RSA and the NCVS. The cognitive interviews will address the structure and language of the current draft of the instrument, which will gather information related to these topics:

1. Demographics including gender identity and sexual orientation

2. Student perceptions of the extent to which campus police/security, faculty/staff and administration create a secure environment.

3. Whether undergraduate students experienced any unwanted sexual contact in the past 12 months

4. Details related to victimization incidents, including location, number of offenders, and use of alcohol or drugs

5. Whether victimization incident was or was not reported, to whom, and reasons for not reporting, as applicable

6. Whether undergraduate students perpetrated any unwanted sexual contact in the past 12 months

7. Student exposure to and perceptions of sexual assault reporting or prevention training

8. Perceptions of acceptance of sexual harassment or confidence with, and willingness to help through bystander intervention

Cognitive Interview Procedures

In this memo, BJS is seeking generic clearance specifically to cover cognitive testing activities. The cognitive testing being planned will utilize two approaches, 1) crowdsourcing and 2) in-person interviewing. Crowdsourcing will be implemented for the purpose of efficiently identifying as many obvious problems with key survey items as possible based on feedback from large numbers of respondents, whereas in-person cognitive interviewing will be used with a smaller number of respondents to obtain a more nuanced understanding of how they are conceptualizing and answering each survey question. All cognitive testing will take place in the Fall of 2014 and will be completed in sufficient time to inform the OMB generic clearance for the pilot test of the full instrument.

Crowdsourcing

RTI has significant expertise in crowdsourcing data collection, which has been developed through externally and internally funded projects over the past several years. Crowdsourcing data on these particular areas of sensitive data collection, however, have not been tried and the value of this approach is unknown. This opportunity presents an important methodological approach that could be useful for future studies. The first crowdsourcing technique will identify approximately 600 respondents with targeted specific demographic characteristics that include U.S. residency, English speaking ability, 18-25 years of age, and self-reported occupations of students. RTI began standardizing crowdsourcing strategies in an effort to employ best practices for capturing large amounts of data in short periods of time, and has developed an evidence-based methodological framework for crowdsourcing. RTI’s past experience with crowdsourcing has shown that some diversity is also captured through crowdsourcing, although it is not likely to yield a representative sample of respondents. However, representativeness is not an essential goal of the cognitive testing phase since all respondents are eligible and their perspectives will be relevant. The target number of 600 respondents was identified as a number that allows for some variation among respondents but also takes into consideration affordability and schedule constraints. RTI’s research has investigated and pilot tested crowdsourcing platforms such as Cint, Amazon Mechanical Turk, GigWalk, TryMyUI, Facebook, Twitter, and others (Murphy, 2013; Keating & Furberg, 2013; Keating, Rhodes, & Richards, 2013; Richards, Dean, & Cook, 2013; Sage, 2013).

RTI staff will recruit approximately 600 volunteers from the opt-in Cint web panel. Cint is an opinion hub that has access to 10 million panelists in 60 countries. The panel allows researchers to gain insights by targeting specific panelist demographics (e.g., race, age, gender) and characteristics (e.g., occupation). The Cint panelists are pre-registered panel members who are looking to complete small web-based surveys for minimal compensation. Cint will e-mail the invitation (which uses standard text) to participate in the survey only to panelists whose characteristics include: residence within the United States, English speaking, are 18 to 25 years of age, and have self-reported occupations as students. RTI will not have access to any information about the respondents who are invited (i.e., we will not receive any information about the sampling frame from Cint, either during sample selection or after panelists actually complete the survey). More information about Cint can be found at http://www.cint.com. Quite simply, Cint provides for efficient data collection with a motivated crowd who can supply the input and information needed for initial cognitive testing of the CCSVS instrument.

In an effort to limit respondent fatigue and burden, the crowdsourced respondents will receive a streamlined version of the CCS instrument that will take between 5 and 10 minutes to complete. Included will be the survey items that RTI and BJS deem most critical to the mission of the CCSVS. Next to each question that a respondent receives will be a box for him/her to add open-ended comments about the question, including any difficulty understanding specific terms or recommendations for improvement. In addition, for some survey questions, an additional comment box in which we have inserted specific probes for them to answer (e.g., definitions of certain terms, how respondents formulated their answers), will be included above the standard comment box. This will allow us to capture feedback from the participants about terminology or questions that we are particularly interested in wording correctly or as clearly as possible.

After the respondents complete the last survey question, they will be taken to a webpage that includes a list of national hotline/helpline telephone numbers the respondent can contact if they are interested in these resources. After clicking on this page, the respondent will be redirected to Cint’s platform, where they will receive their payment of $1 incentive paid through Cint’s payment system. Importantly, Cint staff cannot view panelists’ responses to survey questions because the survey website is entirely outside of Cint’s platform.

In-person Interviewing

The target population for the in-person cognitive interviewing is undergraduate college students living in North Carolina; Washington, D.C. and Portland, Oregon. The only eligibility criteria are that the student be an undergraduate, be 18 or older, and speak English. All in-person interviews will take place either at an RTI office or another private location agreed upon by the interviewer and the participant. Interviews will last approximately 1 hour and participants will receive $40 in appreciation for their time. In-person cognitive interviews will be conducted with up to 60 respondents. We feel $40 is a justifiable incentive amount as it is effective in attracting a wide diversity of respondents.

The recruitment procedures for the in-person cognitive interviewing are designed to ensure that participants include up to 30 female survivors of sexual assault (who have experienced sexual assault within the past 12 months), in addition to up to 30 other college students who have not necessarily experienced sexual assault.

RTI has prior experience recruiting quite a few victims of sexual assault using a similar approach for NISVS by communicating with local domestic violence shelters and by advertising in craigslist.com. Detailed procedures are presented below.

Sexual assault victims

To identify and recruit female sexual assault survivors, RTI staff will reach out to staff at crisis and women’s centers at universities in RTP, North Carolina; Washington, D.C.; and Portland, Oregon. When reaching out to staff at these centers, we will begin by explaining the purpose of the overall CCS and the cognitive interviewing, and explain that we would like them to inform women who have potentially been victimized within the past 12 months about the opportunity to participate in cognitive interviewing. We will ask these staff members to think about sexual assault survivors with whom they have worked and who they feel are ready to talk about their experiences and would be able to provide useful information about developing surveys on this topic. We will also ask campus staff whether there are private rooms at the center that might be suitable for holding the cognitive interviews. We will also confirm that staff at the crisis and women’s centers will be available and can be contacted by our respondents during the cognitive interviewing process, should they need assistance.

Once the center has indicated its willingness to assist with recruitment, we will ask campus staff to notify potentially eligible students about the cognitive interviews via word of mouth and recruitment flyers. The flyers, which will provide a phone number and email address for the volunteers interested in participating to contact the project, will not be posted on campus but rather handed out in person by crisis/women’s center staff.

Once a volunteer has contacted the project, a brief screener to obtain demographic information will be administered and a place and time to conduct the interview will be scheduled. Volunteers will be informed about the nature of the study during this phone screening. A screening log will be kept on the project share and include only first names. The only other identifying information will be a phone number where they can be reached for an appointment reminder call (if appointment is made far in advance) and to notify them of any last minute changes in appointment time or location. Volunteers do not have to provide a phone number if they do not want to.

RTI has prior experience recruiting victims of sexual assault using a similar approach for the National Intimate Partner and Sexual Violence Survey (NISVS). In advance of the NISVS pilot test and national implementation, a total of 18 victims of intimate partner violence and sexual violence were recruited. Recruitment was done via the distribution of flyers at domestic violence shelters, ads on craigslist.com, and ads in local newspapers.

Non-victims

To identify and recruit male and female non-victims for the in-person cognitive interviews, RTI staff will recruit using the local craigslist.com website. We will post craigslist.com ads in RTP, North Carolina; Washington, D.C.; and Portland, Oregon. When potential volunteers call, they will be informed about the purpose and content of the interviews, and they will be screened and scheduled using the same procedures that were used with victims. Because some of the respondents will be men, and our cognitive interviewers are women, we will schedule interviews with male subjects at RTI offices or other offices where we know others will be present.

Language

The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

The burden for this task consists of participants being screened for and subsequently participating in either crowdsourcing or in-person cognitive interviews. The burden associated with these activities is presented in the following table.

Minimum and Maximum Burden Associated with Planned CCSVS Cognitive Testing Activities

	Minimum # of Respondents	Average Administration Time (minutes)	Minimum Burden (hours)	Maximum # of Respondents	Average Administration Time (minutes)	Maximum Burden (hours)
Crowdsourcing Cognitive Interviewing	400	7.5	50.0	600	7.5	75.0
In-person Cognitive Interviewing	30	60	30.0	60	60	60.0
Total			80			115

Reporting

Upon completion of all cognitive testing, a draft cognitive interviewing report will be delivered to BJS that will include recommendations for final revisions to the survey. These recommendations will be based upon the findings of both crowdsourcing and in-person cognitive testing, and will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, average time needed to complete the survey instrument, and any issues with question comprehension or with questions causing undo distress. The report will also document changes made to the initial draft survey prior to the cognitive testing and all changes made between rounds. RTI will also include a draft of the instrument that is being recommended for use in the pilot test.

Protection of Human Subjects

There is some risk of emotional distress for the respondents given the sensitive nature of the topic, particularly since the questions are of a personal nature; however, appropriate safeguards are in place and the planned cognitive testing has been reviewed and approved by RTI’s Institutional Review Board (IRB), which has Federal-wide assurance.

Informed Consent, Data Confidentiality and Data Security

Crowdsourced Cognitive Interviewing

After clicking on the link displayed in the recruitment e-mail, the Cint panelist will be brought to the first page of the survey, which will be an online informed consent form. If the respondent wants to proceed, they will indicate that they consent and will then proceed into the survey.

In-person Cognitive Interviewing

At the beginning of the in-person cognitive interview appointment, the respondent will be handed a hard copy of the informed consent form and the interviewer will review the highlights of the informed consent form.as the respondent follows along. If the respondent wants to proceed with the cognitive interview, they will "X" the appropriate lines to participate, allow for recording and, if applicable, allow for observers. If a participant does not want to consent to audio recording or observation, no recording device will be used and observers will be asked to leave. The interviewer will sign the RTI copy of the consent form and leave a blank hard copy with the respondent. We are not asking respondents to sign the consent form as a measure to protect their confidentiality.

Data Confidentiality and Security

BJS’ pledge of confidentiality is based on its governing statutes Title 42 USC, Section 3735 and 3789g, which establish the allowable use of data collected by BJS. Under these sections, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 3735). BJS staff, other federal employees, and RTI staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 42 U.S.C. Sec. 3789g, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). To protect the identity of the respondents, no identifying information will be kept on the final data file. The survey will not be collecting the name of any of the respondents.

References:

Black, M.C., Basile, K.C., Breiding, M.J., Smith, S.G., Walters, M.L., Merrick, M.T., Chen, J., & Stevens, M.R. (2011). The National Intimate Partner and Sexual Violence Survey (NISVS): 2010 Summary Report. Atlanta, GA: National Center for Injury Prevention and Control, Centers for Disease Control and Prevention.

Keating, M. D., & Furberg, R. D. (2013, November). A methodological framework for

crowdsourcing in research. Presented at 2013 Federal Committee on Statistical

Methodology Research Conference, Washington, DC.

Keating, M. D., Rhodes, B. B., & Richards, A. K. (2013). Crowdsourcing: A flexible method for

innovation, data collection, and analysis in social science research. In Social media,

sociality, and survey research. (pp. 179–201). Hoboken, NJ: John Wiley & Sons, Inc.

Murphy, J. J. (2013, March). Ten things every survey researcher should know about Twitter.

Presented at 2013 Federal Computer Assisted Survey Information Collection (CASIC)

Workshops, Washington, DC.

Richards, A. K., Dean, E. F., & Cook, S. L. (2013). Collecting diary data on Twitter. In Social

Media, Sociality, and Survey Research. (pp. 203–230). Hoboken, NJ: John Wiley &

Sons, Inc.

Sage, A. J. (2013). The Facebook Platform and the Future of Social Research. In Social Media,

Sociality, and Survey Research. (pp. 87–106). Hoboken, NJ: John Wiley & Sons, Inc.

6