Memo to OMB re: ASPP Questionnaire Development

MEMORANDUM

MEMORANDUM TO: Shelly Wilkie Martinez

Acting Chief, Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Lynn Murray

Clearance Officer

Justice Management Division

William J. Sabol

Director

Bureau of Justice Statistics

FROM: Laura Maruschak

Statistician and ASPP Project Manager

Bureau of Justice Statistics

DATE: March 17, 2015

SUBJECT: BJS Request for OMB Clearance to Conduct Questionnaire Development Work for the Annual Surveys of Probation and Parole (ASPP) through the generic clearance agreement OMB Number 1121-0339

INTRODUCTION

As part of an ongoing effort by BJS to expand and improve its Community Corrections Statistics Program (CCSP), the Bureau of Justice Statistics (BJS) would like to engage in activities to inform the redesign of its ASPP questionnaires to a core set of data elements that can be used to produce key statistics to track changes in size and composition of the nation’s probation and parole populations. The redesigned ASPP questionnaires will meet BJS’s core mandates to collect, analyze, and disseminate information on the population of the nation’s criminal justice system at the federal, state and local levels as specified in its authorizing statute under 42 U.S.C. 3732 Sec. 302 (see Appendix A – Title 42 BJS Authorizing Legislation). This generic clearance seeks approval to conduct a series of discussions with directors (or their designees) of public parole and probation agencies and cognitive interviews with agency staff who would likely serve as respondents during the regular annual survey collection. This effort is part of a possible longer term assessment of the current ASPP questionnaires. Currently, the ASPP questionnaires have approval from OMB through 8/31/17. If any revisions to the questionnaires result from this effort prior to this expiration date, BJS will submit a modification for approval.

BACKGROUND

Authorized by the Omnibus Crime Control and Safe Streets Act of 1968, as amended (42 U.S.C. 3732), BJS has collected annual yearend counts and movements of adult community corrections populations for more than 30 years through its Annual Probation Survey and Annual Parole Survey. Periodically, BJS has made minor modifications to survey content, question wording, and instructions. However, during the past 3 decades, there have been significant changes in the structure and organization of probation and parole supervision within states, the way agencies define the populations under their supervision, and the supervision procedures used by the agencies. In response, BJS has begun developing supplemental sets of questions to address emerging issues that have been identified by data users and other stakeholders and is currently collecting data through the Census of Adult Supervising Probation Agencies to better understand the organization, structure and policies of adult supervising probation agencies in the U.S. However, given the broader changes experienced by supervising agencies, an examination of the current ASPP survey content and design is warranted. BJS is seeking approval under this generic clearance to conduct discussions with directors (or their designees) of probation and parole agencies and retrospective cognitive debriefings with ASPP respondents. The findings from these efforts will be used to identify the necessary changes to the survey instruments to enhance data quality, reduce burden, and increase efficiencies for the annual collections.

REQUEST FOR DEVELOPMENTAL WORK

Overview and objective – Under the generic clearance (OMB Number 1121-0339), BJS is seeking approval to conduct developmental research for the ASPP through a series of discussions with directors (or their designees) of public parole and probation agencies and cognitive debriefings with individuals that responded to the 2014 ASPP. The proposed investigation will achieve the following goals:

• Obtain feedback from agencies about the relevance of the current survey content and the impact of possible changes to the survey content, instructions, and definitions on their ability to report.

• Determine the capacity of respondents to answer the current survey questions, identify sections of the questionnaires that are unclear, and examine the causes of item nonresponse or incomplete information.

• Examine the factors associated with possible changes to the instruments that would positively or negatively affect burden.

This process will provide BJS with information needed to refine and improve the ASPP questionnaires and enable the bureau to continue to report, with a high level of accuracy, on changes in the size and composition of the community corrections populations in the United States. Through a cooperative agreement with BJS, project staff from Westat will serve as the data collection agents for the developmental work.

The developmental work proposed in this request will consist of two distinct, yet related, lines of inquiry; the activities will be performed concurrently:

• Discussions with up to 12 agency heads (or their designees). The discussions will focus on the relevance of specific survey items (e.g., details on entries by population type, number of probationers who had previously served a sentence to prison for the same offense); alignment between categories/definitions of population types used by the agency and those used on the surveys (e.g., active supervision versus inactive supervision); and capacity of the agency to report on other population characteristics (e.g., exits due to expiration versus early positive discharge, average number of cases per individual). (See Attachment A for a copy of the draft Agency Head Discussion Guides.)

• Cognitive debriefing interviews with up to 40 respondents to the 2014 ASPP. During the conversations, participants will be asked to describe the cognitive and data retrieval processes they used to respond to the most recent collection. Specifically, they will be asked to describe 1) how they interpreted survey definitions and instructions and how they applied their interpretations to their agency data; 2) any types of populations that are reported in more than one category or are not reported in any category; and 3) how possible changes to survey content, definitions, and instructions might affect their cognitive and data retrieval process. Participants also will be asked to report on their level of confidence in the data they provided on the 2014 ASPP and the burden associated with their participation, and to describe how these might be affected given changes to the survey. (See Attachment B for copies of the 2014 ASPP questionnaires. See Attachment C for a copy of the draft Respondent Discussion Guides.)

Sample – A purposive sample of agencies will be selected based on information from the 2014 ASPP data. Because there are only a relatively small number of parole agencies in the ASPP frame, the majority of the agencies asked to participate in the developmental work will be those that supervise probationers. Since this investigation will focus largely on the processes respondents use to answer questions about types of populations (e.g., counts), it will be necessary to include agencies that report supervising a broad array of population types. For example, to examine how agencies differentiate between active and inactive populations, or between absconders and those on warrant status, participants should be drawn from agencies that report each population type as well as those that reported only one population type. For illustrative purposes, as suggested by the content of the draft Probation Survey protocol, we will select agencies that supervise the following population types:

• Type of discharge

  • to incarceration to receive treatment and for “other” reasons (Q3b3 and Q3b4)¹

  • as absconders and to a warrant or detainer (Q3c and Q3d)

  • to another agency (Q3e)

  • for another reason (Q3g)

• Most serious offense

  • defined as “other traffic offense” (Q9g)

• Route of placement

  • placed through a suspended imposition of a sentence (Q14d)

• Status

  • active and inactive (Q15a and Q15d)

  • financial condition only (Q15c)

  • absconders and warrant or detainer (Q15e and Q15f)

  • out of jurisdiction (Q15g)

• Use of GPS

  • used as a tracking method (16)

In addition to asking about response processes used to report on population types, participants will be asked about their ability to report using a different basis or at a finer level of detail. Data from the 2014 ASPP will be used to identify agencies that can address the following issues.

• Basis for reported counts (Q5). We will select a number of agencies that have changed their count basis at least once in the past 3 survey cycles.

• Sub-dividing discharges for completion of sentence (Q3a). We will select participants that have reported low and high counts in this category.

Forty parole and probation agencies will be selected based on their responses to the 2014 ASPP questions. While the literature on optimal sample sizes for developmental work is not extensive, the research that has been conducted recommends that a reasonable minimum sample size for survey or scale development is about 30 participants from the population of interest. But if different groups exist within the population, the literature recommends about 12 participants per group.² The sample sizes for the ASPP developmental work are based on the latter recommendation. We examined the data from the 2013 Annual Probation Survey and found that a selection of just under 30 probation agencies would include approximately 10 agencies with each of the focal characteristics. We believe that expanding this set to 35 probation agencies and 5 parole agencies will achieve the target number of 12 participants per group.

Data Collection – Data collection for the developmental work is proposed to begin in April 2015 and end in July 2015. Westat will telephone heads of 12 agencies and invite each to schedule and participate in a one-on-one telephone discussion. Approximately 2 weeks prior to the scheduled call, Westat will send to each agency head a study packet that includes a cover letter explaining the purpose of the call, a copy of the 2014 ASPP questionnaire(s), and a list of topics that will be addressed during the call. (See Attachment D for copies of the draft Agency Head cover letters and topic lists.) A researcher from Westat will conduct and summarize each call using a standardized protocol to guide the discussion. Once all the calls are completed, a comprehensive report will be prepared based on the individual summaries; the report will include recommendations for revisions to the ASPP questionnaires.

Concurrent with the discussions with agency heads, Westat will telephone 2014 ASPP respondents and invite each to participate in an individual retrospective cognitive debriefing call. Prior to each schedule call, Westat will send a study packet that includes a cover letter explaining the purpose of the data collection and a copy of their agency’s 2014 completed survey. (See Attachment E for copies of the draft Respondent cover letters.) Both individual and comprehensive summaries will be written and recommended revisions will be described.

As ASPP collects information about organizations and not individuals, Westat’s Institutional Review Board determined that the activities associated with data collection are not considered human subjects research and obtaining informed consent is not required. However, all participants in the proposed development work will still be informed of the purpose of the collection, the voluntary nature of their involvement, and the time associated with participation. No incentives are planned.

The estimated maximum time to conduct the proposed developmental work is about 78 hours. The burden hour estimates are shown in Table 1.

• Identifying designated respondents (10.4 hours). All agencies will be asked to identify a staff member who will serve as respondent for either the discussion or cognitive debriefing. In some cases, this might be the agency head, but many times the agency head will designate this responsibility to a member of the agency staff. For the discussions, the agency head will be asked to consider who is most knowledgeable about the types of data requested when making the designation. For the cognitive debriefings, the person who served as the respondent to the 2014 ASPP will be selected. Agency heads will be asked to provide contact information for the respondent to reply to Westat via fax, telephone or email. This is estimated to take an average of 10 minutes per agency head. Two minutes, on average, is estimated for follow-up if the form is not returned by the due date. Westat will follow up with the agency by telephone to obtain the information.

• Reviewing the survey content, definitions/instructions, and participating in a discussion (18 hours). Agency heads (or their designees) will be asked to review materials in anticipation of the call with Westat. Based on the amount of information that will be examined, we anticipate each person will spend 30 minutes in preparation for the call (6 hours). In addition, we anticipate that each call will last 1 hour (12 hours) in order to review all of the concepts and definitions addressed in the protocol.

• Reviewing the 2014 questionnaire and data submitted by the agency, and participating in a cognitive debriefing (50 hours). The burden associated with reviewing the 2014 ASPP questionnaires and data is approximately 15 minutes per response (10 hours). We anticipate that each cognitive debriefing will last 1 hour (40 hours).

Table 1. Estimated burden

Reporting and data retention – According to Title 42 U.S.C. 3735, the information gathered in this data collection shall be used only for statistical or research purposes, and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes. The data collected from the agencies represent characteristics of the agencies. No individually identifiable information will be collected.

Data will be housed at Westat in Rockville, MD on Westat’s secure computer system which is password protected. Data will be protected by access privileges assigned by the appropriate system administrator. All systems are backed up on a regular basis and are kept in a secure storage facility. Westat will transfer the data to BJS using Secure File Transfer Protocol (FTP). All files copied to Westat's Secure Transfer Site are securely stored and transferred using Federal Information Processing Standards (FIPS) 140-2 validated Advanced Encryption Standard (AES) encryption, recognized as the U.S. Federal government encryption standard. Westat will create a personal account for BJS with login ID and password.  Westat will retain the data on site until the end of the grant, at which time the data will be destroyed.

Once the data are made available to BJS to securely download, they will be physically stored at BJS which is located in a secure building that includes the DOJ’s, Office of Justice Programs (OJP) offices. All OJP servers are backed up on a regular basis and stored in a secure location, specifically a locked room with access limited to only information technology personnel from OJP and requiring a badge swipe to enter. Technical control of the test data will be maintained through a system of firewalls and protections. Specifically, the data will be stored on a standard secure server behind the DOJ’s firewall and will be protected by access privileges assigned by the BJS information technology specialist.

CONTACT INFORMATION

Questions regarding any aspect of this project can be directed to:

Laura Maruschak

Statistician and ASPP Project Manager

Bureau of Justice Statistics

801 7^th St, NW

Washington, DC 20531

Office phone: (202) 307-5986

E-mail: Laura.Maruschak@usdoj.gov

ATTACHMENTS

See the list below for a complete list of attachments.

Attachment A – Agency Head Discussion Guides

A1. Parole Agency Head Discussion Guide

A2. Probation Agency Head Discussion Guide

Attachment B – 2014 ASPP Questionnaires

B1. 2014 Annual Parole Survey

B2. 2014 Annual Probation Survey

Attachment C – Respondent Discussion Guides

C1. Parole Respondent Discussion Guide

C2. Probation Respondent Discussion Guide

Attachment D – Agency Head Cover Letters and Topic Lists

Attachment E – Respondent Cover Letters

1 Question numbers refer to items from the 2014 Annual Probation Survey (see Attachment B.)

2 Johanson, George A. & Brooks, G.P. (2010). “Initial Scale Development: Sample Size for Pilot Studies.” Educational and Psychological Measurement, 70(3): 394-400. SAGE