1121-006_supp_statement_B_final_Revised_passback_final_8-17_11

B. Statistical Methods

1. Universe and Respondent Selection

This collection involves the entire universe of state, federal and locally administered parole and probation departments. For probation, these include 33 central state reporters, the federal system, the District of Columbia, and 431 separate city, county, or court reporters. For parole, these include 50 central state reporters, the federal system, the District of Columbia, 1 city agency, the state agency in Pennsylvania that reports the county data, and the California Department of Rehabilitation and Corrections, Division of Juvenile Justice (formerly the California Youth Authority). In 2009, the response rate for the Annual Probation Survey was 99.8% (one refusal form a locality) and the response rate for the Annual Parole Survey was 100%. We expect similar rates in 2011, 2012, and 2013.

Respondents	Number
Probation:	466
Central State	33
Federal System	1
District of Columbia	1
Local	431
Parole:	55
Central State	50
Federal System	1
District of Columbia	1
Local	1
PA county data	1
CA DRC, DJJ	1

To assess the population universe of probation agencies is complete, BJS and the Census Bureau have used contacts with respondents that occur during the data collection process to determine whether changes in the probation population under supervision, including unexplained changes which occur from the end of one year to the beginning of the next, as well as growth or decline in the probation population exceeding plus or minus 10%, represent changes in probation agency’s responsibility. If this occurs for a local agency, the respondent is asked if their agency has taken responsibility for probationers that were previously being supervised by another agency, or if another agency has taken responsibility for a portion of the probationers for which the respondent’s agency previously had provided supervision services, as appropriate. Respondents are asked for information about any new local agencies which would allow BJS to begin the process of contacting and enlisting such agencies, as necessary.

Such methods also, for example, have helped to alert BJS to changes in state probation supervision authority in Georgia in the late 1990’s, when the supervision of misdemeanors was turned over to local authorities. As a consequence, the Georgia Supreme Court was included as a respondent for the Annual Probation Survey to obtain counts of adult misdemeanant probationers supervised by private probation agencies for local courts.

As discussed in section A, item 2, “Needs and Uses”, as part of the 2012 CAPSA project, BJS has been working with Westat, Inc. researching available data sources which might provide more comprehensive information on the location of adult probation supervision agencies, including those whose probation agencies that may currently not be reported to BJS as part of the Annual Probation Survey whether through a central state reporter or by direct response to BJS. The frame of probation supervising agencies and offices that will be developed through the CAPSA project will be used to systematically assess coverage of the population universe for the Annual Probation Survey and potentially reduce coverage error.

Ongoing monitoring of the reporting of parole populations for accurate reporting, similar to the monitoring of probation populations, has not yielded information about changes in parole agencies, as parole is generally a state function. The 2006 Census of Adult Parole Supervising Agencies (1121-0169) which took an in depth look at the organization, supervision and staffing of parole supervising agencies served to confirm that the universe of agencies which had been included in the Annual Parole Survey is complete.

2. Procedures for Collecting Information

Starting with the 2011 reference year, BJS plans to emphasize the web as the primary mode of data collection, with paper forms being sent to respondents upon request and those probation agencies that submitted a 2010 Annual Probation Survey short form (CJ-8A) by means other than the web (see section A, item 3, “Use of Information Technology” for more information). This is not a drastic change, as nearly 80% of parole (CJ-7) respondents, and approximately 44% of probation long form (CJ-8) respondents already use the web.

All respondents will receive a survey packet containing a letter requesting that they complete the survey (see Attachment 10) explaining the importance of the survey, stating that the survey is voluntary, making it clear that no other sources are available to provide all of the data requested in the surveys, and for respondents that respond to multiple BJS corrections surveys, acknowledging that BJS is aware of the multiple survey requests and both thanking them and informing them how important their participation is. The survey packet will also include a copy of the bulletin presenting the prior year’s probation and parole data (see Attachment 4), and a web flyer to reinforce the request to submit data via the web and will provide respondents with the website address (see Attachment 11). (Currently, the website is hosted by the U.S. Bureau of the Census, our collection agent: http://harvester.census.gov/aspp/.) The letter sent to CJ-8A respondents will also include a paper form and the web flyer (see Attachments 8 and 11). Each respondent will be provided with an individual user ID and password to enter the website and can view and print their data, but will not have access to another agency’s information. All respondents will also receive a second flyer inviting them to attend the BJS workshop to be held at the American Probation and Parole Association (APPA) conference (see Attachment 12). In an effort to maintain a current list of centralized reporting agents, the name of the respondent and address mailing list is continuously maintained throughout the year as respondents provide information through the P&P or other BJS Corrections Statistics collection.

Respondents will be asked to submit their data by the due date indicated on the web and paper forms. BJS’ data collection agent will receive surveys and send reminders until the last 4-5 respondents remain; these are often the same agencies from year to year due to information system issues or the agency being short-staffed. By this time, BJS is often already working with the data to produce a report, thus, it increases efficiency to have BJS handle the remainder of the data collection.

After a majority of respondents have submitted data, preliminary analysis will begin. BJS staff check the data for out-of-range values, missing data, and other types of responses that generate logical inconsistencies. These preliminary analyses are undertaken while data collection is still in progress in order to provide time for making callbacks to clarify data. As necessary, data from respondents are aggregated to the state, regional, and national totals, with the goal of producing distributions for each variable.

Follow-up efforts are conducted throughout the data collection period to either resolve discrepancies or to find out more about unreported quantities. If key quantities are missing, such as the beginning or yearend populations or the number of entries to or exits from supervision, as well as other quantities, such as the number of men or women or the type of offense, the data collection agent or BJS contacts those respondents to determine whether or not they can easily provide estimates of those unreported quantities to maximize response. For example, if a respondent does not provide the yearend probation population, which is the most important data element and usually respondents can provide this without follow-up, then the data collection agent/BJS finds out from the respondent about how much the probation population in their agency changed (i.e., percent change) during the year. BJS then uses the method of ratio estimation by applying the change in the population to the known quantity (i.e., the beginning of year probation population) to estimate the yearend probation population, and reports it as the agency’s estimate after discussing this with the respondent. For missing data elements that describe the characteristics of the population, such as the number of men and women on probation or the type of offense of offenders, if the agency was able to report counts in a recent, previous collection cycle and their population has not changed significantly since that time, then the data collection agent/BJS asks the respondent if it is reasonable to use ratio estimation to estimate the distribution for the current reporting year, and reports those estimates as the agency’s estimates after receiving approval from the agency.

If a respondent is unable to provide the number of entries onto probation (or parole) during the year then they will be asked if their information system tracks the number that entered probation weekly or monthly. Depending on how their information systems or records track that information, the data collection agent/BJS will work with the respondent to estimate an annual count based on an average weekly or monthly count, and this estimate will be reported as the agency’s estimate after discussing this with the respondent. Usually the data collection agent/BJS is able to work with the respondents to generate an estimate of their beginning or yearend population, but this is not always the case for entries and exits, which are the other two key quantities. If respondents are not able to provide any information to estimate entries and exits, BJS uses ratio adjustment methods to estimate values for these particular quantities in order to generate state- or national-level totals of movements onto and off of community supervision, along with the beginning and yearend populations. (BJS publishes both reported and imputed entries and exits at the state-level and each is labeled accordingly; see Attachment 4, appendix tables 2 and 12.) The ratio adjustments are done one of two ways; within agency using reported data from the prior year or within categories of agencies that are similar in size or geographic location. The decision depends on the availability of data. Within categories of comparable agencies, the data from those that reported complete data in either the current or prior year’s collections are used to estimate quantities for comparable agencies that did not provide data on entries and exits from community supervision.

After all follow-up efforts, data cleaning, estimation, and analysis are completed, the report will be written and the data will be released to the public less than a year after they are collected. Not only does the data result in a published report, but the data will be made available through a Department of Justice press release and additional detailed tables, all of which be posted on the BJS.

3. Methods to Maximize Response

BJS believes that the key to maximizing response rates is engaging respondents in the data collection process, providing them with useful information, analyzing response patterns to determine the most effective methods for contacting respondents, and making it easy for them to participate. To draw attention to the P&P collection in advance of the formal request to participate, respondents will be e-mailed or faxed a flyer on December 1, 2011 to notify them of the upcoming 2011 data collection (see Attachment 9). The flyer will provide information on how to complete the surveys as well as the type of information to be requested so they can make plans to retain yearend information they will need.

In addition to materials provided to respondents in the initial survey packet (explained in section B, item 2 above “Procedures for Collecting Information”),

additional communications will be attempt to inform respondents of the progress of the data collection or serve to remind them to respond. These will include the following:

• Thank you letters will be sent to those who have completed the survey, to be sent in batches on the first of the month through June, and will include information on the percent of responses that have been completed as of that date (see Attachment 17).

• E-mail and fax reminders will be sent to nonrespondents in mid-February to alert them to the impending survey due date (see Attachments 14 and 15).

• E-mail and fax reminders will be sent one week after the survey due date to nonrespondents who did not submit their data by the due date (see Attachments 18 and 19).

• Telephone calls will be made one week after the survey due date to nonrespondents of state agencies and large probation agencies (agencies that had a probation population of approximate 10,000 or more on December 31, 2010, as recorded by the 2010 Annual Probation Survey, CJ-8) (see Attachments 5 and 16).

• Additional follow-up will be conducted as needed to nonrespondents who indicate they would need an additional amount of time to provide their response.

Other procedures to maximize response include:

• Follow-up interviews will be conducted as necessary by way of telephone and e-mail if there are data discrepancies (see Attachment 5), and to collect data on unreported quantities or determine if estimates can be easily provided.

• At the time of the initial survey request, the CJ-8A (Short Form) will be provided as a data collection option to selected smaller agencies that previously have exhibited a pattern of providing limited data, that have been chronically late in providing responses, and that have submitted data using that paper options in previous years. This has been shown to improve overall data quality (see section A, item 5, “Impact on Small Businesses or Entities/Efforts to Minimize Burden” for more information).

• The American Probation and Parole Association (APPA) will be asked to include a flyer in its semi-annual training institutes’ packets to announce BJS workshops which presents the data collected in the most recent P&P surveys. BJS will also ask APPA to announce the P&P surveys to its membership association in APPA’s quarterly newsletter, Community Corrections Headlines.

The P&P surveys are hosted on a website with access restricted to survey participants, and to Census Bureau and BJS staff directly involved with these collections using unique IDs and passwords assigned to particular individuals. Edit checks have been incorporated into the web interface and database to provide immediate feedback to respondents with the goal of resolving data issues more quickly, and ultimately, reducing error and item nonresponse. (See also section B, item 4, “Testing of Procedures” for additional clarifications to particular data elements which are expected to improve data quality and contribute to a reduction in item nonresponse.)

The website was also designed to permit BJS and its data collection agent staff to download data as needed to monitor the progress of the data collection. BJS’ data collection agent will provide BJS with biweekly status reports during the data collection period which will serve as the basis for evaluating response rates throughout the data collection cycle. During the collection cycle, BJS and its data collection agent will analyze the data about the collected forms to assess response patterns (e.g., are the same respondent consistently late responders or do the patterns vary) and missing data on submitted forms. BJS will use this information to develop strategies to address issues affecting the timeliness and completeness of data submissions.

All of these methods have enabled BJS to achieve a minimum response rate of 98%, but typically both surveys have achieved a response rate of 100%.

Item nonresponse has been more of an issue. Forty-six probation agencies (10% of total) and zero parole agencies reported critical items only in 2009. For probation, this was an improvement from 67 probation agencies (14% of total) in 2008; for parole, the results were the same in 2008. For probation, generally there was a decrease in item nonresponse in 2009 compared to 2008, with the exception of two types of data.

As the Table 1 shows, item nonresponse is still an issue because the amount of missing data is still high. One of BJS’ collection objectives for 2009 was to reduce the number of probation agencies with missing data for entries and exits. This goal was achieved with respect to the totals in that more probation agencies reported total entries and exits in 2009 compared to 2008 (see table 1). The number decreased in almost all states. There was also a slight increase in item nonresponse related to information on the types of entries and exits. Although item nonresponse is less problematic for parole, it remains a topic of interest which BJS seeks to address. See Table 2 below.

BJS will continue to work with its data collection agent to address both unit and item nonresponse through the methods described above in section A, item 3, “Use of Information Technology”, along with exploring other methods of imputation in order to potentially compare imputed estimates across methods and potentially estimate the imputation error; for example, using the method of multiple imputation as a way of generating imputation error, or a way to add a measure of uncertainty due to missing data. BJS also intends to explore the development of a web interface which displays questions to respondents in a guided fashion, with just one item, or a group of related items, per page, and which provides navigation, definitions, and other information to the respondent, as needed. Interactive features such as these are believed to make web based surveys more user-friendly, and may result in higher completion rates.¹ BJS believes that such an approach would also help to reduce visual clutter which can confuse respondents, and contribute to both item and unit nonresponse.²

4. Testing of Procedures

Any significant changes to the form requires that BJS obtains feedback from several state and local agencies as well as the federal system to ensure definitions and counting rules are clear and consistent across jurisdictions. In addition, OMB approval must be obtained, which includes an outside review of the survey (see section A, item 8, “Federal Register Publication and Outside Consultation”). Prior to offering the web option beginning in 2006, BJS and Census staff tested the instrument extensively. Follow-up testing is performed each year before the forms are made available for use by the P&P agencies, after minor updates are made to revise collection year references.

The CJ-7 Annual Parole Survey, CJ-8 Annual Probation Survey, and CJ-8A Annual Probation Survey (Short Form) (see Attachments 13, 7, and 8; draft 2011 forms) instruments have proved to have a basic, reliable and readily understood format, which BJS staff have only sought to refine as necessary. Only minimal changes are proposed to the data collection instruments for the 2011 reference year compared with the last OMB submission, based on feedback BJS has received from respondents during data collection and also the feedback P&P stakeholders have provided, as explained in section A, item 2, “Needs and Uses (see Attachments 20, 21, and 22 for the collection instruments that were approved in the 2008 OMB submission).

To help educate respondents about race categories and definitions, BJS has provided respondents with an insert for the CJ-7 and CJ-8 forms that explains the OMB racial categories and definitions for race and ethnic groups, along with instructions for respondents to refer to the insert for guidance (see Attachments 23 and 24).

A few small, but important, changes to instructions are recommended. For the Annual Probation Survey, CJ-8, the instruction for item 2b, which asks for a count of the number of persons who were given a sentence which included both probation and incarceration, has been clarified the instruction to include “probationers with a suspended execution of sentence to incarceration.” It was brought to BJS’ attention that all probationers may have their probation revoked for failure to abide by the terms of supervision, even those who received a direct sentence to probation without incarceration (response category 2a).

The instruction for item 3 on both the probation long form (CJ-8) and short form (CJ-8a), has been revised to clarify how to count discharges from probation. The revised instruction indicates discharges should include “…individuals who are discharged from all probation supervision…” rather than simply “all supervision”. Some respondents in combined probation and parole agencies had been misinterpreting this item to mean that for an individual to be counted as a discharge on the Annual Probation Survey, they had to have been discharged from any type of community supervision, including parole supervision. A similar revision has been made to item 3 on the Annual Parole Survey, CJ-7, except that the revised instruction indicates that discharges from parole are those individuals who have been “discharged from all parole supervision.”

Item 12a on the Annual Parole Survey, CJ-7, and item 15a on the Annual Probation Survey, CJ-9, has been expanded to include offenders who also actively report to a probation or parole agency through electronic means, such as kiosk systems, (explained in more detail in section A, item 2,“Needs and Uses”).

The arrangement of items on the forms reflects a logical flow of information to facilitate comprehension of requested items and to reduce the need for follow-up. In addition, instructions and definitions are contained with each item, where necessary, and revised when feedback from respondents and users indicate they are needed for clarification purposes. Also, respondents are provided with the bulletin and detailed tables from the previous year as a reference point for compiling data, and can review and print their data through the website.

External reviewers have found the survey instrument formats, including item content, item display, instructions for compiling the questionnaire, and publication design to be effective and efficient in collecting needed information while minimizing the burden.

5. Contacts for Statistical Aspects and Data Collection

The Correction Statistics Unit at BJS takes responsibility for the overall design and management of the activities described in this submission, including fielding of the survey, data cleaning, and data analysis. BJS contacts include:

Thomas P. Bonczar, Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh St., NW

Washington, DC 20531

(202) 616-36158

Tom.Bonczar@usdoj.gov

Lauren Glaze, Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh St., NW

Washington, DC 20531

(202) 305-9628

Lauren.Glaze@usdoj.gov

William J. Sabol, Deputy Director

Statistical Collections & Analysis

Bureau of Justice Statistics

810 Seventh St, NW

Washington, DC 20531

(202) 514-1062

William.Sabol@usdoj.gov

C. Attachments

Attachment 1: BJS authorizing statute; Title 42, United States Code, Section 3732 (A1.bjslegauth.doc)

Attachment 2: BJS criminal justice flowchart (A2.flow_bw.pdf)

Attachment 3: Correctional Populations in the United States, 2009

(A3.cpus09.pdf)

Attachment 4: Probation and Parole in the United States, 2009

(A4.ppus09.pdf)

Attachment 5: Copy of telephone and e-mail follow-up scripts

(A5.scripts.doc)

Attachment 6: Screenshots of the web option (A6.webshots.doc)

Attachment 7: Draft of 2011 CJ-8 Annual Probation Survey

(A7.CJ8.2011.draft.pdf)

Attachment 8: Draft of 2011 CJ-8A Annual Probation Survey (Short Form) (A8.CJ8A.2011.draft.pdf)

Attachment 9: Draft announcement flyer sent to respondents announcing collection (A9.announcement_flyer.doc)

Attachment 10: Draft cover letter sent to respondents with survey forms (A10.cover.letter.doc)

Attachment 11: Draft flyer to respondents, sent with cover letter

(A11.web_flyer.doc)

Attachment 12: American Probation and Parole Association workshop flyer, sent with cover letter (A12.APPA flyer.doc)

Attachment 13: Draft of 2011 CJ-7 Annual Parole Survey

(A13.CJ7.2011.draft.pdf)

Attachment 14: Draft e-mail sent to nonrespondents prior to due date (A14.nonrespondents_email_before_due_date.doc)

Attachment 15: Draft fax sent to nonrespondents prior to due date (A15.nonrespondents_fax_before_due_date.doc)

Attachment 16: 2010 CJ-8 Annual Probation Survey

(A16.CJ8.2010.OMB.pdf)

Attachment 17: Draft thank you letter sent to respondents

(A17.thankyouletter.doc)

Attachment 18: Draft e-mail sent to nonrespondents after due date (A18.overdue_nonrespondents_email.doc)

Attachment 19: Draft fax sent to nonrespondents after due date (A19.overdue_nonrespondents_fax.doc)

Attachment 20: 2008 Annual Parole Survey (A20.CJ7.2008.OMB.pdf)

Attachment 21: 2008 Annual Probation Survey

(A21.CJ8.2008.OMB.pdf)

Attachment 22: 2008 Annual Probation Survey (Short Form)

(A22.CJ8a.OMB.2008.pdf)

Attachment 23: CJ-7 Insert: Definitions for Racial Categories (A23.race insert CJ-7 parole.doc)

Attachment 24: CJ-8 Insert: Definitions for Racial Categories (A24.race insert CJ-8 probation.doc)

1 Manfreda, L.K., Batagelj, Z., and Vehovar, V. (2002) Design of web survey questionnaires: Three basic experiments, Journal of Computer Mediated Communication 7(3), and Couper, M.P. (2008). Designing effective web surveys. New York: Cambridge University Press.

2 Dillman, D.A., Smith, J.D., and Christian, L.M. (2009) Internet, mail, and mixed-mode surveys: The tailored design method. Hoboken, New Jersey: Wiley.