ASPP2020 Supporting Statement Part B_FINAL

B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The Annual Surveys of Probation and Parole (ASPP) are designed to collect all probation and parole data from community-supervising jurisdictions within each state. The universe includes all federal, state, and locally administered probation and parole departments. Information is collected from central reporters within each state wherever possible, to reduce the burden on individual agencies. For parole, there are 52 respondents: 50 central state reports, the District of Columbia and the federal system. For probation, there are approximately 808 respondents: 40 state reporters and 768 separate city, county, or court reporters. The District of Columbia self-reports, and the data for the federal system are obtained indirectly from the Administrative Office of the U.S. Courts through BJS’ Federal Justice Statistics Program.

As described in part A, since 2012, BJS has been working to update and validate the Annual Survey of Probation frame. In RY 2019, BJS added 66 new agencies to the frame, all of which indicated they supervised at least one felon. In RY2020, BJS will add all of the remaining agencies discovered through this frame development research, including 20 agencies supervising felons and 268 agencies supervising only misdemeanants. BJS will confirm that these agencies supervised probationers in the past year, and gather the critical data elements that will allow BJS to produce a comprehensive, valid estimate of the probation population. The RY2020 collection will refine the frame resulting in an up-to-date list of agencies to survey in future years.

Routinely, BJS employs methods to maintain the accuracy and the completeness of the population frame for each survey:

• Agency staff provide information about newly formed, merged, and closed supervising agencies while data collection is ongoing via data retrieval phone calls and emails. This information is used to update the frame prior to the start of each data collection year.

• Close attention is paid to unexplained changes in the total population that occur from the end of one year to the beginning of the next, and large increases or decreases in the total population during the current reporting year. During survey administration, a comparison is made between the previous yearend population to the reported beginning current year population and, if there is a difference of 10% or greater, respondents are prompted to review their data. They are then asked to enter a reason for the discrepancy between the populations over two days.

Following data submission, all data are reviewed. Probation agencies with populations of 100 or more and parole agencies of any size, whose previous yearend population differs by more than 5% from that of their reported beginning year population, are flagged for review and potential follow up. This also occurs for probation agencies with a population less than 100 probationers and a 10% or greater population difference. RTI also reviews the information provided by agencies when January 1 to December 31 growth for the current reporting year exceeds 10%.

During follow-up, RTI uses open-ended probes to determine the reasons for differences in yearend to beginning year population or in the current reporting year. Differences may be explained by a variety of reasons, such as a data entry error, a reporting method change, a change in the agency’s responsibility (e.g., an agency has taken responsibility for probationers or parolees who were previously supervised by another agency), or, in the case of within-reporting year change, to genuine growth or decline of the population.

Over the past 2 years of the ASPP surveys, these methods have enabled BJS to achieve a minimum survey response rate of 90%. In 2018, the response rate for the Annual Probation Survey was 92% of surveyed agencies (representing 99% of the 2018 yearend probation population) and the response rate for the Annual Parole Survey was 100% (table 1).

2. Procedures for Collecting Information

Collection Procedure

BJS emphasizes the web as the primary mode of data collection. Hardcopy forms are sent to respondents upon request only. To draw attention to the ASPP collection in advance of the formal request to participate, a pre-notification letter is mailed and emailed to agencies in early November (Attachment 15). The letter provides information about the purpose and importance of the surveys as well as the type of information to be requested so they can plan to retain the yearend information that they will need. Agencies are asked to update or confirm contact information for the most appropriate person to respond to the survey by logging onto the website or using an enclosed designation form. They are also asked to indicate whether the designated respondent is from a private company contracted to supervise probationers.

In December, all agencies receive a survey invitation letter requesting that they complete the survey on the web (Attachment 13). The letter explains the importance of the survey and provides a link to the most recent BJS Probation and Parole in the United States bulletin, states that participation is voluntary and thanks them for their involvement. Each agency is provided with a unique user ID and password to securely access the survey website to complete the questionnaire. Shortly after the letter is sent, each agency receives a follow-up email that references their invitation, provides a direct link to the survey website, and encourages the agency’s participation.

After this invitation, other communications inform respondents of the status of data collection or serve to remind them to respond. These include the following:

• Automatic thank-you emails are sent to those that have submitted their web survey (Attachment 16).

• Three reminder messages are sent to non-respondents throughout the data collection period.

• The first is sent via email to alert respondents of the impending survey due date (Attachment 17).

• The second is sent both via USPS and e-mail within a month of the survey due date (Attachment 18).

• The third is sent three weeks before the final cutoff of data collection, around the third week of April (Attachment 19), via USPS.

• Telephone calls, as a reminder to non-respondents, are made to non-respondents immediately following the survey due date. The scripts are tailored to the size, type, and reporting history of the agency (Attachment 20).

• BJS instituted a practice of sending a closeout email towards the end of data collection. This letter describes the status of the agency’s submission. There are three versions of the closeout email message: no data, partial data, or data that required clarification (Attachments 21, 22, 23).

• Additional follow-up is conducted as needed with non-respondents:

• Non-respondents that indicate they need more time to provide data receive follow-up contact by telephone in an attempt to resolve data discrepancies and obtain answers to items left unanswered in the survey (attachment 20).

• Certain subsets of non-responding agencies receive tailored communications applicable to their agency characteristics. These subsets may be identified using RTI’s adaptive technology dashboard (ATD).

Within 2 weeks of survey submission, follow-up activities begin. If critical items are missing or inconsistent, such as the beginning year or yearend population or the number of entries to or exits from supervision, staff contact respondents to determine if they can provide estimates or explanations for inconsistencies (see Attachments 24 and 25). Staff work with the respondents to estimate missing information if it cannot be easily provided, making sure to obtain agreement from the respondent before disseminating data containing any revisions.

Within the first four weeks of the start of the data collection period, preliminary analysis begins. RTI staff check the data for out-of-range values, missing data, and other types of responses that need data editing/cleaning. These preliminary analyses are undertaken while data collection is still in progress to provide adequate time for follow-up clarification calls. BJS sends a final submission thank you letter letting the agency know their data have been processed and will be included in analyses for the annual Probation and Parole bulletin (attachment 26).

Imputation Procedures

BJS has developed several imputation methods to estimate January 1 and December 31 populations as well as entries and exits if respondents are unable to provide any of the key information. For unit non-response, a combination of the population, entry and exit imputation methods are applied. When the January 1 probation population is missing, the December 31 population from the prior year is carried over. When the December 31 probation population is missing and January 1 exits and entries from the current year are also missing, the December 31 population is set to the last reported December 31 number.

When the January 1, 2018, probation population is missing, the December 31 probation population from the last reported year going back to 2010 is carried forward. When the January 1, parole population is missing, the December 31 probation population from the prior year is also carried forward.

When the December 31, 2018, parole population, total entries, or exits is missing, the missing values are imputed by adding to (or subtracting from) the current January 1, parole population to estimate population change based on what was observed for the prior year. The intra-year change in population from January 1 to December 31 of prior year—expressed as a proportion of the prior year January 1 total—is multiplied by the current year January 1 total to estimate the current year population change

BJS uses four methods of ratio estimation to impute probation entries for agencies not reporting these data. The first method is used to estimate entries for probation agencies that do not report entries in the current year but did report in the prior year. BJS estimates probation entries in the current year by using the ratio of entries in the prior year to the agency’s prior year probation population on January 1, and applying that ratio to the agency’s current year January 1 population. This method is used for agencies that report all four key items in at least one year since 2010 and for which the current year January 1 and December 31 populations are equal (likely due to the imputation of one or both of those variables). The entries and exits in the most recent of those years are divided by the beginning and year-end populations from the same year (stock overflow), and the resulting ratio is multiplied by the current year January 1 population.

The second method is used to estimate probation entries for agencies that do not report all four core variables in any single year since 2010 or have different beginning and year-end populations. The ratio of prior year entries to the January 1, population is multiplied by the January 1 population to derive the current year entries.

The third method estimates entries in agencies with small populations. This method estimates the relationship between current year entries and the January 1 population by calculating the ratio of the sums of these variables across similarly sized agencies within the same state. This ratio is then multiplied by the January 1 value to obtain current year entries. To ensure the stability of the ratio estimator, this method is only employed in states with at least 30 reporting units.

The fourth method used to estimate probation entries takes the ratio of prior year imputed entries to the prior year January 1 probation population and applies that ratio to the agency’s current year January 1 population.

The specific methods detailed above, and the jurisdictions to which they apply, are documented in the “Methodology” section of reports in the series Probation and Parole in the United States¹. The imputed values are used for all analyses and reports published by BJS. Imputed values are flagged as such in the files that are sent to NACJD (http://www.icpsr.umich.edu/icpsrweb/NACJD/index.jsp).

3. Methods to Maximize Response

BJS employs several techniques to maximize response rates. They include—

• Contacting the agencies prior to the start of data collection and making frequent contacts during the data collection period to solicit participation.

• Sending web survey invitations which include login instructions for the web survey in both hard-copy through the USPS and in electronic format.

• Making it easy for agencies to participate by providing technical support and other help with the survey as needed, offering a response mode other than web, if requested, and providing respondents with real-time online data checks to add efficiency to the response process.

• Engaging respondents in the data collection process by highlighting BJS reports that provide information central to agency needs (see http://www.bjs.gov/index.cfm?ty=tp&tid=15#pubs).

• Using RTI’s ATD tool to analyze response patterns and determine the most effective methods for contacting and following up with agencies. The ATD provides BJS and RTI with real-time response data broken down by agency characteristics and compared to historical trends.

• Providing the CJ-8A as a data collection option to smaller probation agencies. This has been shown to improve overall data quality (see section A, item 5, “Impact on Small Businesses or Entities/Efforts to Minimize Burden” for more information).

BJS monitors the progress of the data collection by reviewing data about respondents and non-respondents, their response characteristics, and their communications throughout the data collection period to inform and enhance non-response follow-up. BJS currently has real-time access to the following data:

• Agency contact information (e.g., names of agency heads and designated respondents, street and email addresses, telephone numbers);

• Individual files containing the image of each submitted survey from the including notes provided by the respondents;

• Mode and date of survey submission;

• Notes describing contacts with agencies as well as follow-up efforts; and

• Statistics on the current year’s overall response rates and the response rates for each survey type.

• Historical data for the agency, including submission status, images of submitted surveys, and notes for the past 5 years.

During the collection cycle, the data are analyzed to assess response patterns (e.g., whether the same respondents are consistently late responders) and missing data on submitted forms, and to develop strategies to address the timeliness and completeness of data submissions.

Tailored follow-up timelines using the response patterns are put into place to not needlessly contact a respondent when their time of submission can be predicted. For example, data collected labels respondents as on time, up to 1 month late, 1 to 3 months late, and more than 3 months late. Non-respondents who have historically submitted on time or up to 1 month late receive a decreased non-response effort until later in the follow-up period.

Unit Non-Response

As seen in table 1, survey response rates for the past 2 years are at least 96% for parole respondents and 90% for probation respondents. To publish national totals of people supervised on probation BJS developed strategies to impute missing data for key items for agencies that do not respond. These items include the beginning of the year count, total entries, total exits, and the end of year count. Imputation for the end of year population count continues to be very low and concentrated in agencies who supervise very small populations. In 2018, only about 29,121 people in 37 agencies were imputed in a population of over 4.4 million.

Item Non-Response

Rates of item nonresponse on the parole survey vary, with maximum sentence continuing to be the largest nonresponse (table 2). Item nonresponse rates for the probation surveys have been higher than for the parole surveys.

4. Testing of Procedures

During the 2019 collection, BJS performed non-response follow-up using a stratified approached based on the date of submission in the previous year. If a respondent had a very late response in 2018, they were not contacted right after the due date. This resulted in fewer phone calls, but did not affect the response-rate. Targeted outreach was also conducted by using the ATD to identify specific groups that may be falling behind in survey response. After the conclusion of each year of data collection, BJS reviews the effectiveness of each method of contact with the respondents.

5. Contacts for Statistical Aspects and Data Collection

The Correction Statistics Unit at BJS takes responsibility for the overall design and management of the activities described in this submission, including fielding of the survey, data cleaning, and data analysis. BJS contacts include:

Barbara Oudekerk, Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh Street, NW

Washington, DC 20531

Barbara.A.Oudekerk@usdoj.gov

202-616-3904

Danielle Kaeble, Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 Seventh Street, NW

Washington, DC 20531

Danielle.Kaeble@usdoj.gov

202-305-2017

1 See Attachment 4, Probation and Parole in the United States, 2017-2018; other reports in the series are available on the BJS website at http://www.bjs.gov/index.cfm?ty=pbse&sid=42.