CJRP Supporting Statement B Statistical Methodology

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

This data collection will be a census of juveniles in residential placements on the reference date. The CJRP universe includes all public, private, territory and tribal juvenile facilities in the United States.

1. Universe and Respondent Selection

For this census, OJJDP has defined the universe to include all facilities that hold juveniles as offenders. Juvenile offenders are defined in the census as persons younger than age 21 who are held in a residential setting as a result of some contact with the justice system (that is, they are charged with or adjudicated for an offense). This encompasses both status offenses¹ and delinquency offenses, and includes youth who are either temporarily detained by the court or committed after adjudication for an offense. The census does not include federal facilities or facilities holding youth exclusively for drug or mental health treatment or for abused/ neglected youth. It also does not capture data from adult prisons or jails.

NIJ, in coordination with OJJDP, intends to survey all public and private facilities in the United States that fulfill these requirements. The 2019 CJRP includes a total of 2,204 facilities: 1,058 public and 1,146 private residential facilities. These facilities run the gamut of environments from open facilities in which the youth reside in a group home environment to the high-security training schools that house upwards of 400 youth.

Based on expert consultation, OJJDP determined that a census is most efficient and effective in meeting the government’s interest because a nationally representative sample of facilities would not allow for between state comparisons, hampering states’ ability to develop informed juvenile justice policy at the state-level. States wish to make comparisons among themselves, and given that juvenile justice policy is made at the state level, a national sample would not serve those purposes. Furthermore, a number of states have only a few facilities (some just one or two). To create a sample large enough to make adequate state-level estimates, OJJDP would in effect conduct a census in states with a small number of facilities. Thus, creation of the state-level estimates of residential placement would almost require a national census.

2. Procedures for Information Collection

To maintain an accurate and complete list of all facilities of interest, OJJDP funds the Census Bureau to maintain a list that includes the facilities’ names, addresses, locations, phone numbers, e-mail addresses, and classification information. Although the CJRP collection occurs biennially, the universe of juvenile residential facilities is supported and maintained on an annual basis because it is used for both the CJRP and Juvenile Residential Facilities Census (JRFC), which occur in alternating years.

To maintain this list, the Census Bureau regularly receives resource materials from NIJ, OJJDP, and other professional and state juvenile justice organizations, and periodically contacts OJJDP grantees, juvenile justice stakeholders, and state juvenile justice agency personnel to gather information on new facilities (births), facility closings (deaths), and changes in facility characteristics. The following individuals and organizations are queried annually to provide updates to the juvenile justice facility universe:

• OJJDP Core Protections Division (compliance data submitted by states);

• Council of Juvenile Correctional Administrators;

• National Center for Juvenile Justice;

• Performance-based Standards (a program for juvenile justice agencies, facilities and residential care providers to identify, monitor and improve conditions and rehabilitation services provided to youths using national standards and outcome measures).

• Center for Coordinated Assistance to States (a training and technical assistance provider that assist states in complying with the four core protections of the Juvenile Justice and Delinquency Prevention Act)

• State Juvenile Justice Specialists (these individuals oversee the management of OJJDP’s State Formula and Block grant funds); and

• State Compliance Monitors (these individuals oversee the monitoring of juvenile justice and adult facilities within the state for compliance with requirements of the Juvenile Justice and Delinquency Prevention Act).

Census Bureau staff compare each facility listed in the OJJDP’s state compliance documentation to the Juvenile Agency Listing (juvenile facility universe). Census Bureau staff research all facilities that are not currently included in the frame to determine if the facility falls within the scope of the data collection. After comparing the facilities to known out of scope facilities, conducting Internet research and phone calls, a determination is made regarding the status of the facility (i.e., new facility, closed or out of scope facility, etc.) In addition, Census Bureau staff conduct verification calls and nonresponse follow-up calls to facilities. These calls often yield useful information regarding the status of facilities, including the existence of new facilities.

Since the initiation of the CJRP and JRFC in 1997 and 2000 respectively, positive, long-term relationships have developed among the data collection agents at the Census Bureau, OJJDP, and the CJRP/JRFC respondents. Many of the updates result from direct respondent contact with the Census Bureau. For example, many respondents write new or updated information on submitted CJRP or JRFC survey forms, prompting follow-up from the Census Bureau as necessary. Additionally, a non-deliverable form returned via the United States Postal Service or a non-response will prompt the Census Bureau to initiate research on a facility to gain updated information. This outreach is often as simple as contacting the respondent on file or a state agency. Depending on the outcome of the outreach, a more in-depth search may be implemented, at times with NIJ and/or OJJDP assistance.

During 2017 (the most recent period for which data are available--through the CJRP), the Census Bureau identified facilities that reported changes affecting the CJRP/JRFC universe. These changes included information updates (changes to respondent contact, facility address and facility name) and status changes (facility births, deaths, merges, and temporary closures). For example, the Census Bureau identified 80 permanently closed facilities, 23 temporarily closed facilities, and 49 out-of-scope facilities.² The Census Bureau also identified 110 point of contact changes and 249 address changes for respondents. Research through the methods described above, including review of state compliance documents, identified 24 new facilities.

The information collection schedule (below) was developed based on experience in testing and administering the CJRP, eleven times since 1997, as well as experience conducting other establishment surveys. Should circumstances require schedule changes (i.e., a decision to make the telephone follow-ups earlier in the schedule), it will be adjusted accordingly. After the initial mailout and due date have passed, Census Bureau staff prepare and execute a second mailout of the form to those facilities that have not responded. Subsequent follow-up involve staff contacting the remaining respondents via phone to secure participation and/or to gather critical data elements. Additional details on nonresponse contact protocols can be found in Attachment M.

Typically, OJJDP has been able to achieve a high response rate (83 to 95 percent) for its facility-based censuses. Such a level of response has proven sufficient for purposes of the designated analysis. NIJ and OJJDP expect to continue this high response rate in future administrations of the CJRP.

Table 1. Information collection schedule for the 2019 CJRP

TASK NAME	DATE		TASK NAME	DATE
Project Planning			Mailout
OMB clearance	04/2019		CJRP Questionnaires/letter/web flyer mailed to respondents	10/21/2019
Questionnaire design	04/2019		Reference date: October 23, 2019	10/23/2019*
JAL/Universe Update	Continuous		Data collection due	11/27/2019*
Centurion (online data collection)			Keying Requirements
Kickoff Meeting	07/2019		Develop keying specifications	11/2019
Bi-weekly meetings	Continuous		Conduct Keying Operations	11/2019
Requirements Document	07/2019		Nonresponse Follow-up
Changes to the Centurion Site document	08/2019		Conduct 1st nonresponse follow-up (mass email)	11/2019 or 12/2019
Design and Submit Fillable PDF	09/2019		Conduct 2nd nonresponse follow-up (mass email)	01/2020
Test Online Data Collection Tool	09/2019		Develop non-response specifications	01/2019
Centurion User Guide Document	09/2019		Conduct training	01/2019
Upload Fillable PDF to Instrument	09/2019		Conduct nonresponse follow-up phone calls	01/2020 - 07/2020
Create Input File	10/2019		Print letters/web flyers/mailing labels for 2nd mailing	01/2020
Re-test Online Data Collection Tool	10/2019		Conduct 2nd nonresponse follow-up mailing	02/2020
Instrument moved to production	10/23/2019		Closeout	09/2020
General Processing System (GPS)			Tabulation/Analysis
Kickoff Meeting	07/2019		Imputations
Bi-weekly meetings	continuous		Develop specifications	09/2020
Updates to Application: includes all updates, presentation, and edits that were documented on specifications	08/2019		Program specifications	09/2020
Requirements: Submit Crosswalk	09/2019		Generate and review imputations	10/2020
Specifications for mandatory updates, enhancements and corrections submitted	10/2019		Tables/Estimates
Test Output transfer from Centurion to GPS	10/2019		Develop specifications	09/2020
Test transfer from Processing App to Database	10/2019		Program specifications	09/2020
Test Processing application (GPS) includes testing usability, functionality and edits	11/2019		Generate and review tables	10/2020
Data Review, Analysis & Editing	11/2019 - 09/2020		Data File Preparation
Data Collection			Review/correct data	09/2020
Mailout Preparation			Prepare preliminary file	09/2020
Submit Draft CJ-14 form to CENDOCs	04/2019		Deliver file to ESMPD	10/2020
Final Approval of CJ-14 Form	07/2019		Prepare documentation package	11/2020
Receive updated mailout letter from OJJDP	06/2019		Deliver final file to NIJ/OJJDP	12/2020
Order mailout materials (envelopes, supplies, etc.)	07/2019		Archive Data	01/2021
Receive letter approval from sponsor	08/2019		Evaluation
Print letters and web flyers	09/2019		Evaluate survey operations	continuous
Send mailout specification	09/2019		Develop recommendations	continuous
Send mail file to NPC	10/2019
Questionnaire printing	10/2019		* Subject to OMB approval.

3. Methods to Maximize Response

NIJ, OJJDP, and the Census Bureau are committed to achieving a very high response rates and high-quality data. Using the number of in-scope facilities as a base, the CJRP facility response rate is typically between 83 and 95 percent.

Historically, a small proportion of in-scope facilities neither return CJRP forms nor respond to telephone calls from the Census Bureau requesting that the institution participate in the census. In 2017, there were a total of 2,101 “in-scope” facilities in the universe.³ The 2017 CJRP had a response rate of 84 percent, meaning a total of 337 facilities did not respond. The Census Bureau imputed records for the nontribal facilities in the 50 states and the District of Columbia that were refusals. In other cases, some facilities were not able to provide all of the information requested and these data are imputed from complete records to fill in incomplete records. Techniques to maximize response rates and nonresponse adjustment procedures are summarized below.

NIJ will use the following techniques to maximize response:

• Electronic reporting in a manner acceptable to the respondent (e.g., in 2017, the Census Bureau continued the use of a user friendly online reporting mechanism that has yielded good results);

• Streamlined forms and clear response instructions (e.g., the addition of facility type definitions noted under Section A.15);

• Continued support at the Census Bureau through a toll-free number to answer any questions from respondents that arise;

• Continuous contact with respondents through e-mail and U.S. mail (see the schedule for mailout and reminder notices, above); and

• Call-back procedures that continue until data closeout in May/June of the following year.

Notably, based on feedback from respondents and analysis of CJRP paradata, the Census Bureau implemented enhancements to the functionality of its online data collection system to improve respondent experiences, including: adding comment boxes to each section, adding a filter to the review pages, and adding totals to the edit message to help respondents correct those responses that do not sum as expected.

Additionally, staff at the Census Bureau have reported that facilities may be more likely to respond if they are able to see the importance of their data in understanding national trends, and those respondents with confidentiality concerns may be reassured that their data are only reported in the aggregate. Accordingly, in an effort to demonstrate to facilities how their data is used by policymakers and the public, the OJJDP-published bulletin: Juveniles in Residential Placement, 2015 (see Attachment J), was included in the October 2017 mailout of the CJRP instrument. This technique is also planned for the 2019 CJRP data collection which will include copies of the OJJDP bulletin: Juveniles in Residential Placement, 2017. Subsequent follow-ups by mail will include resources from OJJDP’s Data Snapshot series, one-page data visualizations that focus on policy relevant statistical information (for example, see Attachment L, Service Availability Increased in Juvenile Residential Placement Facilities).

NIJ and OJJDP anticipate this ongoing effort to engage respondents will continue to yield high response rates. In the best of all possible scenarios, statistical estimation would not be required as this is a census. However, given the inevitable facility nonresponse and item nonresponse, NIJ and OJJDP will work with the Census Bureau, to ensure valid and reliable procedures to estimate the population characteristics.

Nonresponse Adjustments

The CJRP is a census of all the juvenile residential facilities in the United States and its territories. No sample weights are used in the data collection. As noted above, it is expected that a small proportion of in-scope facilities will not respond to the census and some facilities will not respond to all items. Overall, unit and item-level response rates are expected to remain sufficient for producing national and state-level estimates of juvenile offenders in residential placement. Nevertheless, it is expected that missing values will be imputed for both unit and item nonresponse. The imputation methods include:

• Growth rate applied to prior year data,

• Mean value reported in imputation cell,

• Value derived based on other fields on the facility record,

• Hierarchical hot-deck (for item non-response).

Adjustments for missing population counts and check box questions

If the number of juvenile offenders, or juveniles assigned to beds for reasons other than offense are missing, the Census Bureau applies a growth rate to the prior year value and rounds it to a whole number. If no prior year value is available, the Census Bureau uses the mean value reported within the imputation cell rounded to a whole number.

If the number of juveniles assigned to beds is missing, the Census Bureau derives the value by adding juvenile offenders and juveniles assigned to beds for reasons other than offense. If persons assigned to beds is missing, the Census Bureau derives the value by adding juveniles assigned to beds and adults assigned to beds.

If the values for the check box questions in the General Facility Information section are missing, the Census Bureau pulls forward prior year data if available. Those questions include the questions about the physical layout of the facility, whether or not on-site residential treatment, foster care, or independent living programs are provided, the types of sleeping units in the facility, who owns and operates the facility, and how juveniles are confined in specific areas.

Adjustments for item nonresponse

The imputation methodology for item nonresponse is hierarchical hot-deck. The Census Bureau matches the record requiring imputation to a pool of records where none of the information is missing, and then selects a donor record. The missing values are replaced in the record requiring imputation with the values from the donor record. The procedure first attempts to match on all available information. If a match on all information is not found, the match is made less restrictive until a donor record is found.

Juvenile offenders in tribal facilities or territorial facilities are not eligible to be donors for juvenile offenders in nontribal facilities in the 50 states and District of Columbia. The Juvenile Justice and Delinquency Prevention Act prohibits the placement of status offenders in secure facilities in most instances, and as a result, status offense are not imputed on records for secure facilities.

Additional details on unit and item level nonresponse and nonresponse adjustments are available in methodology reports produced by the Census Bureau. The most recent report from 2017 is included in Attachment N.⁴

4. Test of Procedures or Methods

There are no changes to the forms or data collection methods at this time. However, in the future, any significant changes to the form would require that NIJ and OJJDP obtain feedback from state, local, and private agencies to ensure new items, definitions, and counting rules are clear across jurisdictions. In addition, OMB approval would need to be obtained, which includes an outside review of the survey. Consistently high response rates in recent CJRP and JRFC data collection cycles suggest that that the planned procedures for the 2019 CJRP will yield similar or improved results.

The Census Bureau and OJJDP originally tested the data collection procedures in a field test that began in mid-1990s. The Center for Survey Methods Research (CSMR) at the Census Bureau began with semi-structured exploratory interviews of 20 respondents. The respondents were varied based on size and type of facility. These interviews were designed to learn how respondents think about the population in their facilities and how they understand various important concepts OJJDP wished to report on (for example, delinquent versus status offense).

Results of these interviews informed the development of a test instrument, which was reviewed and refined by OJJDP staff and a group of consultants. CSMR used the refined draft instrument to conduct multiple rounds of cognitive interviews with respondents. Based on these interviews, CSMR and OJJDP produced an instrument for pretesting. Using a reference date of October 30, 1996, the Governments Division of the Census Bureau conducted a pre-test of 400 facilities. This test included a small sample of facilities (96) that would receive the Children in Custody (CIC) form so that CSMR could compare the results of these two tests to more accurately judge how the CJRP form performed compared with the CIC. Statisticians at the CSMR at the Census Bureau analyzed the data and submitted a report to OJJDP.

Since the first full administration of the CJRP in 1997, OJJDP has worked with the Census Bureau to establish and maintain appropriate statistical procedures for the data files. As part of the normal procedures for each CJRP file, the Economic Statistical Methods and Programming Division (ESMPD) of the Census Bureau analyzes the quality of the data and develops methods for imputing for facility nonresponse and item nonresponse. ESMPD provides a detailed report for each file. This report, described above, demonstrates the quality of the data collection efforts and the procedures the Economic Reimbursable Surveys Division uses to collect the CJRP data.

In addition, following the 2017 data collection, the Census Bureau reviewed survey performance through a variety of methods including analyzing mode of response consistency, web respondent paradata, respondent comments and text responses to questions that include an “other, specify” response option, and consulting with a sample of respondents (fewer than 10) to better understand their thoughts on the survey content, instruments, and burden.

As noted in Supporting Statement A, in fiscal year 2018, OJJDP made a separate award, now managed by NIJ, to assess and improve the data collection instruments and methodologies currently used in the CJRP and JRFC to generate useful, timely, and reliable national and state-level statistics on juveniles held in out-of-home placement and the facilities in which they are held. To achieve this goal, RTI International will collaborate with NIJ, OJJDP, and an expert panel to evaluate the content and methods used in the CJRP and JRFC and develop new approaches that will augment existing gaps in the current instruments and data collection designs. The project team will design a pilot test, which will be submitted at a later date for OMB review and approval, to evaluate the proposed approach and determine to what extent it achieves the goal of generating more useful, reliable, and timely data. NIJ and OJJDP expect these improvements will be implemented in future data collections cycles (following the current requested extension period).

5. Consultation Information

Presently, OJJDP funds an Interagency Agreement (IAA) with the Economic Reimbursable Surveys Division at the Census Bureau to perform data collection and to maintain the data file and address lists. This IAA also funds the imputation activity related to the CJRP file.

Relevant Contacts:

Benjamin Adams

Social Science Analyst

National Institute of Justice

(202) 616-3687

Brecht Donoghue

Senior Social Science Analyst

National Institute of Justice

(202) 305-1270

Nicole Adolph

Chief, Criminal Justice Statistics Branch

Economic Reimbursable Surveys Division

US Census Bureau

(301) 763-1577

 

Crecilla Scott

Supervisory Statistician, Criminal Justice Statistics Branch

Economic Reimbursable Surveys Division

  US Census Bureau

(301) 763-2761

Krystal Jimerson

Statistician, Criminal Justice Statistics Branch

Economic Reimbursable Surveys Division

  US Census Bureau

Sabrina Webb

Statistician, Criminal Justice Statistics Branch

Economic Reimbursable Surveys Division

US Census Bureau

Suzanne Dorinski

Mathematical Statistician

Economic Statistical Methods Division

  US Census Bureau

Magdalena Ramos

Assistant Division Chief

Methodology Director for Public Sector Statistics

Economic Statistical Methods Division

US Census Bureau

1 “Status offenses” are offenses that are illegal for minors but not for adults. For example, truancy or running away may be a status offense depending on the state in which the juvenile resides. Other status offenses include incorrigibility, underage drinking, and curfew violations.

2 A facility is temporarily out-of-scope when it does not hold juveniles on the reference date. A facility is permanently out-of-scope for one of several reasons (see Attachment N), for example, if the facility is no longer a residential facility (e.g., converted to day treatment only) or the facility no longer holds any juveniles (only handles adults).

3 Includes 20 tribal facilities and 11 territorial facilities which are handled separately. Their records were not used for imputation or published national estimates.

4 Dorinski, Suzanne M. Documentation of the Imputation Methodology for the 2017 Census of Juvenile in Residential Placement. Washington, DC: U.S. Census Bureau.

18

8/19/2021