JRFC Supporting Statement B Statistical Methodology

JRFC Supporting Statement B Statistical Methodology.docx

Juvenile Residential Facilty Census (JRFC)

OMB: 1121-0219

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1121-0219 can be found here:

2022-05-19 - No material or nonsubstantive change to a currently approved collection

Document [docx]

Download: docx | pdf

B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

This data collection will be a census of juvenile residential facilities that hold juvenile offenders on the reference date. The JRFC universe includes all public, private, territory and tribal juvenile facilities in the United States.

1. Universe and Respondent Selection

For this census, OJJDP has defined the universe to include all facilities that hold juveniles as offenders. Juvenile offenders are defined in the census as persons younger than age 21 who are held in a residential setting as a result of some contact with the justice system (that is, they are charged with or adjudicated for an offense). This encompasses both status offenses^¹ and delinquency offenses, and includes youth who are either temporarily detained by the court or committed after adjudication for an offense. The census does not include federal facilities or facilities holding youth exclusively for drug or mental health treatment or for abused/ neglected youth. It also does not capture data from adult prisons or jails.

NIJ, in coordination with OJJDP, intends to survey all public and private facilities in the United States that fulfill these requirements. The 2020 JRFC frame includes 2,031 facilities. These facilities run the gamut of environments from open facilities in which the youth reside in a group home environment to the high-security training schools that house upwards of 400 youth.

Based on expert consultation, OJJDP determined that a census is most efficient and effective in meeting the government’s interest because a nationally representative sample of facilities would not allow for between state comparisons, hampering states’ ability to develop informed juvenile justice policy at the state-level. States wish to make comparisons among themselves, and given that juvenile justice policy is made at the state level, a national sample would not serve those purposes. Furthermore, a number of states have only a few facilities (some just one or two). To create a sample large enough to make adequate state-level estimates, OJJDP would in effect conduct a census in states with a small number of facilities. Thus, creation of the state-level estimates of residential placement would essentially require a national census.

2. Procedures for Information Collection

To maintain an accurate and complete list of all facilities of interest, OJJDP funds the Census Bureau to maintain a list that includes the facilities’ names, addresses, locations, phone numbers, e-mail addresses, and classification information. Although the JRFC collection occurs biennially, the universe of juvenile residential facilities is supported and maintained on an annual basis because it is used for both the JRFC and Census of Juveniles in Residential Placement (CJRP), which occur in alternating years.

To maintain this list, the Census Bureau regularly receives resource materials from NIJ, OJJDP, and other professional and state juvenile justice organizations, and periodically contacts OJJDP grantees, juvenile justice stakeholders, and state juvenile justice agency personnel to gather information on new facilities (births), facility closings (deaths), and changes in facility characteristics. The following individuals and organizations are queried annually to provide updates to the juvenile justice facility universe:

OJJDP Core Protections Division (compliance data submitted by states);
Council of Juvenile Justice Administrators;
National Center for Juvenile Justice;
Performance-based Standards (a program for juvenile justice agencies, facilities and residential care providers to identify, monitor and improve conditions and rehabilitation services provided to youths using national standards and outcome measures).
Center for Coordinated Assistance to States (a training and technical assistance provider that assist states in complying with the four core protections of the Juvenile Justice and Delinquency Prevention Act)
State Juvenile Justice Specialists (these individuals oversee the management of OJJDP’s State Formula and Block grant funds); and
State Compliance Monitors (these individuals oversee the monitoring of juvenile justice and adult facilities within the state for compliance with requirements of the Juvenile Justice and Delinquency Prevention Act).

Census Bureau staff compare each facility listed in the OJJDP’s state compliance documentation to the Juvenile Agency Listing (juvenile facility universe). Census Bureau staff research all facilities that are not currently included in the frame to determine if the facility falls within the scope of the data collection. After comparing the facilities to known out of scope facilities, conducting Internet research and phone calls, a determination is made regarding the status of the facility (i.e., new facility, closed or out of scope facility, etc.) In addition, Census Bureau staff conduct verification calls and nonresponse follow-up calls to facilities. These calls often yield useful information regarding the status of facilities, including the existence of new facilities.

Since the initiation of the CJRP and JRFC in 1997 and 2000 respectively, positive, long-term relationships have developed among the data collection agents at the Census Bureau, OJJDP, and the JRFC/CJRP respondents. Many of the updates result from direct respondent contact with the Census Bureau. For example, many respondents write new or updated information on submitted JRFC or CJRP survey forms, prompting follow-up from the Census Bureau as necessary. Additionally, a non-deliverable form returned via the United States Postal Service or a non-response will prompt the Census Bureau to initiate research on a facility to gain updated information. This outreach is often as simple as contacting the respondent on file or a state agency. Depending on the outcome of the outreach, a more in-depth search may be implemented, at times with NIJ and/or OJJDP assistance.

During 2018 (the most recent period for which data are available), the Census Bureau identified facilities that reported changes affecting the JRFC/CJRP universe.^² These changes included information updates (changes to respondent contact, facility address and facility name) and status changes (facility births, deaths, merges, and temporary closures). For example, the Census Bureau identified 91 permanently closed facilities, 30 temporarily closed facilities, and 39 out-of-scope facilities.^³ The Census Bureau also identified 49 facility name changes, 279 point of contact changes and 83 address changes for respondents. Research through the methods described above, including review of state compliance documents, identified 74 new facilities.

The information collection schedule (below) was developed based on experience in testing and administering the JRFC, ten times since 2000, as well as experience conducting other establishment surveys. Should circumstances require schedule changes (i.e., a decision to make the telephone follow-ups earlier in the schedule), it will be adjusted accordingly. After the initial mailout and due date have passed, Census Bureau staff prepare and execute a second mailout of the form to those facilities that have not responded. Copies of the initial mailout and non-response follow-up letters can be found in Attachments F and L, respectively. Subsequent follow-up involves several follow-up emails as well as staff contacting the remaining respondents via phone to secure participation and/or to gather critical data elements. Additional details on nonresponse contact protocols can be found in Attachment M.

Typically, OJJDP has been able to achieve a high response rate (83 to 95 percent) for its facility-based censuses. Such a level of response has proven sufficient for purposes of the designated analysis. NIJ and OJJDP expect to continue this high response rate in future administrations of the CJRP.

Table 1. Information collection schedule for the 2020 JRFC

TASK NAME	DATE	TASK NAME	DATE
Project Planning		Mailout
OMB clearance	09/2019	JRFC Questionnaires/letter/web flyer mailed to respondents	10/22/2020
Questionnaire design	04/2020	Reference date: October 28, 2020	10/28/2020*
JAL/Universe Update	Continuous	Data collection due	11/25/2020*
Centurion (online data collection)		Keying Requirements
Requirements Document	04/2020	Develop keying specifications	11/2020
Changes to the Centurion Site document	05/2020	Conduct Keying Operations	11/2020
Kickoff Meeting	06/2020	Nonresponse Follow-up
Bi-weekly meetings	Continuous	Conduct 1st nonresponse follow-up (mass email)	11/20/20 or 12/2020
Design and Submit Fillable PDF	09/2020	Conduct 2nd nonresponse follow-up (mass email)	01/2021
Test Online Data Collection Tool	09/2020	Develop non-response follow-up phone call specifications	01/2021
Centurion User Guide Document	10/2020	Conduct training	01/2021
Upload Fillable PDF to Instrument	10/2020	Conduct nonresponse follow-up phone calls	01/2021 - 07/2021
Create Input File	10/2020	Print letters/web flyers/mailing labels for 2nd mailing	01/2021
Re-test Online Data Collection Tool	10/2020	Conduct 2nd nonresponse follow-up mailing	02/2021
Instrument moved to production	10/28/2020	Closeout	09/2021
General Processing System (GPS)		Tabulation/Analysis
Kickoff Meeting	07/2020	Imputations
Bi-weekly meetings	continuous	Develop specifications	08/2021
Updates to Application: includes all updates, presentation, and edits that were documented on specifications	07/2020	Program specifications	09/2021
Requirements: Submit Crosswalk	09/2020	Generate and review imputations	10/2021
Specifications for mandatory updates, enhancements and corrections submitted	09/2020	Tables/Estimates
Test Output transfer from Centurion to GPS	10/2020	Develop specifications	09/2021
Test transfer from Processing App to Database	10/2020	Program specifications	09/2021
Test Processing application (GPS) includes testing usability, functionality and edits	11/2020	Generate and review tables	10/2021
Data Review, Analysis & Editing	11/2020 - 09/2021	Data File Preparation
Data Collection		Review/correct data	09/2021
Mailout Preparation		Prepare preliminary file	09/2021
Submit Draft CJ-15 form to CENDOCs	04/2020	Deliver file to ESMPD	10/2021
Receive updated mailout letter from OJJDP	06/2020	Prepare documentation package	11/2021
Final Approval of CJ-15 Form	07/2020	Deliver final file to NIJ/OJJDP	12/2021
Order mailout materials (envelopes, supplies, etc.)	07/2020	Archive Data	01/2022
Receive letter approval from sponsor	08/2020	Evaluation
Print letters and web flyers	09/2020	Evaluate survey operations	continuous
Send mailout specification	09/2020	Develop recommendations	continuous
Send mail file to NPC	10/2020
Questionnaire printing	10/2020	* Subject to OMB approval.

3. Methods to Maximize Response

NIJ, OJJDP, and the Census Bureau are committed to achieving a high response rate and high-quality data. Using the number of in-scope facilities as a base, the JRFC facility response rate is typically between 83 and 95 percent.

Historically, a small proportion of in-scope facilities neither return JRFC forms nor respond to telephone calls from the Census Bureau requesting that the institution participate in the census. In 2016, the most recent JRFC collection with final data, there were a total of 2,173 “in-scope” facilities in the universe.^⁴ A total of 195 facilities did not respond and 58 facilities reported only critical items, meaning the 2016 JRFC had a response rate of 88.4 percent.^⁵ The Census Bureau imputed records for the nontribal facilities in the 50 states and the District of Columbia that were refusals. In other cases, some facilities were not able to provide all of the information requested and these data are imputed from complete records to fill in incomplete records. Techniques to maximize response rates and nonresponse adjustment procedures are summarized below.

NIJ will use the following techniques to maximize response:

Electronic reporting in a manner acceptable to the respondent (e.g., in 2018, the Census Bureau continued the use of a user friendly online reporting mechanism that has yielded good results);
Streamlined forms and clear response instructions (e.g., the addition of facility type definitions noted under Section A.15);
Continued support from the Census Bureau through a toll-free number to answer any questions from respondents that arise;
Continuous contact with respondents through e-mail and U.S. mail (see the schedule for mailout and reminder notices, above); and
Call-back procedures that continue until data closeout in May/June of the following year.

Notably, based on feedback from respondents and analysis of JRFC paradata, the Census Bureau implemented enhancements to the functionality of its online data collection system to improve respondent experiences, including: adding comment boxes to each section, adding a filter to the review pages, and adding totals to the edit message to help respondents correct those responses that do not sum as expected.

Additionally, staff at the Census Bureau have reported that facilities may be more likely to respond if they are able to see the importance of their data in understanding national trends, and those respondents with confidentiality concerns may be reassured that their data are only reported in the aggregate. Accordingly, in an effort to demonstrate to facilities how their data is used by policymakers and the public, an OJJDP Data Snapshot titled Service Availability Increased in Juvenile Residential Placement Facilities was included in the October 2018 mailout of the JRFC instrument (See Attachment K). Non-response follow-up by mail included a flier highlighting online data resources and publication advisory summarizing the most recent Juvenile Residential Facility Census, 2016: Selected Findings bulletin (See Attachment J). This technique is also planned for the 2020 JRFC data collection which will include a Data Snapshot and/or bulletin from the 2018 JRFC.

NIJ and OJJDP anticipate this ongoing effort to engage respondents will continue to yield high response rates. In the best of all possible scenarios, statistical estimation would not be required as this is a census. However, given the inevitable facility nonresponse and item nonresponse, NIJ and OJJDP will work with the Census Bureau, to ensure valid and reliable procedures to estimate the population characteristics.

Key Item Responses

The JRFC and CJRP are sent to juvenile residential facilities in alternating years. Prior year data exist for most facilities, and researchers and policy makers are interested in longitudinal trends. The key items collected in the JRFC are:

Persons assigned to beds,
Persons assigned to beds age 21 or older,
Persons under age 21 assigned to beds,
Persons under age 21 assigned to beds for offenses,
Persons under age 21 assigned to beds for reasons other than offenses,
Total number of standard beds,
Total occupied makeshift beds.

In addition to tracking the key item response rates over time, the Census Bureau also tracks the total quantity response rates over time. The total quantity response rates for 2016 was more than 90 percent, with the exception of the occupied makeshift beds. Very few facilities report occupied makeshift beds, and very few facilities had nonzero values imputed for occupied makeshift beds.

Nonresponse Adjustments

The 2020 JRFC is a census of all the juvenile residential facilities in the United States and its territories. No sample weights will be used in the data collection. As noted above, it is expected that a small proportion of in-scope facilities will not respond to the census and some facilities will not respond to all items. Overall, unit and item-level response rates are expected to remain sufficient for producing national and state-level estimates of juvenile offenders in residential placement. Nevertheless, it is expected that missing values will be imputed for both unit and item nonresponse.

Imputations

The imputation process starts by pulling forward prior year data for non- responding facilities due to refusal, responding facilities that only submitted critical items, and responding facilities who completed more than critical items, but did not complete all items (missing data for tribal and territorial facilities are not imputed). Questions for which data are pulled forward include, but are not limited to, facility ownership, who operates the facility, physical layout of the facility, and on-site residential treatment. Next, imputation groups are created. The various imputation groups include, but are not limited to, collapsed facility types, a flag to indicate if each record needs imputing or not, and geographic groups including divisions and regions. After imputation groups are created and reviewed, several calculations are performed within each imputation group. Growth rates are calculated for the donors in each imputation group. Next, ratios, median values, and most commonly reported answers are calculated for selected data items. After performing calculations within each imputation group, population counts imputed, then other missing data items are imputed. Finally, data quality checks are run to review values, check that all the flags are properly set and all imputed fields have valid values.

Additional details on unit and item level nonresponse and nonresponse adjustments are available in methodology reports produced by the Census Bureau. The most recent report from 2016 is included in Attachment N.^⁶

4. Test of Procedures or Methods

There are no changes to the forms or data collection methods at this time. However, in the future, any significant changes to the form would require that NIJ and OJJDP obtain feedback from state, local, and private agencies to ensure new items, definitions, and counting rules are clear across jurisdictions. In addition, OMB approval would need to be obtained, which includes an outside review of the survey. Consistently high response rates in recent JRFC and CJRP data collection cycles suggest that that the planned procedures for the 2020 CJRP will yield similar or improved results.

The Census Bureau and OJJDP originally tested the data collection procedures in a field test that began in October 1998. This test included a sample of 500 facilities. It was not a random sample but rather a sampled based on facility type and facility size. The combination of these variables created a series of strata. The goal was to test the procedures on each stratum and to modify the data collection procedures as necessary. The results of this test led to several changes to the instrument and to the collection methods; however, these changes were more cosmetic rather than substantive. The content of the survey did not require changing, but the Census Bureau learned much about how to administer this type of survey to the facilities included.

Since the first full administration of the JRFC in 2000, OJJDP has worked with the Census Bureau to establish and maintain appropriate statistical procedures for the data files. As part of the normal procedures for each CJRP file, the Economic Statistical Methods Division (ESMD) of the Census Bureau analyzes the quality of the data and develops methods for imputing for facility nonresponse and item nonresponse. ESMD provides a detailed report for each file. This report, described above, demonstrates the quality of the data collection efforts and the procedures the Economic Reimbursable Surveys Division uses to collect the CJRP data.

In addition, following the 2016 data collection and during the 2018 collection, the Census Bureau reviewed survey performance through a variety of methods including analyzing mode of response consistency, web respondent paradata, respondent comments and text responses to questions that include an “other, specify” response option, and consulting with a sample of respondents (fewer than 10) to better understand their thoughts on the survey content, instruments, and burden. The results of the consultation indicate that the majority of the respondents interviewed are able to provide the data requested for JRFC and CJRP. The respondents stressed the importance of identifying the correct contact and indicated that the data request does not always reach the correct person. Staff turnover continues to be a challenge with continuity of reporting data. Lack of time was also frequently cited as a challenge to providing all the requested data. Recommendations based on the consultations include: contacting respondents in advance of the data collection via a calendar invitation and post card; beginning nonresponse follow-up earlier in the survey cycle.

In April 2020, NIJ, OJJDP, and the Census Bureau decided to add new questions relating to the Coronavirus (COVID-19) pandemic to the 2020 JRFC instrument to capture the impact of this public health emergency on juvenile facilities. Several outside experts provided comments and suggestions to inform the design and content of the questions. In addition, the Census Bureau conducted cognitive interviews with nine eligible respondents to investigate how the questions performed, data availability, question wording and flow, and estimated burden.

As noted in Supporting Statement A, in fiscal year 2018, OJJDP made a separate award, now managed by NIJ, to assess and improve the data collection instruments and methodologies currently used in the JRFC and CJRP to generate useful, timely, and reliable national and state-level statistics on juveniles held in out-of-home placement and the facilities in which they are held. To achieve this goal, RTI International will collaborate with NIJ, OJJDP, and an expert panel to evaluate the content and methods used in the JRFC and CJRP and develop new approaches that will augment existing gaps in the current instruments and data collection designs. The project team will design a pilot test, which will be submitted for OMB review and approval (anticipated in 2020), to evaluate the proposed approach and determine to what extent it achieves the goal of generating more useful, reliable, and timely data. NIJ and OJJDP expect these improvements will be implemented in subsequent data collections cycles (following the current requested extension period).

5. Consultation Information

Presently, OJJDP funds an Interagency Agreement (IAA) with the Economic Reimbursable Surveys Division at the Census Bureau to perform data collection and to maintain the data file and address lists. This IAA also funds the imputation activity related to the JRFC file.

Relevant Contacts:

Benjamin Adams

Social Science Analyst

National Institute of Justice

(202) 616-3687

Nicole Adolph

Chief, Data Acquisitions Branch