Supporting Statement B - CIP

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Understanding Judicial Decision-Making and Hearing Quality in Child Welfare: Descriptive Study of Child Welfare Courts

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

September 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Christine Fortunato

Alysia Blandon

Part B

B1. Objectives

Study Objectives

The objectives of this information collection request are:

1. To understand child welfare court processes related to timing of initial child welfare hearings, frequency of review hearings, and use of pre-hearing conferences,

2. To understand characteristics of Court Improvement Projects (CIP) hearing quality and judicial decision-making projects, and

3. To determine CIPs’ interest in participating in a future research study.

This descriptive information will be used to inform a future research study or studies on how judicial decision-making and hearing quality are related to case progress and case outcomes for children in child welfare. The current request is the first phase of a future research study or studies. These activities will inform site selection, any potential pre-testing or feasibility, as well as further study design(s), which will be documented in a later information request package.

Generalizability of Results

This study is intended to present internally-valid description of select child welfare court processes and CIP projects in chosen sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The information to answer the research questions listed in A2 will be collected through the CIP Administrator Web Survey and the CIP Administrator Follow-Up Telephone Interview. Each of these data collection activities are designed to collect the minimum information necessary to allow the study team to understand the variation of CIP projects and court processes. This information will be used to assess what particular study design options will be feasible given the structure of a range of CIP projects.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.  

B2. Methods and Design

Target Population

Court improvement project grants are awarded to the highest court systems in all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. The target population for our study is the 53 CIP Administrators.

Sampling

The CIP Administrator Web Survey is a census survey of all 53 CIP Administrators. A census survey was selected because of the relatively small number of CIP Administrators and the desire to capture the full range of variation in court practices across the country and in the and projects reported by CIPs that may be missed if a sample were used. Using a sample would miss important examples of how child welfare courts function, therefore, limiting the information available for a future study of child welfare courts.

The CIP Administrator Follow-Up Telephone Interview will only be conducted with the subset of CIP Administrators for whom:

1. More information is needed about responses provided in the CIP Administrator Web Survey, and/or

2. Clarity is needed about information that was provided in the CIP self-assessment.

B3. Design of Data Collection Instruments

Development of Data Collection Instrument(s)

CIP Administrator Web Survey

The CIP Administrator Web Survey was developed by identifying information needed by the study team that was not included in the annually reported CIP self-assessments.¹ The survey includes questions necessary to engage study sites for a future research study that are not readily available from other sources. Study objectives addressed by the survey are listed in the table below.

CIP Administrator Follow-Up Telephone Interview Guide

This guide was developed by including potential open-ended questions that allow CIP Administrators to expand upon information provided in the CIP self-assessments and the CIP Administrator Web Survey. The actual follow-up questions will only include those needed to get the specific information that is needed for a specific CIP. The follow-up telephone interview minimizes the chance of survey measurement error by providing an opportunity to ask clarifying questions about information provided on the survey. Study objectives addressed by the interview guide are listed in the table below.

Both instruments were reviewed by two former CIP Administrators; Christine Kiesel and Katherine Malzahn-Bass. Ms. Kiesel and Ms. Malzahn-Bass reviewed written copies of the CIP Administrator Web Survey and CIP Administrator Follow-Up Telephone Interview Guide and provided written feedback regarding readability and length. They also participated in a field test of both instruments by completing the survey using a computer and participating in a mock-telephone interview with project staff. Afterwards, they provided feedback by phone about the usability of the survey, length of time to complete the survey, satisfaction with the length of the interview, and other thoughts about how to improve the survey and interview protocols. The study team made needed revisions based on their feedback.

Data Collection Instrument	Related Study Objective(s)
CIP Administrator Web Survey	1) To understand child welfare court processes related to timing of initial hearings, frequency of review hearings, and use of pre-hearing conferences. 3) To determine CIPs’ interest in participating in a future research study.
CIP Administrator Follow-Up Telephone Interview Guide*	1) To understand child welfare court processes related to timing of initial hearings, frequency of review hearings, and use of pre-hearing conferences. 2) To understand characteristics of CIP hearing quality and judicial decision-making projects.

* The interview will only be conducted when information is not available from the CIP Self-Assessment or from the CIP Administrator Web Survey.

B4. Collection of Data and Quality Control

CIP Administrator Web Survey

Each month, all CIP Administrators attend a mandatory call (“All Call”) hosted by the Capacity Building Center for Courts (CBCC). During one of these calls Scott Trowbridge (Children’s Bureau Child Welfare Program Specialist and federal program officer for the CIPs) will let the CIP Administrators know about the project by reading the survey recruitment script (see Appendix C). The script will notify Administrators that they will receive an email invitation (see Appendix D) from Project Director Anne Fromknecht to participate in the CIP Administrator Web Survey. To ensure quality control, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per CIP is submitted. This will also allow the study team to easily track response rates. The email invitation will also include a link to a project description to provide more background information. A letter of support from Scott Trowbridge, Children’s Bureau Child Welfare Program Specialist, encouraging participation and supporting our research study will be attached to the email invitation (see Appendix E). Mr. Trowbridge is the federal program officer for the CIP and is well known and respected by the CIP community.

At the end of week 2, the study team will individually email all CIP Administrators who have not yet responded (see Appendix F). At the end of week 4 the study team will call all CIP Administrators who have not yet responded and offer an additional week to respond, if needed (see Appendix G). If we cannot reach the CIP Administrator by phone, we will resend the email message with the deadline extended (see Appendix F). The survey will be open for 4 weeks, unless the CIP Administrator requests additional time to complete the survey.

CIP Administrator Follow-Up Telephone Interview Guide

After the CIP Administrator Web Survey data is reviewed and processed, the study team will identify which CIP Administrators need to be interviewed. Specifically, when (1) More information is needed about responses provided in the CIP Administrator Web Survey related to variation in court processes in their state and/or (2) More information is needed about information provided in or missing from the CIP self-assessment. The contractor will conduct the follow-up interviews by phone when the CIP Administrator agreed to be contacted for additional follow-up on the CIP Administrator Web Survey (see question 6 on Instrument 1).

Selected CIP Administrators will be emailed an interview request (see Appendix H) that asks them to respond within 2 weeks. At the end of week 2, the study team will call each invited CIP Administrator who has not yet responded to ask whether they want to participate (see Appendix I). If we cannot reach the CIP Administrator by phone, we will resend the invitation email with the request to respond extended 1 week (see Appendix H).

Interviews will be scheduled at a mutually convenient time. All interviewers will complete training with the Co-Principal Investigators. One member of the study team will conduct the interview following the interview guide (see Instrument 2) and another will take notes. Permission will be requested to record the interview to make sure that all responses are accurately captured. Office of Planning, Research, and Evaluation (OPRE) federal staff will join the interviews as their schedules allow. After each interview, the notetaker will clean and save the notes, referring to the interview recording as needed. All interviews will be completed in the 3 months following the close of the CIP Administrator Web Survey.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

CIP Administrator Web Survey

The universe of respondents for the CIP Administrator Web Survey is 53 CIP Administrators. Response rates for the CIP Administrator Web Survey will be calculated by dividing the number of completed surveys by 53. Other survey administrations of similar length with CIP Administrators conducted by the study team have yielded 72 percent response rates. Given our announcement during a CIP All Call, an endorsement from Scott Trowbridge (Children’s Bureau Child Welfare Program Specialist and federal program officer for the CIPs), current and historic working relationships with our study team, and extensive follow-up efforts we estimate a response rate of 80 percent for the CIP Administrator Web Survey.

CIP Administrator Follow-Up Telephone Interview

Based on previous review of CIP self-assessments and data collection with this population, we estimate that follow-up interviews will be required with up to 45 CIP Administrators. The interviews are not designed to produce statistically generalizable findings. Response rates will not be calculated or reported.

NonResponse

As described above, we are planning extensive follow-up activities to ensure that respondents have every chance to participate. A list of the reasons why CIP Administrators decline to participate in the survey and/or interview will be maintained and regularly reviewed during data collection to determine barriers to participation. Since our data collection requires only a computer with internet connection and a telephone, the biggest barrier to participation will likely be lack of time, particularly given the challenges presented by the COVID-19 pandemic. To mitigate this barrier we will consider extending data collection, offer times outside of standard work hours for interviews to accommodate respondents who may be working non-standard schedules, and allow the survey and interview to be completed by a CIP staff member designated by the CIP Administrator.

B6. Production of Estimates and Projections

All findings from the web survey and interview are for internal ACF purposes only. Data collected will not be used to make population estimates either for internal use or dissemination. For the web survey, we will produce descriptive statistics to understand the variation in court practices and for informing a future study or studies. Data from the CIP Administrator will be used to inform a future study or studies A study design options report is under review by ACF. Once a study design is selected, the study team will do pretesting to assess the feasibility of the protocols and measures and then conduct the selected study or studies.

B7. Data Handling and Analysis

Data Handling

To mitigate errors, survey responses will be extracted directly from Qualtrics into an Excel database. The study team will cross-reference the number of survey entries in the extracted Excel database with the raw data in Qualtrics before analysis. Additionally, all interviews (upon participant agreement) will be audio recorded and transcribed before analysis is completed. For participants who do not consent to being audio recorded, a note taker will be used to ensure responses are recorded.

Data Analysis

Data from the CIP Administrator Web Survey and the CIP Administrator Follow-Up Telephone Interview will be combined with information from the Excel spreadsheet summary of CIP Self-Assessments to understand the timing of initial court hearings, frequency of review hearings, use of pre-hearing conferences, and projects related to judicial decision-making and hearing quality in courts across the country. Plans for analysis of data collected from instruments included in this request are provided below.

CIP Administrator Web Survey

Descriptive statistics will be conducted. For example, frequencies of court practices (e.g., CIPs that hold review hearings more frequently than every 6 months). This will allow the study team to identify CIPs by characteristic and or groups of characteristics (e.g., all CIPs that video record hearings and hold review hearings more than every 6 months).

CIP Administrator Follow-Up Telephone Interview

Transcriptions of recorded interviews will be coded in Dedoose, qualitative data analysis software. Transcripts will be deductively coded using a codebook developed based on the research questions. After coding data into codes based on research question, sub-coding will be completed to identify patterns within each research question.

Data Use

Data from this information collection will be internally used by ACF to inform a future study or studies and for future planning purposes. Information from this data collection may be securely shared with qualified researchers to help guide future research and support program improvement.

B8. Contact Person(s)

The information for this study is being collected by James Bell Associates and consultants Dr. Sophia Gatowski and Dr. Alicia Summers on behalf of ACF. Co-Principal Investigators Dr. Sophia Gatowski (sgatowski@ymail.com) and Dr. Alicia Summers (alicia.d.summers@gmail.com) and Project Director Anne Fromknecht (fromknecht@jbassoc.com) led development of the study design and data collection protocols and will oversee collection and data analysis.

The agency responsible for receiving and approving contract deliverables is:

The Office of Planning, Research, and Evaluation (OPRE),

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services

The Federal project officers for this project are Christine Fortunato and Alysia Blandon.

Attachments

Instrument 1_CIP Web Survey

Instrument 2_ CIP Follow-Up Telephone Interview Guide

Appendix A_CIP Self-Assessment

Appendix B_CIP Interview, Consent Form

Appendix C_CIP Web Survey, All Call Script

Appendix D_CIP Web Survey, Email Invitation

Appendix E_CIP Web Survey, Scott Trowbridge Letter

Appendix F_ CIP Web Survey, Non-Responder Email

Appendix G_ CIP Web Survey, Non-Responder Telephone Script

Appendix H_CIP Interview, Email Invitation

Appendix I_CIP Interview, Non-Responder Telephone Script

1As noted in Supporting Statement A, CIP self-assessments (OMB Control No: 0970-0307, expiration date: 11/30/2022) are completed annually by all 53 state CIPs and submitted to the Children’s Bureau and the Capacity Building Center for Courts (CBCC), the program’s technical assistance provider.

8