SCWS-Supporting Statement-B_fnl_9.8.23

Strengthening Child Welfare Systems to Achieve Expected Child and Family Outcomes

Evaluation

OMB Information Collection Request

0970 – New Collection

Supporting Statement Part B –

Statistical Methods

September 2023

Submitted By:

Children’s Bureau

Administration for Children and Families

U.S. Department of Health and Human Services

1. Respondent Universe and Sampling Methods

The Children’s Bureau awarded grants to five agencies for the Strengthening Child Welfare Systems (SCWS) to Achieve Expected Child and Family Outcomes cluster. The target population for the study will be all five SCWS grant recipients.

The SCWS Survey will be administered using a purposive sample of individuals that have been involved with the grant—respondents in the purposive sample will include SCWS grant recipient staff, project evaluators, and community agency partners. A purposive sample was selected because the study is focused on understanding (1) implementation processes and the impact of grant interventions and (2) examination of whether and the degree to which grant recipients were able to address common Child and Family Services Reviews (CFSR) outcomes. Therefore, respondents who have participated in the implementation and evaluation of grant strategies and interventions are needed. Due to the longstanding working relationships between respondents in the sample and our study team, combined with planned follow-up efforts, we estimate a response rate of 80 percent for the survey. We expect to administer the survey to about 12 respondents from each project for a total of 60 individuals.

As with the survey, the SCWS Focus Groups will also be a purposive sample of individuals that have been involved with the SCWS grants. We expect to recruit about 6 participants from each project for a total of 30 participants for the focus groups.

Table 1. Sample

Instruments	Role	Number of Respondents per Grant Recipient
SCWS Grant and Partner Survey	Grant recipient staff (project directors, managers, and staff)	5-6
	Project evaluator(s)	1-2
	Community agency partners (agency directors/managers and staff)	5-6
SCWS Grant Focus Groups	Grant recipient staff (project directors/managers and staff)	2-3
	Evaluator(s)	1-2
	Community agency partners (agency directors/managers and staff)	2-3

2. Procedures for the Collection of Information

The study team will communicate with the grant project director and/or manager to request contact information for grant staff, evaluator(s), and community agency partner staff who have knowledge of and have participated in the implementation and/or evaluation of grant strategies and interventions to be contacted to participate in study data collection activities. The study team will determine with the grant project director/manager which staff would have the requisite knowledge and experience to be contacted to participate in the focus group and those who would have the requisite knowledge and experience to be contacted to complete the survey.

Once it is determined which staff will be invited to participate in each data collection activity and contact information has been obtained, the study team will send the appropriate email invitation to each potential respondent.

• For the SCWS Grant and Partner Survey, each email invitation will include a unique survey link (generated by Qualtrics, the survey platform) so that only one response per respondent is submitted. This will also allow the study team to easily track response rates. After two weeks, the study team will send a follow-up email to those who still need to complete the survey (see Appendix B). At the end of week 4 we will resend the email message to those who have still not responded with an extended deadline (see Appendix B). The survey will be open for 4 weeks unless the respondent requests additional time to complete the survey.

• For the SCWS Grant Focus Group, the email invitation (Appendix C) will be re-sent after 7 business days to any non-respondents to the initial invitation to schedule the focus group. If 50% of invitees either decline or do not respond the study team will communicate with the grant project director/manager to review the list of non-respondents/declines to determine if alternates should be contacted for participation.

3. Methods to Maximize Response Rates and Deal with Nonresponse

As noted above, several attempts will be made to follow-up with non-respondents, or a process will be conducted to determine if alternate participants should be identified. Given the close working relationship between the study team and the grant project directors/managers, as well as the fact that participation in the study was stipulated by the funding agency (the Children’s Bureau) as a grant requirement, the study team does not anticipate a large number of non-responses.

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. The study team will, however, track refusal rates and refusal demographics (where possible), to gain an understanding of potential patterns in data collection participation and refusal.

4. Test of Procedures or Methods to be Undertaken

The SCWS Grant and Partner Survey and SCWS Grant Focus Group protocol were pilot tested by two JBA staff members with knowledge of the grant recipients and one staff person from another organization (ICF, Inc.) who works with the SCWS grant recipients on implementation activities but who is otherwise not involved in this study. These three individuals pilot tested the survey and participated in a mock focus group. This pilot process was intended to ensure that the content of the survey instrument and focus group protocol is clear and to estimate the time (in minutes) required to complete each of the data collection activities.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The information for this study is being collected by Nicole Miller and Julie Murphy, James Bell Associates, on behalf of the Children’s Bureau. Ms. Miller (703-247-2625) and Ms. Murphy (703-842-0955) led the development of the study and data collection protocols and will oversee collection and data analysis.

The agency responsible for receiving and approving contract deliverables is:

Children’s Bureau (CB)

Administration for Children and Families (ACF)

U.S. Department of Health and Human Services

The Federal project officer for this project is Trilisza Trent.