Supporting Statement B - OCS AHSSD Emergency Approval 1.2.24

OMB Information Collection Request

0970 - NEW

Supporting Statement Part B –

Statistical Methods

February 2024

Submitted By:

Office of Community Services

Administration for Children and Families

U.S. Department of Health and Human Services

1. Respondent Universe and Sampling Methods

For the Arizona Self-Sufficiency Matrix and the Service Receipt Questionnaire, the respondent universe is intensively case-managed adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration. The Office of Community Services (OCS) hopes to collect Matrix and Questionnaire data from the majority of these residents. As this is a demonstration program, there is a wide variety of the anticipated number of case-managed residents by different grant recipients, with some sites planning to only offer intensive services to a handful of residents and other sites planning to case-manage close to a hundred residents. Due to this variety and the fact that most grant recipients are already collecting the Matrix, OCS is not planning to stratify or take a random sample of intensively case-managed residents.

OCS estimates that the maximum number of possible residents who may receive intensive case-management is 700 individuals, based on conversations with the grant recipients and a review of the grant recipients’ applications. By focusing only on the universe of residents who are receiving the highest intensity of services and who are in regular contact with their caseworkers, OCS anticipates a high overall response rate and measurable change on the Matrix and Questionnaire. These Matrix and Questionnaire results will not be generalizable to all the residents living in affordable housing funded by the demonstration program.

OCS plans to conduct the Matrix and Questionnaire at two separate time points (baseline and follow up) to evaluate change over time. At baseline, OCS anticipates a response rate of 85% for the two instruments. For the second time point, OCS anticipates that some residents may move away from the affordable housing community or finish receiving services, and thus the response rate for the follow-up is anticipated to be 75%.

For the interviews with program staff (program directors and caseworkers) and the focus groups with residents, OCS anticipates a response rate approaching 100%. All grant recipients have expressed a willingness for their staff to be interviewed and have offered estimates of the total number of staff currently involved with the program (36 individuals, i.e., 18 program directors and 18 caseworkers). By interviewing several different staff members from each grant recipient organization, OCS hopes to obtain a holistic understanding of perceived program impact and implementation. Most of the grant recipients have also previously conducted focus groups with their residents and have stated that it would be fairly easy to convene groups of 5-7 residents to discuss their experiences of the program. To accommodate OCS staff constraints and to be able to conduct the majority of the focus groups in person, OCS plans to conduct focus groups at only half of the sites. OCS plans to work with the grant recipients to recruit residents with diverse experiences of the program, including intensively case-managed residents and residents who only briefly engaged with offered services. OCS hopes to include residents from a variety of backgrounds and lived experiences in the focus groups (including people of different genders, racial and ethnic backgrounds, languages spoken, marital and family status, and age).

OCS anticipates that 100% of the grant recipients will complete the Quarterly PPRs and Semi-Annual Mandatory Reports. For the Mandatory Reports, many of the grant recipients have indicated that they are already tracking much of the information proposed to be requested. OCS anticipates that only around a third of the grant recipients will complete the Semi-Annual Optional Reports. The questions in the Optional Reports are based on information that some of the grant recipients indicated in their applications that they were planning to collect.

Instrument	Respondent Universe	Intended Sample	Response Rate
Self-Sufficiency Matrix	Intensively case-managed adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	Entire respondent universe (maximum: 700 residents)	85% for baseline; 75% for follow-up
Service Receipt Questionnaire	Intensively case-managed adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	Entire respondent universe (maximum: 700 residents)	85% for baseline; 75% for follow-up
Resident Focus Group	All adult residents living in affordable housing funded by the Affordable Housing and Supportive Services Demonstration	25 residents chosen by grant recipients to represent	100%
Quarterly PPR Collections	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	All 9 grant recipients	100%
Semi Annual Report MANDATORY	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	All 9 grant recipients	100%
Semi Annual Report OPTIONAL	All grant recipients funded by the Affordable Housing and Supportive Services Demonstration	All 9 grant recipients	33%
Interviews with Program Directors	All program directors overseeing programs associated with the Affordable Housing and Supportive Services Demonstration	All program directors overseeing programs associated with the Affordable Housing and Supportive Services Demonstration (18)	100%
Interviews with Caseworkers	All caseworkers working on the Affordable Housing and Supportive Services Demonstration	All caseworkers working on the Affordable Housing and Supportive Services Demonstration (18)	100%

2. Procedures for the Collection of Information

The majority of the grant recipients are already administering the Arizona Self-Sufficiency Matrix and collecting the results in their internal information databases. For the grant recipients with pre-existing collection procedures, OCS is requesting that the recipients report the results of the Matrix from their databases at each requested time point for the subset of intensively case-managed residents. Residents will receive a gift card for each completion of the Matrix, to ensure a high response rate and help ensure quality data with utility for the government. For the two recipients that are not currently using the Matrix, OCS will create an online version of the Matrix and meet with the caseworkers to discuss collection procedures. The Matrix is typically conducted via a collaborative conversation between a caseworker and a resident (see Attachment D). The case-managed residents will each be assigned a unique identifier for the Matrix, so that OCS can evaluate change over time at the individual level, at the grant recipient level, and for the demonstration program collectively.

For the Service Receipt Questionnaire, OCS plans to create an online survey that can be quickly filled out during a meeting between a resident and a caseworker. OCS will request that this Questionnaire be filled out around the same time as the corresponding Matrix (ideally in the same meeting, but alternatively at another meeting during the same quarter). This Questionnaire will also have the unique identifier attached, so that OCS will be able to not only assess if the program is increasing access to services, but also analyze if there are particular services associated with change on the Matrix.

For the Resident Focus Groups, OCS plans to send Attachments B and C to the grant recipients and ask them to recruit residents who have had varying levels of engagement with the program. OCS staff will then travel to the sites and conduct the focus groups in the affordable housing complexes where the residents live. OCS will work with the grant recipients to ensure that participants have adequate childcare (and transportation if needed). Recipients will be given a gift card at the close of the discussion. The focus groups will be recorded, and OCS staff will then code the discussions to identify how residents experienced the program.

OCS plans to conduct the interviews with program directors and caseworkers virtually. OCS will record the virtual meetings and then qualitatively code the interviews to assess implementation and perceived best practices.

For the Mandatory Semi Annual Reports, the grant recipients will use their existing databases to track service delivery, referrals, and demographics of residents. The recipients will then provide total counts of these indicators in the Mandatory Report and will also list organizational partners and attributes of the housing community. The Optional Report is a place for grant recipients to detail other activities or metrics that they may be tracking on resident outcomes. The grant recipients will supplement these Reports with the narrative PPRs, which will include information about the progress of implementing the program.

3. Methods to Maximize Response Rates and Deal with Nonresponse

The primary procedures to deal with non-response on the Arizona Self-Sufficiency Matrix and Service Receipt Questionnaire are to limit the respondent universe to intensively case-managed residents and to offer a token of appreciation for each completion of the Matrix. By offering a gift card at each time point, OCS anticipates that the response rate will remain high for all collections and that residents will answer all or most of the questions on the Matrix. OCS’ goal is for most of the submitted matrices to be fully complete (i.e., a numerical selection on the scale from 1 to 5 selected for all self-sufficiency domains). OCS is planning to primarily base analysis of the Matrix on surveys that were completely filled out, in order to reduce non-response bias in results. In addition, full completion of the Matrix is necessary to calculate the full self-sufficiency score for an individual (i.e., the sum across all domains). OCS also needs a high item response rate in order to have sufficient power to detect change over time from administration to administration. The Arizona Matrix was chosen in part because most of the grant recipients are already familiar with this instrument, and many of the residents may have encountered it previously, which should thus improve both response rates and data quality.

For the Resident Focus Groups, OCS plans to offer gift cards to ensure that both a diversity of people is included and that participants give rich, clear answers. By offering a gift card, respondents who might ordinarily refrain from participating (i.e., busy parents) are more likely to take part in the focus group. This will give OCS a more complete understanding of the experiences of residents. Furthermore, if respondents feel that their input is valuable, they will be more likely to offer full answers to each focus group question.

OCS’ main method to maximize response rates for the Program Director and Caseworker Interviews are to conduct them virtually, so that the respondents can choose any date or time that is convenient to them. By also interviewing two of each staff member (i.e., two caseworkers from each site), OCS will improve the information gained about each grant recipient.

For the Reports, OCS is requesting metrics that almost all of the grant recipients already collect or have previously indicated that they would like to collect, which OCS believes will lead to a high response rate.

4. Test of Procedures or Methods to be Undertaken

The Arizona Self-Sufficiency Matrix is a validated and reliable measurement tool. In a review of ten different instruments for assessing the self-sufficiency of homeless clients, Culhane et al. (2007) found that that the Arizona Matrix was the only tool that “met acceptable reliability and validity standards” (p. 12-21). The instrument has subsequently been successfully tested and is used by many cities, counties, and states to identify the needs of people receiving housing stability services and to predict outcomes for clients based on baseline Matrix characteristics. The majority of the grant recipients have already used the Arizona Matrix with their clients.

OCS also anticipates conducting an internal pilot of the Interview protocol with ACF staff serving as test respondents. If staff capacity permits, OCS may also perform an internal test of the Focus Group protocol.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Dr. Jessica Hale, Program Evaluator, Division of Community Discretionary and Demonstration Programs, Office of Community Services, Administration for Children and Families, jessica.hale@acf.hhs.gov, (202) 401-4835

Juliana Melara, Policy Advisor, Office of the Director, Office of Community Services, Administration for Children & Families, juliana.melara@acf.hhs.gov, 315-744-2490

Benjamin Wills, Program Specialist, Office of Community Services, Administration for Children and Families, Benjamin.Wills@acf.hhs.gov, 206-550-2471

Regine Adrien, Program Specialist, Office of Community Services, Administration for Children and Families, regine.adrien@acf.hhs.gov, 202-205-9741

Brontë Nevins, Community Development Project Support Specialist, Division of Community Discretionary and Demonstration Programs, Office of Community Services, Administration for Children & Families, bronte.nevins@acf.hhs.gov, 240-274-1442

References

Culhane, D.P., Parker, W. D., Poppe, B., Gross, K.D., and Sykes, E. (2007). Accountability, Cost-Effectiveness, and Program Performance: Progress Since 1998. 2007 National Symposium on Homelessness Research. Retrieved from: https://aspe.hhs.gov/sites/default/files/migrated_legacy_files//84166/report.pdf