S811PRA_OMB_PartB_11.27.2017 - clean

An Implementation Study to document the implementation of the PRA program based on interviews with key program administrators and review of PRA program documents and data;
An Impact Study to assess the effects of the PRA program on participants’ quality of life and care, housing and neighborhood, and utilization and access to health services and supports compared to three matched comparison groups. The impact study includes a Descriptive Analysis of PRA program outputs for all 28 Section 811 PRA grantees, based on HUD administrative data; and
An Economic Study to measure the costs of housing and supportive services provided by the Section 811 PRA program and to attempt to measure the costs of healthcare utilization potentially affected by the PRA program, to compare these costs to other HUD assisted housing programs with and without structured access to services, and to compare costs to measured impacts.

The data collection includes:

In-person surveys of:
- 240 Section 811 PRA program residents
- 240 Section 811 Project Rental Assistance Contract (PRAC) Program residents
In-person and telephone interviews with:
- 24 property owners and managers
- 48 service providers
- 12 public housing authority (PHA) staff and managers

The remainder of the study’s data collection methods involves secondary administrative data or primary data collection and highly-tailored discussions with representatives of the six grantee state housing agencies and their partner state Medicaid agencies, each of which do not involve standardized data collection with more than nine respondents.

This data collection effort does not employ statistical methods. The research team will select respondents for interviews and surveys purposively.

Selection of State PRA Programs

Through two rounds of funding, the Section 811 Supportive Housing for Persons with Disabilities Project Rental Assistance (PRA) Program has awarded grants to 28 states. In Phase II, the evaluation’s implementation, impact, and cost studies are focused on six states selected from 28 state grantees from the first and second rounds of PRA funding. The study also includes a descriptive analysis which analysis the administrative data of all 28 PRA grantees funded in both rounds.

The impact and cost studies rely on tenant-level data of residents living in PRA-funded units and PRAC-funded units. The six states were selected for participation in the evaluation based on the number of residents living in PRA-funded units expected to be assisted by PRA funds by the time the research team conducts the resident surveys, expected to be between September and December 2017. The number of units under lease and occupied by PRA residents is the most important consideration for choosing states for the evaluation. Leased units are a key milestone in the program. For the evaluation, leased units identify residents receiving assistance through the Section 811 PRA program and are the way to find the related entities – namely property owners and managers and service providers – who will be interviewed for the evaluation.

Exhibit B-1 presents the PRA grantee states with units leased to PRA residents as of March 2017, ranking the states by the number of units leased. The table includes both 2012 and 2013 rounds of PRA funding. As shown in Exhibit B-1, as of March 31, 2017, there were 381 residents living in PRA-funded units in 14 grantee states.

Exhibit B-1

PRA Grantee States by Number of Units Leased, March 2017

By Number of Units Leased

	States	Number of Units Leased	Percentage
1	Louisiana	86	22.6%
2	California	58	15.2%
3	Minnesota	55	14.4%
4	Washington	46	12.1%
5	Maryland	36	9.4%
6	Delaware	30	7.9%
	Subtotal	311	81.6%
7	Georgia	19	5.0%
8	Pennsylvania	15	3.9%
9	Montana	10	2.6%
10	Massachusetts	6	1.6%
11	Texas	6	1.6%
12	Illinois	5	1.3%
13	New Jersey	5	1.3%
14	Connecticut	4	1.0%
	Subtotal	70	18.4%
	Total	381	100.0%

Source: Section 811 PRA Quarterly Reports for quarter ending March 31, 2017.

The six states with the highest numbers of units leased have been selected for the Section 811 PRA evaluation: California, Delaware, Louisiana, Maryland, Minnesota, and Washington. As of March 2017, these six states had 311 units leased, representing 82 percent of the total PRA units leased across all grantees. Adding an additional state to the sample of states for the evaluation would only increase the number of PRA residents in the comparison sample by less than 20 (according to data from March 2017) while increasing the cost of field data collection. This study is Phase II of a multi-phase evaluation. The Phase I evaluation was focused on the 12 states and the first year of implementation of the 2012 grants. In selecting these six states for the Phase II evaluation, the study will build on the information collected in the Phase I evaluation, by further documenting the implementation of the Section 811 PRA program.

Identification of Section 811 PRA Resident Survey Respondents

The target number of PRA residents to be surveyed for the study is 240. As the number of potential PRA residents living in the six states is 311 as of March 31, 2017, the research team will attempt to recruit the universe of PRA residents in the six states. While the study team plans to outreach to all PRA residents, surveys will be conducted in person and it may not be economically feasible for researchers to visit the geographic locations where all residents live.

Identification of Section 811 PRAC Resident Survey Respondents

The target number of PRAC residents to be surveyed for the study is 240. Section 811 PRAC residents will be selected based on locations and characteristics of Section 811 PRA residents in each state. PRAC properties are designated to serve people with similar disabling conditions. Using HUD administrative data, the research team will select PRAC properties that are located near the leased PRA units and which serve people with similar disabling conditions as those reported for PRA residents. The locations of PRAC properties to where PRA units are located in each state will be determined individually for each state.

Exhibit B-2 shows the universe of PRAC resident in the six study states. A total of 360 PRAC residents will be recruited to complete the survey, approximately five percent of PRAC residents in the six study states. Based on results on initial testing of the resident survey with PRAC residents, the research team assumes a survey response rate of 66 percent.

Exhibit B-2

Number of Units Leased through the Section 811 PRAC Program in Selected States, April 2017

	States	Number of Units Leased
1	California	2,684
2	Delaware	362
3	Louisiana	1,091
4	Maryland	1,510
5	Minnesota	748
6	Washington	619
	Total	7,014

Source: HUD’s Tenant Rental Certification Systems

Identification of Section 811 Property Owner Respondents

Evaluators will select property owners to interview purposively using the location of where PRA residents live. The evaluation staff will work with the state housing agency to identify four property owners or managers per state for interviews, for a total of 24 interviews. Through March 31, 2017, the PRA grantees report 56 properties with units under contract for PRA subsidies in the six evaluation states. Properties will be selected based on the number of residents living in PRA units at the time of the interviews and to ensure geographic distribution.

Identification of Section 811 Service Provider Respondents

Evaluators will select service providers to interview purposively based on the number of PRA residents they serve and the type of services they provide. Up to eight service providers per state will be recruited for the evaluation, for up to 48 service provider interviews. Once PRA residents have been identified, evaluation staff will work with each state’s Medicaid agency to learn about the service provider organizations that serve the PRA residents. The universe of potential service providers will not be known until prior to field data collection. As PRA residents can select their own service providers, the number of potential service provider organizations who work with participants in the Section 811 PRA program may be quite large.

Identification of Section 811 Property Owner Respondents

In the Section 811 PRA grantee application process, states received extra points for obtaining commitments from one or more PHAs to leverage other housing units for people with disabilities. Twelve PHA respondents will be recruited to be interviewed. Exhibit B-3 shows the universe of PHAs that leveraged units for the PRA program. A total of 18 PHAs leveraged units in five of the six study states. The study team will recruit the universe of PHAs in California, Louisiana, Minnesota, and Washington, and will recruit four of the 10 PHAs that leveraged units in Maryland.

Exhibit B-3

Number of Public Housing Agencies that Committed HCVs for PRA Grant Application

	States	Number of PHAs that Committed HCVs
1	California	2
2	Delaware	0
3	Louisiana	3
4	Maryland	10
5	Minnesota	2
6	Washington	1
	Total	18

Source: Section 811 PRA Quarterly Reports, March 2017

Administration of the Survey

All six states that are the focus of the evaluation of the Section 811 PRA Program (California, Delaware Louisiana, Maryland, Minnesota, and Washington) received funding in the first two rounds of funding, and together, the units leased with PRA program funds in these six states make up 82 percent of all units leased with PRA program funds throughout the country, as of March 2017. Researchers will recruit the universe of PRA residents and will purposively identify PRAC residents to recruit to complete the resident survey. The evaluation team will also purposively select property owners/managers, service providers, and public housing authority (PHA) representatives for semi-structured interviews designed to learn about their experiences with and perspectives on the PRA program.

Selection of Interview and Survey Respondents

Section 811 PRA Residents

As of March 2017, there are 381 units leased in the Section 811 PRA Program, and 311 individuals in PRA units in the six evaluation states. The target number of PRA residents to be recruited to be surveyed for the study is 240, which is a large fraction of all of the leased units in the PRA program in the six evaluation states. As such, the research team will attempt to recruit the universe of PRA residents to complete surveys.

Section 811 PRAC Residents

Surveys are also planned for residents in the Section 811 Project Rental Assistance Contract (PRAC) Program. PRAC properties are primarily group homes or small apartment buildings (typically not more than 24 units.) We plan to survey no more than 10 residents within an individual PRAC property.

The sample for PRAC resident surveys will be based on HUD administrative data including the proximity of PRAC properties relative to the PRA properties where PRA residents live; the disabling condition of the residents that the PRAC property serves relative to the reported disabling conditions of PRA residents; and the resident’s time of entry into the Section 811 program. PRA residents moved into PRA-funded units beginning in January 2015 but most (86 percent of residents) did not move into PRA units until 2016. The evaluation team expects to conduct resident surveys between September and December 2017. Since most PRA residents will have fewer than two years participating in the PRA program at the time of the interview, the research team will select residents with similar program entrance times. Subject to obtaining medical record data, we will also identify PRAC properties where individuals have similar medical spending histories (2014-2015) and diagnoses to the sampled PRA participants.¹

Based on experience surveying similar populations, we assume a response rate of 66 percent. In total, the evaluation team will recruit 360 PRAC residents to obtain the target number of 240 completed surveys. Because the impact and cost studies need to control for geographic regions with similar services and rental market conditions, to the extent possible, we will select PRAC properties that are in the same city or metropolitan area where PRA residents live. If there are no PRAC properties or sufficient properties with enough residents that have similar characteristics (i.e., disabling condition, admission into the PRA program) in these areas, we will expand the selection criteria to geographic regions that are targets for the PRA property but have no or few PRA residents living in them.

Justification of Sample Size for PRA and PRAC Resident Surveys

The research team has assumed a small target sample size because of the limited number of PRA participants in the six sample sizes and because only modest improvements in power are achievable by expanding comparison groups while holding treatment sample fixed (the smaller sample is the limiting factor in power for comparisons of conditional means).

The primary analytic approach to selecting the PRAC comparison group will be matching on historic service utilization, demographic, and diagnostic information.

We will also use HUD administrative data (along with healthcare data) to construct the analytic sample for the study. The analysis sample will include four groups. Individuals living in Project Rental Assistance units represent the primary “treatment” group of interest. Three groups provide distinct counterfactual comparison groups. The first is composed of individuals in Section 811 PRAC units in the same geographic areas, who are selected to be similar to PRA residents in terms of diagnoses, health care utilization, disability types, and demographic characteristics such as age and gender. The second is composed of individuals with disabilities in other HUD-assisted housing (Nonelderly Disabled Vouchers, Housing Choice Vouchers, public housing, multifamily assisted housing) in the same geographic areas, also selected to be similar to PRA residents. The third comparison group is a business-as-usual group that includes similar non-elderly disabled individuals not assisted by HUD programs and who may be in any other housing situation (living with family, homeless, in market-rate housing, or in institutional settings). The study team will use HUD data to identify PRA, PRAC, and other similar HUD-assisted tenants with disabilities. We will match HUD data to healthcare data to create matched comparison groups based on demographic information, historic healthcare utilization patterns, diagnoses, and other factors.

Within coarsely matched samples, regression models will be estimated using a propensity-score-based weight to approximate the “propensity-score double robust” (PS-DR) method, using both a weighted sample and regression adjustment to determine the treatment effect on treated units. The smallest detectable effect sizes using PS-DR for sample sizes under 500 are not dramatically higher than the lower bound, given by a randomized control trial with equal-sized treatment and control groups. The power analysis demonstrates that a minimum target sample size of 400 (500) cases can detect differences as small as 37 (32) percent of a standard deviation in each outcome measure, compared with the theoretical lower bound of 28 (25) percent of a standard deviation (effect size detectable using random assignment with equal probabilities of treatment and control conditions).

The study assumes a target sample size of 480 residents in PRA and PRAC units and a minimum analysis sample of 400 cases. If the expected response rate is not achieved through the initial selection of 360 PRAC residents and the universe of PRA residents, the research team will select additional residents in targeted geographic areas until we obtain the minimum sample needed to conduct the impact and cost analyses (approximately 200 PRA residents and 200 PRAC residents). If necessary, the research team may need to select additional PRA residents to survey in January 2018 to meet the minimum sample of PRA residents.

Selecting Property Owners and Managers

Using property-level administrative data from HUD, the evaluation team will identify property owners and managers from the properties with Section 811 PRA units under contract as of the time of the scheduled administrative interviews (planned for September through December 2017). Form this list, the evaluation staff will work with the state housing agency grantee to select property owner organizations and specific staff to interview. Property owners will be selected to ensure geographic distribution of PRA units.

Selecting Service Providers

Up to eight service providers per state will be recruited for the evaluation, for a total of up to 48 service provider interviews. The selection of service provider agencies to interview will be purposive based on the type of services the organization provides, the caseload of PRA residents, and the location where the organization provides services. The research team will prioritize providers who are serving the largest number of PRA residents but the providers will not be chosen to match the PRA residents who will be interviewed for the resident surveys.

Evaluation staff will work with each state’s Medicaid agency to learn about the service provider organizations that serve PRA residents. Interviews will be conducted with managers of service providers, as well as direct service providers or case managers, to provide a broader perspective of service provision to PRA residents. Organizations will be selected to represent all the eligible populations of the PRA program in each state by type of Medicaid waiver or disabling condition of clients. Service providers will be selected to represent organizations that provide both transition supports and services to help individuals apply for and move into PRA-funded units and long-term services and supports. At least one of the service providers in each of the states will be an organization that provides transition services under the Money Follows the Person demonstration program and one serving people individuals who are experiencing homelessness or at risk for homelessness.

Selecting PHA Staff

In the Section 811 PRA grantee application process, states received points for leveraging other housing units for people with disabilities by obtaining commitments from one or more PHAs. The 2012 NOFA awarded points to applicants for setting aside a number of Housing Choice Vouchers (HCV) or other rental units specifically for extremely low-income non-elderly people with disabilities, and the 2013/2014 NOFA awarded points to applicants for the commitment of one or more PHAs to establish an admissions preference for the Section 811 target population. As shown in Exhibit B-3, 18 PHAs leveraged units in five of the six study states. The study team will recruit the universe of PHAs in California, Louisiana, Minnesota, and Washington, and will recruit four of the 10 PHAs that leveraged units in Maryland.

Maximizing the Response Rate and Minimizing Non-Response Risk

We expect high response rates from the PRA property owners and from the service providers and PHA staff selected for interviews.

Given the small number of PRA units expected to be under lease at the time the survey is fielded (expected to be September – December 2017), the evaluation team will interview as many PRA residents as feasible. As of September 2016, there were 232 residents in the PRA program, and as of March 2017, the number exceeded 300 (though because of lags in data entry in HUD’s data system, we only observe 226 PRA residents in the March 2017 administrative data for the six evaluation states). The study team anticipates reaching out to all eligible PRA individuals in the study states for screening and consent, although recruitment efforts will focus on locations within a reasonable travel distance to make efficient use of time on site.

Based on previous experience conducting surveys of similar populations, the expected response rate among PRAC residents is 66 percent.

Section 811 PRA and PRAC residents who take part in the one-time surveys will receive a $40 gift card to thank them for their time and to help defray any costs the respondent may incur to participate in the interview. The incentive amounts were determined based on the estimate of respondent burden for participating in the survey, the costs associated with participating the survey, and other studies of comparable populations and burdens. The resident survey instrument includes well-tested items used widely on similar populations, and cognitive testing has indicated respondents were willing to and capable of completing the survey.

With the goal of interviewing as many PRA residents as possible, PRA resident recruitment will be coordinated through property managers or service provider organizations. For PRAC properties, HUD Field Office staff will be asked to assist in identifying and making contact with PRAC property owners. For PRA properties, the research team will ask the state Medicaid agency in each state to provide contact information for service providers serving PRA residents. Advance letters for the PRAC property owners and PRA service provider organizations will describe the research in case residents come to them with questions about the survey.

Test of Procedures

The PRA and PRAC resident survey instrument includes well-tested items used widely on similar populations. The evaluation team has conducted cognitive testing with a convenience sample of nine Section 811 participants who meet the criteria for the target respondent population. Cognitive testing allowed the survey development team to test whether questions and response options are understandable, easy to answer, complete and address the intended constructs.

To minimize travel and associated costs, cognitive testing participants were recruited through property owners/managers in Massachusetts, where HUD’s contractor, Abt Associates, currently works and where the Section 811 PRA program is being administered. Abt staff identified PRA or PRAC property managers in Massachusetts through HUD’s online inventory of housing for people with disabilities and the iREMS database. Abt contacted property managers to help identify potential participants for the cognitive testing by providing Abt researchers with mailing addresses and phone numbers for participants when possible. Abt sent potential survey participants a recruitment letter explaining the research study. This letter was followed up with a phone call approximately a week later. Those who agreed to participate were scheduled to meet with Abt researchers who conducted the cognitive testing interview in person at the participant’s convenience.

An incentive of $75 was offered for participating in cognitive testing interviews. Cognitive testing was conducted in person in the participants’ homes or at a public place, and the testing took at most 90 minutes. Respondents were asked to complete the survey, and researchers asked a series of follow-up questions about the survey to ensure questions were understood and are measuring the intended constructs.

Individuals Consulted on Statistical Aspects of the Design

HUD has contracted with Abt Associates to conduct the data collection. The data collection procedures will be similar to those used in other surveys conducted by Abt Associates. The HUD Government Technical Representative (GTR) reviewed all the procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD GTR, Teresa Souza (202-402-5540) or the Abt Associates co-Principal Investigators, Gretchen Locke (617-349-2373) and Sara Galantowicz (617-520-2510).

1 The selection process for the analysis sample will also reweight PRAC residents to be as similar as possible to PRA residents in terms of diagnoses, health care utilization, disability types, and demographic characteristics, but this alignment of samples happens at the analysis phase, not the sampling phase.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Audrey Hanbury
File Modified	0000-00-00
File Created	2021-01-22