Final -X_7801 Rent Reform Demo OMB Part B

Supporting Statement for Paperwork Reduction Act Submission

Rent Reform Demonstration: 6-Year Follow-Up

OMB Control # 2528-0306

Part B. Justification

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

In 2015, eligible households at four MTW public housing agencies (PHAs) participating in the Rent Reform Demonstration – Lexington Housing Authority, Louisville Metropolitan Housing Authority, San Antonio Housing Authority, and District of Columbia Housing Authority were randomly assigned to either the New Rent Rules group (Program/Treatment group) or the Existing Rent Rules group (Control group). The total number of households enrolled in the study and included in the impact analysis sample is shown in Table 1.

Table 1. Sample Enrolled by Site and Research Group

Site	New Rent Rules  (Program/Treatment)	Existing Rent Rules (Control)	Total Sample
Lexington, KY	486	493	979
Louisville, KY	947	961	1,908
San Antonio, TX	935	934	1,869
Washington, DC	944	964	1,908
Total	3,312	3,352	6,664

As described in Supporting Statement Part A, the 6-Year Follow-Up Study will include data collection with program staff and up to 20 New Rent Rules Group participants in each of the 3 sites continuing to implement the new rent policy (Lexington, Louisville, and San Antonio). Since October 2019, the Washington, DC housing agency (DCHA) has discontinued implementation of the new rent rules. However, the study sample for this site will continue to be tracked using quantitative data for the remainder of the evaluation; no field research is planned for this site.

2. Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

Staff interviews: As with previous rounds of qualitative data collection for this evaluation, MDRC will update the list of staff to be interviewed at each housing agency. The updates to the list will reflect staff departures and changes in staff roles since the last round of field research. The list will include the housing agency executive director, Rent Reform demonstration point of contact/supervisor(s), and housing specialists implementing the new rent rules, all of whom will be invited to participate in a scheduled interview. Supervisors and line staff will also be requested to complete the HCV Function/Activity Checklist in advance of their interviews. They will be asked to check those activities they have worked on in their current position/ assignment (which they will indicate in the box at the bottom of the Checklist). The supervisors and staff will be given a Reference Guide (a list of tasks covered by each activity) to assist them in determining which activities they should check. In addition, ‘other task—please describe’ lines are provided so individuals can write in tasks they do not find on the task list.

The MDRC team will inform the target housing agency staff about this data collection effort and invite them to schedule an appointment for a time that works best for their schedules and the MDRC team will remain very flexible to accommodate staff schedules.

Participant interviews: MDRC will aim for up to 20 completed interviews with participants in the New Rent Rules group at each of the 3 PHAs participating in the demonstration. The sample will include current voucher holders, who are subject to the new rent rules, and represent a mix of employment statuses over the available follow-up period (those who saw no employment or earnings changes and those who saw their earnings increase or decrease significantly. Depending on the timing of the qualitative data collection, we may also be able to include in our sample at each site a small number of families ( 5 to 10 respondents) in the new rules group that recently had their rent recertification under the rent policy each PHA continues to implement (2 of the sites, San Antonio and Louisville, will shift families back to the existing rules at the end of their 2^nd triennial, giving the evaluation an opportunity to understand how these families contrast their current rent policy-related experiences with the alternative rent policy). 

MDRC will conduct outreach through an introductory letter and phone (and email, if available) follow-up.  The introductory note and phone follow-up will include information about the purpose of the interview and the incentive payment that will be offered to respondents. Again, and as in prior rounds of interviews, the MDRC team will be flexible with interviews schedules.^[1] The research team will also send out reminders prior to the scheduled interview date. The outreach materials will be approved by MDRC’s IRB.

And as in the last round of qualitative data collection, MDRC will handle all the participant interview scheduling. MDRC will also draft the introductory letter, which we will request the PHAs to mail. This letter will include an MDRC contact number, which the team will use for fielding interest or questions from participants. MDRC will also provide the PHAs with an FAQ in case they receive questions.

Cost data collection: In order to build accurate estimates of the costs incurred by PHAs to administer the alternative rent policy compared with the current policy, MDRC will collect and analyze financial and organizational data from PHAs on labor and non-labor costs; salaries; number of staff; staff schedules; recorded hours of employment; and the total number of annual (or triennial) and interim recertifications and other types of action recorded on HUD Form 50058 per month. We will collect these data from the PHAs’ financial statements and other administrative records on total revenues and expenditure, and will combine the data with PIC or PHA management information system data on total households served and types of action each month during the follow-up period. As discussed above, the staff interviews (supervisors and line staff) will also be used to understand how staff allocate their time across activities.

Statistical Impact Analysis for the Core Impact Analysis

The basic estimation strategy for the long-term follow-up impact study, which does not rely on the qualitative data, is unchanged. The approach is briefly described here.

The power of the experimental research design will come from the fact that, with an adequate sample size, random assignment ensures that the intervention and control groups will be similar in terms of the distribution of observed and unobserved baseline and pre-baseline characteristics. Thus, post-baseline differences between the two groups can be interpreted as effects of the intervention.

The basic estimation strategy used here is quite analogous to the methodology MDRC and other social science researchers have used in social experiments over the last few decades to generate credible results. The analysis will compare average outcomes for the intervention and control groups, and will use regression adjustments to increase the precision of the statistical estimates that are performed. In making these adjustments, an outcome, such as “employed” or “moved” is regressed on an indicator for intervention group status and a range of other background characteristics.¹ The following basic impact model would be used:

Y_i = α + βP_i + δX_i + ε_i

where: Y_i = the outcome measure for sample member i; P_i = one for program (or intervention) group members and zero for control group members; X_i = a set of background characteristics for sample member i; ε_i = a random error term for sample member i; β= the estimate of the impact of the program on the average value of the outcome; α=the intercept of the regression; and δ = the set of regression coefficients for the background characteristics.

A linear regression framework or a more complex set of methods could be used, depending on the nature of the dependent variable and the type of issues being addressed. For example, logistic regressions could be used for binary outcomes (e.g., employed or not); Poisson or Negative Binomial regressions could be used for outcomes that take on only a few values (e.g., quarters of employment); and quantile regressions could be used to examine the distribution of outcomes for continuous outcomes.

Multiple measures. When multiple outcomes are examined, the probability of finding statistically significant effects increases, even when the intervention has no effect. For example, if 10 outcomes are examined in a study of an ineffective treatment, it is likely that one of them will be statistically significant at the 10 percent level only by chance. Weighing the strengths and weaknesses of different statistical tests to adjust for multiple hypothesis testing and considering some of the limitations of using National Directory of New Hires (NDNH) data, the evaluation team decided to adopt the Benjamini-Hochberg method, applied to p-values.² This strategy was implemented to assess early and interim program impacts, as documented in Riccio and Deitch (2019) and Riccio, Verma, and Deitch (2019).

Site-specific and pooled impacts. The impact analysis also estimates the effects of the alternative rent model for each site separately and for all sites combined. The expected sample size at each housing authority provides adequate statistical power for producing policy-relevant site-specific impact estimates. Site-specific estimates allow the analysis to test the “robustness” of the alternative rent model; that is, each site will provide a type of independent replication test.

The impact analysis will also pool the housing agency samples to produce impact estimates for all sites combined. Pooling would increase the precision of impact estimates, which becomes especially relevant when estimating effects for subgroups of the full sample. In the interim impact reports, the pooled analysis was presented including and excluding DC. The biennial recertification in DC may not differ significantly from the current traditional one-year policy in terms of work incentives. For the six-year impact report, the pooled analysis will exclude DC since, as noted earlier, they exited the demonstration in September 2019.

Subgroup impact estimates

Theory and evidence from other relevant studies (e.g., evaluations of employment programs that included financial work incentives for low-income populations, and for voucher recipients in particular) suggest that changes to the rent structure may have different effects for different types of voucher holders. For example, the alternative rent model may have larger effects on tenants who are not employed at the time of their recertification interview, since it is often easier for individuals to increase their hours in work than for those already working to advance to higher-wage jobs. The new policy may also have different effects depending on a tenant’s barriers to work or preparation for work.

The evaluation thus investigates whether the new rent policy has more pronounced or different effects for particular subgroups. Some subgroups have been pre-specified as “confirmatory” and others are considered “exploratory.” Confirmatory subgroups are ones for which differences in impacts across subgroup categories are predicted based on prior theory or evidence, or because a given subgroup is of great policy interest. For the Rent Reform evaluation, the confirmatory subgroups are subgroups defined according to tenants’ work status/history at the time of random assignment; whether the household head is a single parent with no other adult in the household and is also not employed; whether the household is receiving SNAP benefits, and whether it is receiving TANF benefits. Exploratory subgroups include subgroups defined according to length of time receiving housing subsidies, the number and ages of non-adult children; adults’ education levels; household income levels; and whether the household includes children under age 5.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Based on our prior experience, nonresponse has not been an issue for staff interviews. The MDRC team has been successful in scheduling and conducting interviews with individuals identified for staff interviews.

For the tenant interviews, and we expect to conduct up to 20 interviews in each site, for a total of 60 interviews, MDRC will offer small incentives to help maximize response rates. We will contact study participants using various outreach materials (e.g., advance letters, emails, and follow-up phone calls) to introduce the importance of the study and the contribution their participation will make in rent reform policy for the future. We also maximize response rates by being flexible and accommodating tenants’ availability for participating in such interviews.

It is likely that some sample members may not want to participate in the interview and will not respond to MDRC’s outreach. In cases where sample members inform MDRC that they do not wish to participate, MDRC will drop them from the outreach list. Those who do not respond, will receive up to 10 outreach attempts (phone and email, if available), before they will be placed on a non-response list. As with prior rounds of qualitative data collection for this study, MDRC will draw a large enough sample to accommodate participant non-response. During the outreach process, MDRC will also assesses the characteristics of who is agreeing to participate in the interview and who is not and whether any adjustment in outreach or sample balance is deemed necessary. During both the outreach and interview processes, we will effectively communicate with all participants and notify individuals with disabilities that they may request reasonable accommodations in order to participate. Additionally, we will detail language assistance options for persons with limited English proficiency and offer to conduct the interview in a language they are proficient in. To the best of our understanding, based on information provided by the housing agencies, a very small fraction of the tenants served by the housing agencies in the evaluation may be proficient in a language other than English, particularly Spanish, Arabic, and Amharic. The research team will to be prepared to conduct interviews in Spanish, Arabic, Amharic, if needed. MDRC will translate relevant protocols into the required languages to conduct the interviews. HUD also has a translation services contract and we are prepared to assist the contractor if it is needed.

4. Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The proposed qualitative data collection protocols (which are designed to serve as discussion guides) adapt and build on the protocols used in previous rounds of field research conducted for this study. In this sense, the research team has had a chance to test how the types of questions used in the protocols work in the field and refine some of the questions. Further, for the tenant interview protocols, the MDRC team will review the topics with program staff to get their perspectives on whether additional questions or probes would be useful to understand participants’ experiences with the alternative rent rules. Finally, the HCV Activity Checklist was developed and previously used for HUD’s HCV Administrative Fee Study.

Prior to launching the data collection, the MDRC team will revisit internal guidelines on qualitative data collection, data storage protocols, and review data security requirements.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractors, grantees, or other person(s) who will actually collect or analyze the information for the agency.

HUD’s Office of Policy Development and Research will work with MDRC, to conduct and analyze the proposed data collection. Marina L. Myhre, Ph.D., a Social Science Analyst in HUD’s Office of Policy Development and Research, Program Evaluation Division, serves as Contracting Officer’s Technical Representative (COTR). Her supervisor is Ms. Carol Star. Dr. Myhre and Ms. Star can be contacted at (202) 402-5705 and (202) 402-6139, respectively. MDRC is under contract to HUD to conduct the Rent Reform Demonstration long-term follow-up study. The MDRC team is led by Dr. James Riccio, project director. Other members of the team that worked on the protocols include Dr. Nandita Verma, project manager, and Mr. Keith Olejniczak and Mr. Andrew Rock, study team members.

The statistical aspects of the overall study were developed by the MDRC study team, in consultation with former MDRC colleague, Dr. Stephen Nunez, and MDRC senior economist and impact analyst, Dr. Cynthia Miller.

We provide the following contact information for these individuals:

James Riccio Project Director MDRC 212.340.8822

Nandita Verma Project Manager MDRC 212.340.8849

[1]^[1] The interviews will be conducted in-person or by phone, depending on Covid-19 travel restrictions and respondent preferences.

1To increase the precision of the estimates, we would include a number of key baseline characteristics in the model as covariates. Many of these variables would be measured using the BIF, including work status at the time of random assignment, educational attainment, and potential barriers to employment (such as problems with child care, health, or criminal background). Measures from the administrative data would also be used as covariates, such as long-term unemployment and historical TANF and SNAP receipt.

2See Benjamini and Hochberg (1995). P-values of confirmatory outcomes will be ordered from smallest to largest and assigned a rank (a rank of 1 will be assigned to the smallest p-value, and a rank equal to the number of tests will be assigned to the largest p-value). Starting with the largest p-value, an adjusted p-value is calculated as the (Number of Tests / Rank) * unadjusted p-value. If the adjusted p-value is equal to or less than .10, then that outcome measure and all outcome measures with lower p-values are statistically significant after adjusting for the false discovery rate.

5