Combined Part B_Clean_04 02 14

B. STATISTICAL METHODS

1. Universe and Respondent Selection

The study is comprised of two phases: phase 1, which is completed and is here for reference, collected information from homeowners who have re-defaulted after receiving a HAMP modification, and phase 2 which will collect information from homeowners that are still in good standing (or “active”) under their permanent HAMP modification and are pending an interest rate reset¹.

1. Phase 1

As of July 31, 2013 there have been 1,236,682 permanent modifications under HAMP. Of those, 325,491 (or roughly 26%) have re-defaulted. As demonstrated in Table 1 below, this population is comprised of loans which are owned or guaranteed by Fannie Mae or Freddie Mac (“GSE Loans”), and loans which are privately held (“Non-GSE Loans”).

Table 1: HAMP Re-Defaults As of July 31, 2013

	No. of Permanent HAMP Modifications	No. of Re-Defaults
Non-GSE Loans	639,878	175,708
GSE Loans	596,804	149,783
Total	1,236,682	325,491

While the Administration considers HAMP to be a single, comprehensive program, the management occurs separately for Non-GSE Loans and GSE Loans. Treasury sets program requirements for modifications of Non-GSE Loans, whereas Fannie Mae and Freddie Mac set program requirements for modifications of GSE Loans.

The target population for Treasury’s study was homeowners with Non-GSE Loans who received permanent mortgage modifications under HAMP and subsequently missed three consecutive payments (“re-defaulted”). In order to have a large enough sample of homeowners to survey who have had recent experience with the issue, the universe of potential respondents consisted of those who missed their third consecutive payment on or after August 1, 2011.

The sample for phase 1 of the study was stratified by two criteria: 1) Homeowner’s change in payment between their pre-modification payment and modified payment as a percentage (P&I Change), and 2) Homeowner’s delinquency status at the start of the HAMP modification’s trial period (Delinquency at Trial). Prior research has shown that these two variables significantly relate to subsequent HAMP performance. These criteria were further broken down into six groups as described in the example provided in Table 2: Desired Response Broken Down by Re-Default Target Homeowners Sampling Categories.

Treasury’s goal for phase 1 was to collect a total of 2400 survey responses, with 400 responses in each group.

Table 2: Desired Response Broken Down by Re-Default Target Homeowners Sampling Categories

	P&I Change
Delinquency at Trial	<=24%	>24% - <=42%	>42%	Total
<= 6 months	400	400	400	1200
> 6 months	400	400	400	1200
Total	800	800	800	2400

The above subgroups were defined to mirror the total HAMP re-default population. For example, the ‘6-month” cut-off point will approximately evenly split the HAMP re-default population. This sampling stratification strategy helped ensure that the completed survey sample well represents the universe population with more precision. This also helped reduce sampling biases as certain subgroups might be less or more likely to respond to the survey than other subgroups.

In terms of sampling, the universe of potential respondents was classified into the six groups and then, via random sampling, survey participants were selected for each group to receive the survey.

The population of homeowners with Non-GSE Loans who re-defaulted on their HAMP modifications on or after August 1, 2011 is approximately 100,000 homeowners. Among these, about 40,000 homeowners re-defaulted over the past 12 months. In order to focus on homeowners with recent experiences the survey samples were first drawn from the more recent re-default population.

At the 95% confidence level the margin of error is 1.94% and 1.98% respectively, when the total of 2400 completed sample is drawn from the most recent re-default population of 40,000 homeowners and the total universe of 100,000 homeowners, respectively.

Power Analysis

Assuming an effect size of 0.1 (or 10 percentage points difference on proportional data such as testing the difference between 50% saying Yes from Group A and 60% saying Yes from Group B), a sample size of 400 per group will allow us to statistically test this 0.1 difference (two-tailed test) at the 95% confidence level (alpha = 0.05), yielding a statistical power of 0.81. In addition, the statistical power will increase to nearly 100%, if we examine differences at the higher group level such as comparing the [<=6 months delinquency at trial] and [>6 months delinquency at trial] groups, with 1200 sample size per group.

Non-Response Bias

This study pursued the following methods in reducing non-response biases:

1. We compared the responses group and non-responses group on selected data from the HAMP system of record such as Loan-to- Value (LTV) ratio pre-modification and post-modification, Debt-to-Income (DTI) ratio pre-modification and post-modification, and credit score at modification to assess if borrowers with certain observable characteristics are more or less likely to respond.

2. If significant differences were observed, we would have created non-response adjustment weights and apply the weight to each bucket of “similar” loans, in order for the completed survey sample to represent the target universe.

The non-response bias analysis following phase I showed when comparing the sample and population distributions as well as comparing Responders vs. Non-Responders on 17 key variables from the IR2 database (for example, Loan to Value ratio before and after modification, P&I amount before and after modification, front-end and back-end DTI before and after modification, etc.). The analysis indicated that the Responders’ profile closely matched the Non-Responders’ profile on all the 17 variables. Therefore, there was no concern with the non-response bias that will jeopardize the survey results.

2. Phase 2

As of November, 2013 there have been 1,298,014 permanent modifications under HAMP. Of those, 980,318 (or roughly 75%) are still active. As demonstrated in Table 3 below, this population is comprised of GSE Loans and Non-GSE Loans. .

Table 3: Active HAMP Permanent Modifications As of November, 2013

	No. of Active Permanent HAMP Modifications
Non-GSE Loans	534,443
GSE Loans	445,875
Total	980,318

While the Administration considers HAMP to be a single, comprehensive program, the management occurs separately for Non-GSE Loans and GSE Loans. Treasury sets program requirements for modifications of Non-GSE Loans, whereas Fannie Mae and Freddie Mac set program requirements for modifications of GSE Loans.

The target population for phase 2 of Treasury’s study is homeowners with Non-GSE Loans who have a HAMP permanent modification in good standing with a pending interest rate reset.

The sample for phase 2 of the study will be stratified by two criteria: 1) Homeowner’s with estimated Mark to Market Loan to Value (MTM LTV) ratios greater than 100% and less than or equal to 100% as of November 2013, and 2) Homeowner’s estimated change in monthly Principal & Interest (P&I) payments between the monthly P&I at the start of the HAMP modification and the estimated monthly P&I payment at the final interest rate reset. These criteria will be further broken down into six groups as described in the example provided in Table 4: Desired Response Broken Down by Interest Rate Reset Target Homeowners Sampling Categories.

Treasury’s goal for phase 2 is to collect a total of 2400 survey responses, with 400 responses in each group.

Table 4: Desired Response Broken Down by Interest Rate Reset Target Homeowners Sampling Categories

	P&I Change
Estimated MTM LTV	<=$140	>$140 - <=$300	>$300	Total
<= 100%	400	400	400	1200
> 100%	400	400	400	1200
Total	800	800	800	2400

The above subgroups are defined to mirror the HAMP permanent modification in good standing with a pending interest rate reset.

This sampling stratification strategy will help ensure that the completed survey sample well represents the universe population with more precision. This will also help reduce sampling biases as certain subgroups might be less or more likely to respond to the survey than other subgroups.

In terms of sampling, the universe of potential respondents will be classified into the six groups and then, via random sampling, survey participants will be selected for each group to receive the survey.

The population of homeowners with active Non-GSE permanent modifications with a pending interest rate reset as of November 2013 is approximately 375,000 homeowners. Homeowners will be sampled across all vintages. The sampling error will be 2% at the 95% confidence level.

Power Analysis

Assuming an effect size of 0.1 (or 10 percentage points difference on proportional data such as testing the difference between 50% saying Yes from Group A and 60% saying Yes from Group B), a sample size of 400 per group will allow us to statistically test this 0.1 difference (two-tailed test) at the 95% confidence level (alpha = 0.05), yielding a statistical power of 0.81. In addition, the statistical power will increase to nearly 100%, if we examine differences at the higher group level such as comparing the [MTMLTV < = 100%] and [MTMLTV > 100%] groups, with 1200 sample size per group.

Non-Response Bias

This study will pursue the following methods in reducing non-response biases:

1. We will compare the responses group and non-responses group on selected data from the HAMP system of record such as estimated Loan-to-Value (LTV) ratio, Debt-to-Income (DTI) ratio pre-modification and post-modification, and credit score at modification to assess if borrowers with certain observable characteristics are more or less likely to respond.

2. Should significant differences be observed, we will create non-response adjustment weights and apply the weight to each bucket of “similar” loans, in order for the completed survey sample to represent the target universe.

2. Procedures for Collecting Information

Research Method

Names and addresses of homeowners who have received a HAMP modification are the only contact information we have available within the HAMP system of record. As a result, to optimize the response rate, for both phase 1 and phase 2 a paper survey will be mailed to the target homeowners who have been randomly selected from the groups noted in Table 2 and Table 4 respectively, above. Due to the differences in target populations a homeowner can only be selected to take part in one of the surveys. Each survey will be mailed with a pre-paid return envelope and a cover letter inviting borrowers to participate. Each survey is estimated to take approximately 15-20 minutes to complete. The cover letter will also provide a unique web address for an online version of the surveys to allow respondents the option of completing the survey online instead of on paper. In order to lessen the likelihood of respondents completing multiple survey responses, each survey recipient will be assigned a unique identifying number which will be printed on the paper survey they return. If they elect instead to complete their response online, the interface will require them to enter their unique number in order to gain access. Checking the identifying numbers on responses received against the list of survey recipients will allow Treasury to identify and exclude any duplicate responses.

Survey participation will be completely voluntary. Respondents will not be asked for any personally identifying information, and will be instructed not to provide any extraneous text or responses on the survey form other than to select one of the multiple choice options present following each survey question. Analysis and reporting will be conducted at the aggregate level, with no personal information identified or reported.

The surveys will be designed and constructed to meet the research objectives of obtaining an understanding of the following: 1) Homeowner’s initial understanding of their modification and ability to pay the modified payments, 2) Homeowner’s change in circumstances (positively or negatively) impacting their ability to pay the modified amount, and 3) Homeowner’s perceptions of the MHA Program, process and outcomes.

Test Run followed by the Principal Mailing

In order to decrease the burden and minimize the number of homeowners that will need to be solicited, the data collection for each phase will be conducted in two parts: 1) Test Run and 2) Principal Mailing.

A Test Run will be conducted for each phase to assess the likely homeowner response rate, and therefore inform the number of additional mailings needed for the Principal Mailing to achieve the target of 2,400 responses for each phase (4,800 total responses). Each Test Run will consist of mailing to 2,000 homeowners selected at random, a survey, a pre-paid return envelope, and a cover letter inviting them to complete and return the survey via paper response or use the provided web link to complete the survey online.

The results of each Test Run will be analyzed to determine the effectiveness of the survey materials to solicit a homeowner response, and to estimate the potential response rate that will result from the Principal Mailing. Based on this analysis, appropriate updates to the survey materials will be incorporated and approvals obtained, and a sample size of target homeowners to which the survey will be mailed for each of the six groups described in Table 2 and Table 4 will be established. This will maximize the likelihood of receiving the target 400 completed responses in each group, while minimizing the cost of the Principal Mailings. Responses received as part of the test run will be counted as part of the targeted 2,400 responses for each phase and will be included in the overall analysis; therefore any homeowner solicited as part of the Test Run will not be part of the Principal Mailing.

Data Analysis/Reporting

In order to provide robust insights, data collected from the survey responses will be augmented with data points from the HAMP system of record. In addition to data points used from the HAMP system of record such as delinquency at trial and P&I change used to determine sampling categories, data points such as Loan-to-Value (LTV) pre-modification, LTV post-modification, Debt-to-Income (DTI) pre-modification, DTI post-modification, credit score, amortization term after modification, amount of principal forgiveness and/or forbearance may be used. Prior to the analysis any information used to combine survey responses with HAMP system of record data will be removed to ensure each record is anonymous. None of the raw survey data, data points from the HAMP system of record, the analysis, or the report will contain personally identifiable information.

If the volume of responses received appears unlikely, in the Department’s judgment, to reach the desired 2,400 responses for each phase divided among the groups listed in Table 2 and Table 4 within a reasonable time, the Department may consider restructuring the analysis based on the actual number of responses received.

Research Ethics

Data collection and research protocols will be based on the CASRO Code of Standards and Ethics for Survey Research (http://www.casro.org/?page=TheCASROCode), developed by the Council for Survey and Research Organization (CASRO), which has long been the benchmark of the industry and provides a consistent, industry-accepted standard for implementing similar studies.

Research Timeline

Event	Target Dates
Study of MHA Program Performance – Phase 1
Complete development of survey materials (cover letter, survey questionnaire , FAQs)	Sept 20th
Receive Approval of survey materials and OMB Control Number	Sept 27th
Print survey materials for Test Run	Sept 30th
Mail the Test Run	Oct 4th
Review responses and print documents for Principal Mailing	Oct 21st
Mail the Principal Mailing	Nov 1st
Begin tabulation of responses	Dec18th
Conclude analysis of results	Feb 13th

Event	Target Dates
Study of MHA Program Performance – Phase 2
Complete development of survey materials (cover letter, survey questionnaire , FAQs)	Apr 6th
Receive Approval of survey materials from OMB	Apr 5th
Prepare and Print survey materials for Test Run	Apr 9th
Mail the Test Run	Apr 22nd
Review responses and print documents for Principal Mailing	May 8th
Mail the Principal Mailing	May 27th
Begin tabulation of responses	June 18th
Conclude analysis of results	July 23rd

Note that the foregoing timelines reflect a “best case” scenario. For example, if we cannot commence the Test Run on-time or must revise the survey materials based on the results of the Test Run, the Principal Mailing may be delayed.

3. Methods to Maximize Response

In order to achieve the desired target of 2,400 respondents in each phase (4,800 total respondents), the following methods will be used to maximize response rates:

• The cover letters and surveys will be printed on Treasury letterhead, signed by a Treasury official, and sent in an envelope marked with the Treasury logo.

• The cover letters will be drafted in an engaging way that focuses on understanding the consumer mortgage process to help provide better assistance to all struggling homeowners.

• The surveys will be drafted to minimize the likelihood of even inadvertent disclosure of personally-identifiable information.

• The official MHA website will refer to the ongoing survey activity so participants can confirm its legitimacy.

4. Testing of Procedures

For each phase a Test Run will be administered to 2,000 potential respondents before launching the Principal Mailing to a larger target homeowner population. This Test Run will allow the Treasury to get a better sense of the response rate, to inform the number of mailings needed to achieve the desired sample size. In addition, after each Test Run, the Treasury will review the results and have an opportunity to revise the cover letter and/or the survey for clarity, accuracy, appearance, and tone, in order to minimize cost and achieve success with the Principal Mailings.

Based on the test run for phase 1 yielding a 4.9 % response rate, the Principal Mailing was conducted based on a 4.5% response rate assumption. As a result the phase 1 survey had a 5.2% return (2,753 total responses) surpassing our target of 2,400 for phase 1.

5. Contacts for Statistical Aspects and Data Collection

• Emily.Haeckel@treasury.gov

1 Under HAMP, a homeowner’s interest rate is reduced to as low as 2% for a period of five years. After such time, the interest rate will increase by up to one percentage point per year, until it reaches the prevailing market interest rate at the time of the original HAMP modification. Accordingly, homeowners facing an interest rate reset will experience an increase in their monthly mortgage payment.