Part B

Part B. Collection of Information Employing Statistical Methods

B1 Potential Respondent Universe

The potential respondent universe for the site visits includes the 56 grantees throughout the country awarded an NSP2 grant. The 56 grantees cover 3,068 census tracts in 133 counties and 29 states. Of the $1.93 billion in NSP2 funding, approximately $947 million went to 24 grantees operating exclusively in states that were hit hardest by the national foreclosure crisis—California, Florida, Michigan, Nevada and Ohio.¹ This section describes the plan and methods for selecting a sample of these grantees for the baseline site visits and focus of the analysis.

B2 Statistical Methods

B2.1 Sampling Plan

The sampling plan divided all 133 counties targeted by NSP2 into categories based on the performance of the housing and labor markets during the housing boom and bust periods; selected counties with the highest expected NSP2 investment per census tract within those categories; and then made some substitutions by hand to ensure the selected counties represented a large share of the NSP2 funding and had representation of the NSP2 program on other important grantee characteristics. The plan also defined a minimum expected NSP2 investment level for the county to be included in the sampling frame. Specifically, counties that were neither the primary community a grantee was investing in (defined as the county with the most census tracts targeted by the grantee) nor had an expected NSP2 investment level of at least $10 million were removed from the sampling frame. This step eliminated 78 of the 133 counties where NSP2 investments are occurring, leaving 55 counties eligible for the study.² These counties were removed because, after looking over the distribution of expected NSP2 investments, the researchers judged that the most valuable information on the NSP2 program would be in the communities the grantee was most focused on or that had a substantial amount of investment from grantees. Indeed, more than half of the ineligible counties are from primarily non-urban counties in Ohio and Colorado.

The sample is a purposive sample because the nature of the program makes the purposive sample more relevant to future policy than a representative sample specific to the NSP2 program. NSP2 grants were awarded competitively to communities that could demonstrate both need and effective implementation. This process differed from the formula-based award process for NSP1 and NSP3 grants. The objective of the sampling approach is therefore to allow assessment of the impact of NSP activities across housing markets that vary in their market conditions and in the intensity of NSP investment. This choice reflects the study’s focus on understanding the impact of NSP activities in different housing market contexts, rather than defining the sample with respect to the NSP2 program’s funding allocation.

B2.2 Analysis Plan

The sampling frame was first divided into five strata based on housing market and related labor market conditions. This was done to ensure the evaluation sample covered counties with different experiences during the boom and bust period over the past decade because housing market trends are expected to be an important mediating factor on the impact of NSP to stabilize neighborhoods in a community. The five mutually exclusive housing market categories are as follows:

• Boom-Bust Market Counties. These are counties within MSAs with high price appreciation during the pre-2007 housing bubble and a large decline in housing prices and increase in unemployment during the bust.³ High growth during the bubble is defined as being in the top half of growth in housing prices for MSAs in the nation from the first quarter of 2000 to second quarter of 2007, and the post-bubble bust is defined as being in the top half of the decline in housing prices and increase in the unemployment rate from the third quarter of 2007 to the fourth quarter of 2009.⁴ To account for geographic differences in the housing markets, we divided the boom-bust market counties into two strata.

1. Boom-Bust Market Counties: Sand States (18 Counties). Sand States are used in the literature to refer to states with significant coastal beaches or deserts and substantial recent population growth. These states are considered to be Arizona, California, Florida, and Nevada, but we also included other western states in this category. Most of the counties in this category are in either California or Florida. The NSP2 counties in the Sand States tend to have larger housing price appreciation during the boom and larger price depreciation during the bust compared to the other counties in the boom-bust category.

2. Boom-Bust Market Counties: East Coast States (9 Counties). All except one of the other boom-bust counties are all on the East Coast, either in the New York-New Jersey area or in the Mid-Atlantic area. The one non-East Coast state in this category is Cook County, Illinois. NSP2 counties in these states did not see as extreme housing price appreciation or price depreciation rates as the counties in Sand States, but housing prices still increased by at least 65 percent in the boom. The counties in the East Coast also had modest to low levels of population growth in the first half of the 2000s while the counties in the Sand States tended to have double-digit population growth rates during that period. The Sand States had correspondingly higher rates of housing growth, particularly in exurban locations. These differences in housing markets may affect housing market outcomes and the impact of NSP2, and thus the separation of the Boom-Bust group into two categories is warranted.

3. Boom-Stable Market Counties (9 counties). Counties that had above average housing price growth during the boom, but either did not have an above average decline in housing prices during the bust or did not have an above average increase in unemployment rates during the bust. These counties usually had housing price rate growth rates lower than the Boom-Bust Sand States, but almost as high as the Boom-Bust East Coast States. However, their housing markets appear to be more stable during the bust period than the Boom-Bust East Coast state as neither the housing price declines or the unemployment rate increases tended to be as large.

4. Lagging/Declining Market Counties (16 Counties). For this category, we identify counties with housing markets that appear to be in a longer-term decline than defined by just the national housing market bust period. Operationally, we placed counties in this category if the MSA was in the bottom half of price appreciation during the boom period and had an unemployment rate above the median at the end of the boom period in 2007.⁵ The NSP2 counties in this category are almost exclusively in Ohio and Michigan. Most of these counties had declining population during the first half of the 2000s.

5. Slow Growth Counties (3 Counties). This category consists of counties in MSAs that are similar to the Boom-Stable counties, except housing prices did not increase as much during the boom period. Operationally, the counties in this category were in the bottom half of housing price growth during the boom period, but on the lower half of unemployment rates at the end of the boom period.

The next step in the sampling plan allocated the sample to housing market categories in proportion to the share of counties in that category in the sampling frame. For example, 33 percent of the 55 counties in the sampling frame are Boom-Bust Market Counties in Sand States, so we allocated 35 percent of the sample slots (7 of the 20 sample slots) to that category. The planned allocation is shown in Exhibit 4.

Exhibit 4: Number of Counties in Sampling Frame and Selected Planned Sample by Housing Market Category

	# of Counties in Sampling Frame	% of Counties in Sampling Frame	Planned Sample Allocation	% of Counties in Planned Sample
Boom-Bust Market Counties – Sand States	18	32.7%	7	35%
Boom-Bust Market Counties—East Coast	9	16.4%	3	15%
Boom-Stable	9	16.4%	3	15%
Slow Growth	3	5.5%	1	5%
Lagging/Declining Counties	16	29.0%	6	30%
Total	55	100%	20*	100%
* The allocation of the initial sample we select (described below) is slightly different than this planned allocation and includes 21 rather than 20 counties. Source: HUD NSP2 data. Notes: Counties were categorized in housing markets based on the conditions in the MSA in which they are located.

To select the sample counties within the housing market categories, the counties were ordered from the largest to smallest expected NSP2 investment per targeted census tract. Then research staff selected the counties with the largest expected investment per census tract until reaching the number of counties allocated to the sample from that housing market category. The reason for selecting the counties with the highest NSP2 dollars per census tract is the expectation that the study is more likely to be able to measure impacts of NSP2 in neighborhoods with large investments. In addition to the size of the NSP2 investment, research staff also wanted to make sure the sample covered a large share of the NSP2 grant awards in total, was geographically diverse in terms of states and urban/exurban areas, and represented the various types of activities allowable under NSP2. To achieve these other objectives, the researchers replaced some of the initial sample counties and added an additional county (to make the sample 21 counties). The final sample from this selection process is shown in Exhibit 5. The 21-county sample:

• includes counties where 27 of the 56 NSP2 grantees are implementing NSP2, although the number of NSP2 grantees is expected to change slightly depending on their capacity to participate in the study (described below);

• covers 1,724 census tracts or 56 percent of the 3,068 census tracts targeted by NSP2; and

• has an expected investment level $1.1 billion or 57 percent of the $1.93 billion in NSP2 grants.

Exhibit 5: Original Sample and Reason for Replacements in Revised Sample

Original Sample	Revised Sample	Reason Original Sample Replaced
Boom-Bust Market Counties: Sand States (8)
Washoe County, NV	Same
Sarasota County, FL
Riverside County, CA
Palm Beach County, FL
Miami-Dade County, FL
Hillsborough County, FL	Maricopa County, AZ	Maricopa has second highest level of NSP2 funding in category; wanted to have an Arizona county rather than a fourth Florida county.* We chose to remove Hillsborough County from the sample because their application indicated that approximately half their funding would be used in a neighborhood also undergoing a HOPE VI revitalization.
Escambia County, FL	Los Angeles County, CA	LA County largest NSP2 investment of any county—over 2.5 times higher than next largest county in the category; Also provides a second county in same area as another sample county (Riverside, CA), in addition to Miami-Dade and Palm Beach Counties.
Added Site (Not replacement)	Stanislaus County, CA	Good example of exurban area that NSP2 investing in.
Boom-Bust Market Counties: East Coast (3)
Baltimore City, MD	Same
Camden County, NJ	Kings County, NY	Highest level of expected NSP2 investment in category. Judged that it provides better balance with Baltimore than Camden.
District of Columbia	Cook County, IL	Cook County has the largest expected NSP2 investment in this category—more than triple the next highest. District dropped because NSP2 grant is for DC only, which has had a more stable housing market than the MSA on which the housing market typology was based.
Boom-Stable Counties (2)
Ramsey County, MN	Same
Mobile County, AL	Philadelphia County, PA	Philadelphia County was added because has the largest expected NSP2 county investment in category. Mobile County had largest unemployment rate increase during the period, which did not fit the overall boom-stable profile.
Orleans Parish, LA	Replaced with extra slow growth county	Removed Orleans Parish because New Orleans is the site of many post-Katrina recovery efforts, which might complicate attempts to evaluate the effect of NSP2. Replaced by allocated additional extra slow growth county to sample because of the desire to have a Texas site in the sample and the high levels of expected investment per tract in two of the slow-growth counties.
Original Sample	Revised Sample	Reason Original Sample Replaced
Slow Growth Counties (2)
Davidson County, TN	Same	Note that Dallas County replaces Orleans Parish from Boom-Stable County category as described above.
Dallas County, TX
Lagging/Declining Market Counties (6)
Pulaski County, AR	Same
Ingham County, MI
Wayne County, MI
Genesee County, MI
Calhoun County, MI	Cuyahoga County, OH	Wanted non-Michigan sites in this category and Cuyahoga County has second largest expected NSP2 funding in the category.
Kent County, MI	Montgomery County, OH	Wanted another non-Michigan county and given the number of Ohio counties in NSP2, decided it should be from Ohio. Dropped Kent County rather than Genesee or Ingham because Ingham has larger expected county investment and the city of Flint in Genesee County epitomizes a long-term declining market.
21 Counties	21 Counties	8 of the original 20 counties were replaced, and we added one additional county to the sample.
* Source: HUD NSP2 data Note: Counties were categorized in housing markets based on the conditions in the MSA in which they are located.

As previously discussed, the study sample is subject to further modification. The sample reported here may not be the final study sample. First, HUD PD&R staff will share the sample list with the Program Office to get their input. Program staff may know of NSP2 implementation activities at sites that either makes one of the selected sites a poor candidate for the study or makes an unselected site a particularly good site for the study. Second, the study’s review of sources for property-sales records for the counties in the sample could lead to the need to replace sites. A history of property sales in the study-sample counties is necessary for the analysis. A preliminary review of the availability of these data from vendors suggests that the necessary data are available for the current sample sites, but that we will need to use two different vendors to obtain data for all the sites. Third, researchers need to determine in the reconnaissance calls, one of the methods for approval, whether the grantees have the necessary property-level data available for our analysis−such as the NSP2 acquisition date, NSP2 activity, and final disposition date—to be included in the study sample.

One or more substitutions may be needed for initially selected counties if a county, on closer examination, fails to meet necessary standards on one or more of the three criteria listed above. Since participation by the grantee is voluntary, it is also possible that a specific selected grantee may decline to participate. Any sample counties that need to be replaced will be selected from counties within the same housing market types as the site being replaced.

B2.3 Justification of Level of Accuracy

The study sample is not intended to be statistically representative of all the NSP2 counties or even all the counties in the sampling frame; however, the sample is intended to have representation of the diverse counties receiving NSP2 investments and to oversample the counties that are expected to be most informative about the outcomes and impacts of NSP2—counties with the largest or most concentrated NSP2 investments. Exhibit 6 shows the characteristics of the 21 counties in the study sample compared to the 55 counties in the sampling frame and all 133 NSP2 counties.

The primary differences between the sample counties and the other NSP2 counties are related to the oversampling of boom-bust counties and focus on counties with the largest and most concentrated NSP2 investments. The housing market category distribution in the sample also reflects the substitution of a Slow Growth County for a Boom-Stable County.

The sample counties had a higher price appreciation during the boom period (86.5 percent) than either the sampling-frame counties (78.4 percent) or the entire group of NSP2 counties (67.2 percent) and had a larger average decline during the bust period. The sample counties also have higher expected investments per county and per census tract than the other groups and are more likely to have more than one grantee implementing NSP2. The sample counties also tend to have larger central cities than the other groups. For example, none of the sample counties have a city of less than 200,000 as their largest city whereas 9.1 percent of the sampling-frame counties and 33.1 percent of all NSP2 counties do. Related to the large city sizes, the sample counties are more likely to have a large share of minorities than the other NSP2 counties in the sampling frame. Aiming for a sample that more closely represented the sampling frame in the size of central cities and the share of minorities would require a tradeoff with the goal of selecting counties with the highest expected investments. We followed the priorities listed in the Research Design and focused on choosing the counties that would include a large share of the NSP2 funding in the sample while still having some representation of all except the counties where the largest city is less than 200,000 people.

Exhibit 6: Characteristics of Sampled NSP2 Counties Compared to Sampling Frame and All NSP2 Counties

Characteristic	Sampled Counties (n=21)	Counties in Sampling Frame (n=55)	All NSP2 Counties (=133)
Housing Market Characteristics of MSA in Which County is Located
Housing Market Category
Boom-Bust Market: Sand States	38,1%	32.7%	21.8%
Boom-Bust Market: East Coast	14.2%	16.4%	10.5%
Boom-Stable Markets	9.6%	16.4%	15.0%
Slow Growth Markets	9.6%	5.5%	6.0%
Lagging/Declining Markets	28..6%	29.0%	46.6%
Housing Market Indicators
Average Price Appreciation Rate in MSA: Boom Period	86.5%	78.4%	67.2%
Average Price Appreciation Rate in MSA: Bust Period	-21.5%	-17.0%	-11.6%
Change in Unemployment Rate in County: Bust Period	5.3 pp	5.1 pp	5.2 pp
Average Population Growth Rate in County 2000 to 2006	5.3%	3.3%	4.4%
NSP2 Investment Level in Counties
Average # of Grantees per County	1.8	1.5	1.2
Counties Targeted by more than One Grantee	47.6%	34.5%	14.3%
Median $NSP2 Per County	$30,470,000	$23,000,000	$3,331,529
Median $NSP2 per Targeted Census Tract	$1,787,274	$917,614	$412,973
Geographic and Demographic Characteristics of Counties
Census Region
Northeast	9.5%	14.5%	12.8%
Midwest	33.3%	34.5%	42.9%
South	33.3%	29.1%	20.3%
West	23.8%	21.8%	24.1%
Number of States Represented	14	23	28
In a Judicial Foreclosure State	38.1%	45.5%	53.4%
Population of Largest City in County
< 100,000	0%	0%	26.3%
100,000 to < 200,000	0%	9.1%	6.8%
200,000 to < 400,000	14.3%	10.9%	10.5%
400,000 or More	85.7%	80.0%	56.4%
Minority Concentration of County
<25% minority	9.5%	23.6%	46.6%
25 to <50% minority	47.6%	43.6%	30.1%
50% or higher	42.9%	32.7%	23.3%
Sources: Source: Price appreciation scores generated from FHFA Downloadable Housing Price Index Data http://www.fhfa.gov/Default.aspx?Page=87; Population figures downloaded from the Census Population Estimates Program at http://www.census.gov/popest/archives/2000s/vintage_2006/; Unemployment rates calculated from the Bureau of Labor Statistics Local Area Unemployment Statistics Data at http://www.bls.gov/lau/. State foreclosure laws are based on Cutts & Merrill (2008).6 Notes: Characteristics of the county are based on the MSA in which they are located. The boom period is defined as first quarter of 2006 through the second quarter of 2007; the bust period is defined as the third quarter of 2007 through the fourth quarter of 2009. Average amounts targeted per county and per census tract are calculated by dividing total grant amount of grantee by the number of counties targeted or census tracts targeted. This assumes the grantee will spread their investments equally across targeted counties and census tracts. For categorizing the 35 NSP2 counties outside MSAs in housing market categories and for calculating housing market price trends, we used the information on the nearest MSA. These 35 counties are not in the sample or sampling frame. The District of Columbia is counted as a state in the count of states with counties that have NSP2 investments.

Exhibit 7 compares the characteristics of the 27 grantees targeting counties in the study sample to all 56 grantees receiving NSP2 grants. The grantees in the sample have grantee types and planned activities similar to all the NSP2 grantees. The only small differences are that the sample grantees are a little less likely than the full population of NSP2 grantees to be part of a consortium and are more likely to be led by a national grantee. The average grant size is substantially different between the sample and the population, explaining why the 27 grantees comprise less than half of all NSP2 grantees but have nearly three-fourths of all NSP2 funding.

Exhibit 7: Characteristics of Grantees in Sampled NSP2 Counties Compared to All NSP2 Grantees

Characteristic	Grantees in Sampled Counties (n=27)	All NSP2 Grantees (n=56)
Grantee Type (lead grantee)
Government	66.7%	67.9%
Nonprofit	33.3%	30.4%
For-Profit	0%	1.8%
National Grantee (lead grantee operates in multiple states)
National Grantees	18.5%	14.3%
Consortium Status
In a Local Consortium	55.6%	62.5%
In a National Consortium	7.4%	7.1%
Non-Consortium (Single grantee or partners, but no legal agreement)	37.0%	30.4%
NSP2 Activity Type (multiple activities allowed)
Rehabilitation	88.9%	91.1%
Financing	66.7%	66.1%
Redevelopment	55.6%	58.9%
Demolition	44.4%	46.4%
Land Banking	22.2%	23.2%
NSP2 Funding at Grantee Level
Median Grant Amount	$29,363,660	$21,993,970
Total NSP2 Funding	$1,411,157,925	$1,930,000,000
Share of all NSP2 Funding	73.1%	100%
Source: HUD NSP2 data

The planned analysis includes a number of comparisons between NSP2 grantees across the housing market categories and several grantee-level characteristics.

B3 Maximizing Response Rates

NSP2 Grantee Site Visit Sample

The grantee site visit recruitment process will emphasize the evaluators’ respect for any burdens placed on the grantee and the importance of the study in determining future community stabilization policies in order to maximize the participation rate of the selected sites. The first step in recruiting sites is for HUD to send an advanced letter to the sample grantees to inform them of the importance of the study to both the NSP program and future community stabilization efforts, strongly encourage them to participate, and specify what participation entails. The letter should also mitigate any potential fears that the study is an audit or contract monitoring activity and that the participation burden is minimal so that a grantee will not be reluctant to participate. An additional key point that will be emphasized during the recruitment effort is that the evaluators will be gathering all the information they can from pre-existing sources to keep grantees’ participation burden low. Once the sample is finalized, the response rate to the baseline interviews is expected to be 100 percent.

Review of Local Administrative Data

The burden hours outlined on page 12 provide for time to be spent during the site visits ascertaining the existence of any easily available local data to supplement the analytical resources of the study. Specifically, the data needs to be in electronic format. Given the wide variation in locally generated data, the expectations in two topic areas, education and public transit, are low, but there is some reason to believe that for the analytically most critical topic, crime, some communities will have more advanced data systems. There will be no requests for reports to be generated beyond what is already generally available. Any burden associated with collection of local administrative data is therefore limited to the local agencies’ review and fulfillment of the study team’s data request.

Researchers will initially pursue crime data from the full sample of 21 counties. However, it is unlikely that we will be able to collect crime data from all counties, because of limitations in the availability and quality of data from local agencies. The data collection process may involve collecting information from multiple agencies in a single county. If NSP2 activities are concentrated within a particular city in our target county, researchers may be able to contact only the local police department and request the data. Many departments have specific divisions devoted to maintaining and analyzing crime data. Such departments will be researcher’s first point of contact in counties where they exist. The second step in each county is to identify the full set of local data sources required to comprehensively collect crime records for the county. Compiling this list will occur through consultation with HUD, the first agency contact, and researchers familiar with local data sources.

As a first step, researchers will notify the agency of the study and the data requested with an initial email and a follow-up telephone call. For some counties, this initial contact may be sufficient to acquire the requested data. If local agencies are unresponsive to our letter and initial calls, the second step is to request that HUD send an official communication with the data request and contact information for either the project director or the HUD GTR. A key element of this process is identifying the staff member responsible for granting access to the data. The contractors will review the agency website and consult with the local researcher to identify a first point of contact. Each of the agency communications will also request that the agency staff refer us to the appropriate point of contact.

Although crime records are public information, it is possible that the contractors will run into obstacles that make it difficult to obtain the records. Individual agencies may not be willing to release their records to researchers that do not have an established relationship with the agency. Gatekeepers to the data may also be unresponsive or resistant to releasing the records. If research staff continue to encounter resistance following the official HUD communication, they will stop the data collection activities for that agency.

B4 Tests of Procedures or Methods

Early drafts of the instrument have been reviewed by HUD personnel, Abt Associates staff, and University of Southern California staff in order to ensure that the instrument is clear, flows well, and is as concise as possible.

B5 Statistical Consultation and Information Collection Agents

HUD has contracted with Abt Associates, the University of Southern California, and the Concentrance Consulting Group to design the study, conduct the data collection, and perform the analysis activities.. The HUD Government Technical Representative (GTR) reviewed all the proposed procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call the HUD GTR, Judson James (202-402-5707), or the Abt Associates Project Director, Dr. Alvaro Cortes (301-634-1857).

1 See: www.hud.gov/offices/cpd/communitydevelopment/programs/neighborhoodspg/pdf/nsp2awardtotalAllocation.pdf

2 All 56 Grantees have at least one county in the sampling frame. There are only 55 counties, because some counties have multiple grantees.

3 The National Bureau of Economic Research (NBER) defined the economic recession as starting in December 2007 and ending in June 2009. However, the housing market peak occurred before the start of the recession. The Federal Home Finance Agency (FHFA) House Price Index that indicates the peak price appreciation rate for the nation occurred in the third quarter of 2005 and that negative price appreciation started two years later (the third quarter of 2007), though it started earlier in many areas.

4 To be in the Boom-Bust Category, a county had to be in an MSA with housing price appreciation of 49.6 percent or higher during the boom, a housing appreciation of -3.3 percent or lower during the bust, and an unemployment rate increase of 4 percentage points or higher in the bust period.

5 To be in the Lagging/Declining Market Category, a county had to be in an MSA with housing price appreciation during the boom of 49.5 percent or lower and an unemployment rate of 4.7 percent or higher in 2007.

6 Cutts, Amy Crews & Merrill, William A. 2008. “Interventions in Mortgage Default: Policies and Practices to Prevent Home Loss and Lower Costs.” Freddie Mac Working Paper