The Neighborhood Stabilization Program Tracking Panel:
OMB Clearance Package Part A
Draft
October 9, 2012
Table of Contents
A1 Circumstances That Make the Collection of Information Necessary 1
A2 How and by Whom the Data Will Be Used 1
A2.2 Purpose of the Data Collection 3
A2.3 Who Will Use the Information 5
A2.4 Instrument Item-by-Item Justification 5
A3 Use of Improved Technologies 7
A4 Efforts to Avoid Duplication 8
A5 Involvement of Small Entities 8
A6 Consequences of Less Frequent Data Collection 8
A8 Consultations outside the Agency 9
A10 Arrangements and Assurances Regarding Confidentiality 9
A12 Estimate of Annualized Burden Hours 9
A13 Estimated Record Keeping and Reporting Cost Burden on Respondents 10
A14 Estimated Cost to the Federal Government 10
A15 Reasons for Changes in Burden 10
A16 Tabulation Plan, Statistical Analysis, and Study Schedule 10
The impacts of the foreclosure crisis on neighborhoods are vast and still unfolding. Research conducted since the emergence of the crisis has demonstrated the significant negative impacts of foreclosures on neighborhood housing prices, municipal budgets, and crime. Underlying the concerns raised by these studies is the premise that foreclosed properties are “contagious” in their effects and produce a series of negative spillover effects throughout the surrounding neighborhoods. As concerns about these spillover effects intensified, Congress authorized $4 billion as part of the Housing and Economic Recovery Act of 2008 (HERA) to create the Neighborhood Stabilization Program (commonly referred to as NSP1). In 2009, Congress authorized an additional $2 billion for the Neighborhood Stabilization Program 2 (NSP2) through the American Recovery and Reinvestment Act of 2009 (ARRA). NSP2 was designed to reinforce NSP1 activities and also to refine specific provisions in the program. NSP2 grants were awarded to 56 grantees throughout the country. The 56 grantees cover 3,068 census tracts in 133 counties and 29 states. Of the $1.93 billion in NSP2 funding, approximately $947 million went to 24 grantees operating exclusively in states that were hit hardest by the national foreclosure crisis—California, Florida, Michigan, Nevada and Ohio.1
In February 2011, the U.S. Department of Housing and Urban Development (HUD) commissioned a study to evaluate NSP, with a particular focus on the second round of funding (NSP2). HUD’s evaluation of NSP will likely have lasting implications. The evaluation explores the efficacy of Federal and local policy interventions to arrest neighborhood distress, and thus has the potential to shape the direction of community stabilization efforts for many years to come.
OMB approved the first phase of the data collection process related to this evaluation in February 2012 (OMB Control No. 2528-0282). That effort included a first round of site visits and collection of community-level data. This is a request to conduct phase two of the data collection process: site visits to conduct interviews with staff at NSP2 grantees and to compile property-level data on NSP2-funded activities.
The in-person interviews are necessary to complete a key component of the study—the implementation analysis. The implementation analysis is designed to answer the general question of how grantees implemented NSP2. More specifically, it will answer these research questions:
What activities are grantees pursuing, what outcomes do they expect, what outcomes have been achieved, and who is benefiting from the program?
Are other funds being successfully leveraged by NSP2 funds?
How do grantee characteristics, including pre-existing staff capacity, training, technical assistance, use of innovative technology tools, and partnerships (with other governments, non-profits, and for-profits), contribute to the likelihood of grantees achieving their intended outcomes?
It will also provide a profile of NSP2 grantees and the grant activities they are conducting, as well as establish the context needed to fully understand the results of the impact analysis. For example, the impact analysis will measure the degree of geographic concentration of NSP2 activity and its relationship with measures of neighborhood stability, but the implementation analysis will allow us to understand why grantees pursued strategies of more and less concentrated activity.
This critical information is heavily descriptive and nuanced, and thus is best obtained through open-ended questions with respondents that allow the interviewer to probe as needed. Based on each response, interviewers may follow-up with additional questions to clarify the meaning of the response or probe on topics raised by the respondent that are not addressed elsewhere in the interview protocol. In addition, it is common for respondents to provide divergent responses to the same questions, and the in-person interviewing process allows the interviewer to reconcile the differing perspectives as best as possible. In short, a dialogue between an interviewer and a respondent is the best way to collect descriptive information about how a program was implemented.
Participation in the interviews is voluntary for each entity. The introduction to each of the discussion guides informs each of the potential respondents that “Your participation in this interview is voluntary.”
The main objective of this study – the Neighborhood Stabilization Program (NSP) Tracking Panel – is to evaluate the impacts of NSP2 on neighborhoods that were hit hardest by the national foreclosure crisis. Specifically, the study will address four research questions:
How was NSP2 implemented by grantees?
What outcomes were realized through the NSP2 program?
How would neighborhoods likely have progressed in the absence of the NSP2 program?
What factors are associated with program success in terms of both neighborhood stability and program production outputs?
To answer these questions, researchers will use qualitative and quantitative research approaches to execute five key components:
A descriptive overview of all 56 NSP2 grantees. Using information collected from NSP2 grant applications and through HUD’s Disaster Recovery Grant Reporting (DRGR) system, the project will describe the attributes and outputs of all 56 NSP2 grantees to provide a profile of the program.
An implementation assessment of the NSP2 grantees in the study sample. Two rounds of site visits will be conducted during the course of the project, providing ample opportunities to interview grantee staff and other key stakeholders about their implementation strategies and NSP2 activities. Separate OMB clearance packages have been submitted for each round of site visits. A previous clearance package for the first round of site visits was approved in February 2012. That portion of the data collection effort began in March and concluded in August 2012. This OMB clearance package covers the second round of site visits, which are planned for spring 2013, immediately following the final expenditures of funds by NSP grantees. At this point the property-level data will be readily accessible, and grantees will have clear impressions of their experiences in implementing the NSP2 grant.
A descriptive outcomes analysis of NSP2 neighborhoods and comparison neighborhoods in the study sample. The descriptive outcome analysis will use administrative data to document how NSP2 neighborhoods changed during the period of the housing boom and bust, and the subsequent intervention by NSP2. The descriptive outcome analysis will also compare the trajectories of NSP2 neighborhoods with other tracts in the counties, including the carefully selected comparison tracts.
An impact analysis of NSP2 activities within the study sample. The goal of the impact analysis is to understand the effect that NSP2 had on neighborhoods. The impact analysis will empirically measure the effects of NSP2 activities on property- and tract-level outcomes using administrative data from several sources.
A cross-site analysis of the results. The cross-site analysis will synthesize the results from the various project components across different housing market and grantee types. The object of this analysis is to discover broader conclusions of how NSP2 worked in a variety of housing market settings, methods of intervention, and grantee types.
As suggested by these five project components, the evaluation of the NSP2 program is data intensive. The evaluation relies on program data maintained by HUD, administrative data from the U.S. Census Bureau, property-level data purchased from private vendors and supplemented by grantees, crime data from local police departments and other local data, and quantitative and qualitative data from on-site interviews with grantees and other stakeholders. HUD’s contractors for the study—Abt Associates, the University of Southern California, and the Concentrance Consulting Group—have already secured access to the program data maintained by HUD and property-level data from private vendors. Data from the U.S. Census Bureau are public. Other data collection activities require OMB approval.
To match treatment and comparison tracts, we will calculate propensity scores of treated and non-treated eligible tracts and choose the non-treated tracts with the highest matching scores as the comparison tracts. In brief, propensity scoring is a statistical method that matches pairs of observations that are similar along a range of observable characteristics (such as prior level of foreclosures, housing stock and demographic characteristics) but differ in a key outcome (whether or not they received NSP2 activity). Because the matched pairs are similar along relevant observable characteristics, the non-treated tracts are assumed to provide a good counterfactual to the treated tracts.
The first stage of the matching process is to calculate propensity scores using a Probit or Logit model with the following form:
(1) Prob(ITreati =1|Xit) = β0 + β1Xit + ε
Where ITreat is a dummy variable indicating treatment status (1 = NSP2, 0 = Non-NSP2). The control variables Xit include all pre-NSP2 characteristics, described in the Exhibit below. The predicted probability from this first stage regression is the propensity score for each tract, which can be interpreted as the probability of a tract receiving NSP2 investment, conditional on observable characteristics. For instance, a propensity score of 0.7 would indicate a predicted probability of 0.7 that a tract with those characteristics would receive NSP2 activity.
We will examine the distribution of propensity scores to determine whether some non-treated tracts have similar propensity scores to treated tracts. If so, those tracts would be flagged as comparison tracts. The exhibit below shows the key factors that will be used to define the covariates included in the propensity score matching model. The second column lists the key tract-level variables associated with each factor. The variable lists illustrate the key variables that will be collected for each factor.
Key Variables for Propensity Score Matching
Factor |
Key Variables |
Data Source |
Foreclosure risk |
Foreclosure risk score; Vacancy risk score; Attributes used construct the scores |
HUD NSP data |
Housing prices and sales volume |
Tract average, median, and standard deviation |
Corelogic |
Mortgage characteristics |
Number and types of loan originated and purchased; Subprime loan share; Estimated degree of housing leverage |
HMDA; Corelogic |
Housing stock characteristics |
Total housing units; Mixture of structure type; Average age of units; Occupancy/vacancy rate; Homeownership rate; Unit size |
Census/ACS; USPS |
Demographic and economic characteristics |
Tract population; Race/ethnicity; Income; Poverty rate; Unemployment; Educational attainment; Household composition |
Census/ACS |
Locational characteristics |
Presence of subway/light rail stops; Distance to central business district and other employment centers |
GIS Shapefiles |
Note: The baseline measure for each variable will be 2009 (i.e., prior the start of the NSP2 program), with additional information collected in prior years when historical data is available. |
||
This is a request to conduct phase two of the data collection process, which includes two components:
Follow-up site-visit interviews (round two) with a sample of NSP2 grantees and their partner organizations, and
Collection of property-level data on NSP2 activities from a sample of NSP2 grantees and their partner organizations.
The purpose of the second phase of data collection is two-fold: (1) to understand how the implementation of the NSP2 program evolved since the baseline interviews; and (2) to collect property-level data on NSP2 activities from grantees that will be used to measure impacts. Both activities will occur during a second round of site visits to each NSP2 grantee in the study.
In the previous data collection effort, we selected the study sample and conducted a first round of on-site interviews with representatives from the lead NSP2 grantees and their partner organizations. Once the NSP2 grant period is over, researchers will conduct a second round of on-site interviews with representatives from the same lead grantees and partner organizations that participated in the first round of site visits. Two interview guides--one for the staff that designed the NSP program and the other for operations staff--are presented in Appendix B. Appendix C provides the question-by-question justification.
One of the key objectives of the implementation study is to understand how NSP2 grantees approached their neighborhood stabilization efforts. NSP2 grantees prioritized their interventions differently and attempted to tailor the NSP2 assistance to align with the challenges facing their targeted neighborhoods. Put differently, NSP2 grantees purposively designed their interventions differently. Thus, the inclusion of a protocol specifically for the design staff allows the study team to discuss: grantees’ overall strategies (e.g., what were they trying to accomplish and why?), the factors that shaped those strategies (e.g., what factors influenced grantees’ decision-making processes regarding the design of the NSP approach), and the rationale for pursuing the NSP-eligible activities they did (e.g., why did they target a particular NSP activity to a particular neighborhood?). In short, interviewing design staff will provide an understanding of how grantees designed their strategies and the evolution of the strategy over time.
Information about each grantee’s overarching design approach may or may not be known by local staff who operate the program. Indeed, local staff who were involved in the planning of the NSP strategy and the subsequent development of the grant applications are often senior management-level staff (e.g., executive directors) who understand the “big-picture.” These are staff who can articulate the grantee’s rationale for: establishing specific partnership arrangements to apply for and implement NSP2; selecting targeted tracts and the degree of geographic concentration of NSP2 activity; augmenting their organizational capacity as needed to successfully execute the program; leveraging other funding sources to support their NSP2 activities; and coordinating with concurrent activities undertaken by grantees or other organizations to spur neighborhood revitalization. By contrast, operations staff are typically program managers or middle-management staff who are more knowledge about the program’s implementation status and the factors that may be facilitating or hindering the implementation.
The two rounds of qualitative interviews will serve as the study’s main source of information about the NSP2 program implementation process. As discussed in more detail below, the primary goals of the follow-up interviews are to:
Gather detailed information on grantees’ overall strategy, program design, partnership arrangements, obstacles encountered, and implementation.
Collect grantees’ observations on program impacts in targeted neighborhoods.
Contribute to our understanding of the factors that could potentially affect the success of NSP2 programs. These factors include pre-existing organizational capacity and coordination with other neighborhood revitalization activities and the use of other funding to support NSP2 activities.
Implementation experience. The site visits will gather information on the actual activities grantees pursed. If grantees’ strategies deviated from their baseline plans, the follow-up interview will discuss grantees’ rationale for making these modifications (e.g., changing market conditions or changes in staff capacity). Similarly, the site visits will examine the challenges grantees encountered, how they responded to these challenges, what activities they felt were the most and least successful, and the technical assistance they used and found the most helpful.
The follow-up interviews will help HUD determine how well grantees were able to implement their initial NSP2 strategies and the reasons for their successes or setbacks. We will also enhance our understanding from the baseline interviews of why grantees have chosen the methods actually pursued and determine whether there are common constraints or enabling factors for particular activities or strategies. Ultimately, this understanding will be used to help determine whether the types of NSP-eligible activities pursued are a factor in the relative success of NSP2 programs across grantees in the sample. Last, the interviews will review grantees’ use of technical assistance, for an understanding of whether and how grantees used technical assistance and whether this was a factor in program success.
Program outcomes and impact. The site visits will explore the neighborhood outcomes that resulted from the grantee's NSP2 efforts. This includes measured production outputs such as the number of housing units rehabilitated and low-income beneficiaries served, as well as other outcomes not reported in DRGR such as job creation, support for sustainable development, and integration with other federal policy initiatives. In addition to obtaining and verifying detailed quantitative information on grantees’ actual progress and outcomes, these interviews will allow us to gather detailed qualitative information on grantees’ perceptions of their neighborhood impacts and the reasons for any differences between planned and actual outcomes.
The implementation analysis will use this information to compare grantees’ planned and achieved production outputs within the context provided by grantees about the market conditions, challenges, and other factors affecting NSP2 implementation.
Factors that could potentially affect the success of NSP2 programs. There are several factors that could influence the success of the program, and HUD is interested in one factor in particular: the degree of geographic concentration of NSP2 activity. Put differently, does the concentration of NSP2 resources affect the likelihood of producing noticeable (and measureable) impacts in the neighborhoods were these activities occur? At a basic level, the site visits will confirm and provide context to the data collected from DRGR and grantees’ management information systems (MIS). It will also provide an indication of whether grantees intended to geographically concentrate their NSP2 activity, and if so, whether they were successful in doing this. Last, this portion of the interviews will explore whether grantees found there were some neighborhoods where the intervention was more successful than others. If so, what distinguished these neighborhoods?
NSP2 lessons learned. The follow-up site visits will provide us the opportunity to understand the lessons the grantees consider that they have learned from their experience with NSP2, especially regarding how to structure future efforts to stabilize and revitalize distressed neighborhoods. For example, the interview will explore grantees’ views of the optimal level and type of investments, partnerships, and staff capacity.
Following the in-person interviews with grantees and their partners, a study team member will remain on-site and collect property-level data on NSP2 activities from the 25 local NSP2 grantees in the sample. (In addition, the same data will be collected from the four national grantees remotely.) Key data points include the property address, property characteristics, amount of NSP2 grant spending per property, and NSP2 activity start and completion dates per property.
The process for collecting data from grantees will differ depending on the storage location and formats used by each grantee. NSP2 grantees store data in MIS, spreadsheets, word processing documents, and paper files. These files are sometimes centralized with the lead grantee and sometimes maintained by the grantees’ partners. There are two basic approaches to collecting these data:
Local representatives produce the data. During the first round of interviews, representatives from some NSP2 grantees expressed a preference to do the data collection themselves. These grantees typically had most, if not all, of their data in electronic format and believed that grantee staff could efficiently produce the data with minimal burden. To facilitate this process, the study team has developed a spreadsheet with all the required data points. The spreadsheet has been designed to ease the reporting process by including user-friendly instructions, imbedded pull-down menus with prescribed data fields, and preformatted date and currency fields. This spreadsheet is provided in Appendix A. We expect that approximately half of the grantees will prefer to record the data themselves.
Study team member records the data. Based on the first round of site visits, it appears that many grantees store the needed data in various spreadsheets that can be printed and given to the study team member for extraction or transcription of the relevant data into the research team’s data collection spreadsheet. In a few cases, the study team member will need to review hard-copy files to extract the needed information and enter it into the spreadsheet. In all cases, the study team member will work with a designated data person at each of the local NSP2 grantees to compile the electronic and hard-copy files for each NSP2 property. The local representatives will be asked to pull the files for review by the study team member, but will not be asked to record the data.
For property-level NSP2 activity data, an option is provided where a study member reviews the grantee’s hard copy files and then records the data in a spreadsheet on behalf of the grantee. This option is included to provide grantees with more flexibility in reporting the property-level data to the study team. We expect many grantees to complete this process on their own, but it is also possible that some grantees will be less positioned to compile these data for the study. For example, grantees with staffing, resources, or scheduling constraints may be unable to compile the data in a timely manner. In these situations, the study team may need to assist with the property-level data collection.
The information collected during phase two of the NSP2 evaluation will be used by HUD’s research team: Abt Associates, the University of Southern California and the Concentrance Consulting Group. The information will be compiled and analyzed to execute the six project components described in section A2.1.
Three instruments have been developed for the grantee data collection and the follow-up interviews. The first collects property-level data on NSP2 activities and is summarized in Exhibit 1, along with explanations of the reasons the data are needed. This is a data worksheet constructed in Excel that provides a standardized template for inputting property-level data from grantees. These data will be used to supplement the data purchased from private vendors and many will be used as covariates in the regression analyses. A copy of the data worksheet is provided in Appendix A.
Exhibit 1: Justification for Collection of Property-Level Data from NSP Grantees
Worksheet Section |
Sub-Sections |
Respondents, Content, and Reason for Inclusion |
Grantee Property-Level Data on NSP2 Activities |
Property Characteristics |
Respondents: Data person at each of the 29 NSP2 grantees Content: Attributes of properties and NSP2 interventions for each property with an NSP2 intervention. Attributes include:
Reason: The impact analyses require information on the location, intensity, and timing of NSP2 interventions. The grantees are the only source of this information at the property level. |
Acquisition and Rehabilitation |
||
Financing Activities |
||
Redevelopment |
||
Demolition |
||
Land Banking |
In addition, two interview topic guides have been developed for the follow-up site visits, one for program design staff and the other for program operations staff, with each instrument to be used for staff both at the primary NSP2 grantee and at its partner organizations. The content of the instruments is similar, and they are summarized in Exhibit 2, along with the reasons the information is needed. The instruments review changes in the organization’s overall NSP2 strategy since the first found of site visits, implementation obstacles and challenges, partnership arrangements, neighborhood outcomes and impact, and lessons learned from the NSP2 experience. The complete follow-up interview protocols are provided in Appendix B.
Exhibit 2: Justification of Follow-Up On-Site Interview Topic Guides
Sub-Section |
Respondents |
Content and Reason for Inclusion |
Introduction |
Program design staff
Program operations staff |
Content:
Reason: To ensure the respondents understand the purpose of the interview and their rights as interviewees. |
Overall NSP2 Strategy |
Program design staff
|
Content:
Reason: To determine changes to the organization’s overall strategy and to determine how well the organization was able to implement its initial strategy. |
Partnership Arrangements |
Program design staff
|
Content:
Reason: To identify partnership types and structures, to understand partners’ roles in implementing the program, and to understand how grantees believe their partnership arrangements affected their success in implementing NSP2. |
Target Areas |
Program design staff
|
Content:
Reason: To explore changes in grantees’ rationale for picking targeted census tracts for NSP2 assistance and the degree of concentrating their NSP2 activity, as opposed to a dispersed approach. Also to help understand changes in strategy. |
Obstacles, Challenges, and Supports |
Program design staff
Program operations staff |
Content:
Reason: To determine organization’s overall implementation experience and inform the identification of promising practices for future redevelopment efforts. |
Concurrent Neighborhood Revitalization Activities/ Funding/TA |
Program design staff |
Content:
Reason: To identify changes in other spending on neighborhood revitalization and foreclosure-related activities within targeted tracts and comparison tracts. To identify changes in the sources and amounts of non-NSP2 funding that the grantee has leveraged, and how this affected their NSP2 implementation. To understand the use of technical assistance provided by HUD and how this affected grantees’ ability to accomplish NSP2 activities. |
NSP2 Outcomes and Impact |
Program design staff |
Content:
Reason: To gather grantee and partner accomplishments to date and expectations regarding ultimate program outcomes and neighborhood impacts. |
NSP2 Activity Process |
Program operations staff |
Content:
Reason: To document changes to the process grantees and their partners used in carrying out NSP2 activities, the target population for each activity, and understand challenges faced in conducting NSP2 activities. |
Lessons Learned |
Program design staff
Program operations staff |
Content:
Reason: To gather grantees’ observations and opinions at the end of the grant period about how various factors affected implementation. To gather grantees’ opinions about the scale of intervention needed to stabilize neighborhoods. |
Wrap-Up/ Conclusion |
Program design staff
Program operations staff |
Content:
Reason: Allows the interviewees to ask questions or provide comments and input on topics that were not previously discussed. |
Appendix C provides an item-by-item justification for each question in the interview protocols.
An automated electronic interview format is inappropriate for the on-site interviews with grantees and partners. Instead, these interviews will be administered using follow-up interview guides that allow interviewers the flexibility to accommodate the diverse range of NSP2 grantees interviewed and qualitative information collected. However, this data collection method will not result in increased burden for respondents. Research staff will be fully responsible for documenting participants’ qualitative responses and will later input them in the qualitative research software NVivo.
Wherever possible, property-level data from grantees will be requested in electronic format, ideally using an MIS query.
As part of the design process for the study, HUD and contractor researchers discussed possible areas where duplicative data collection activities were occurring through other research and program implementation efforts. HUD is not aware of any other similar data collection being undertaken for applicants and recipients of NSP2 funds.
In addition to avoiding the duplication of external data collection efforts, HUD’s researchers are committed to reducing any duplication of data grantees are already reporting to HUD. The study’s data collection activities supplement and extend beyond what grantees currently report to HUD. NSP grantees are required to submit an action plan and quarterly performance reports (QPRs) to HUD. These reports occasionally include property addresses, but never with the accompanying property-related information necessary to measure the impact of NSP2 on neighborhoods such as property characteristics, the amount of NSP2 grant spending per property, and activity start and completion dates.
For the on-site interviews, researchers will gather all the information available from pre-existing sources, including grantees’ action plans, QPRs, and applications, so that these data collection activities will only ask participants for information that they alone can provide.
Researchers will ensure that the follow-up on-site interviews avoid duplicating the baseline interviews by reviewing baseline interview notes and summaries. Further, researchers’ experience with the baseline on-site interviews has been used to inform the development of the follow-up interview protocols. For example, researchers learned that lead grantees’ partners were often involved in the design of the NSP2 application and strategy and often have a great deal of autonomy, so design-related questions are appropriate. As a result, HUD’s researchers reduced the number of interview instruments from three to two. Specific questions also build on what researchers learned during the baseline interview and avoid repeating previous information-gathering.
The on-site interviews and collection of property-level data may include small entities. To reduce the burden on these smaller entities, as well as larger organizations participating in the study, the information being requested has been held to the minimum required for the intended use of the data. In addition, research staff will conduct as much of the property-level data collection as possible, including looking up information in paper files and photocopying records where necessary in order to minimize the burden on grantees.
The follow-up interviews will be administered once only in the spring of 2013. The risk of not administering these interviews at all is that the study will not have an understanding of grantees’ implementation progress and program outcomes and how these compared with expectations, information that is needed to conduct the implementation analysis.
Grantees’ property-level data on NSP2 activity will also be conducted only once, in the spring of 2013, immediately following the final expenditures of funds by NSP2 grantees. These data are necessary to measure the size and timing of the NSP2 treatment in targeted census tracts, both of which are essential to analyze the impact of the program.
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public—General Information Collection Guidelines). There are no special circumstances that require deviation from these guidelines.
In accordance with the Paperwork Reduction Act of 1995, HUD published a notice in the Federal Register on September 10, 2012 (see Appendix D), announcing the agency’s intention to request an OMB review of data collection activities for site visits for the Neighborhood Stabilization Program Tracking Study. The notice provided a 60-day period for public comments; none were received.
HUD’s researchers developed the study design and instruments in consultation with staff from HUD and an expert panel of researchers. The panel was composed of academics, policy experts, and methodologists who have relevant substantive and technical experience with the NSP2 evaluation. In addition, HUD staff also reviewed the data collection instruments to ensure that each is clear, flows well, and is as concise as possible. The experience with the first round of on-site interviews also informed the development of the follow-up protocols.
This study will not use incentive payments.
HUD’s contractors take seriously the responsibility to protect the subjects they interview. The data collection plan and instruments were exempt from an Institutional Review Board because the information being collected for the study is program-level data, rather than individual-level information.
Grantee and partner organizations may be identified in the study reports, but the individual respondents within each entity will not be identified.
The site visit interview protocols do not contain questions of a sensitive nature (personal data), but rather focus on program-level topics about how the NSP program was implemented.
The estimate of burden hours for the on-site interviews is shown in Exhibit 3.
Researchers will conduct 20 site visits involving 25 grantees and approximately 50 partner organizations (an average of 2 partners per grantee), as well as four telephone interviews with national grantees. During the site visits and phone interviews, researchers will conduct interviews with two types of personnel at the local level-- one with program design staff and one with program operations staff – with each local NSP2 grantee, local partner agency, and national NSP2 grantee. Accordingly, there are two corresponding interview guides presented in Appendix B. Based on researchers’ experience conducting the baseline site visits, we anticipate that on average, two program design staff and two program operations staff will participate in the on-site interviews (4 staff members total), and one program design staff and one program operations staff will participate in the telephone interviews (2 staff members total). The number of responses entity and hours per response are shown in the top half of Exhibit 3.
As described in section A2.2, the collection of the property-level data on NSP2 activities will be conducted either by grantee representatives or by the researcher who will remain on site after the interviews. Approximately one-half of the 29 grantees (or 14 grantees) and 25 partner organizations will likely chose to report the required data themselves via the study’s preformatted spreadsheet. HUD estimates that each spreadsheet will take one person about 1.5 working days (12 hours) to complete, on average.
For the remaining 15 grantees and 25 partner organizations, the data will be compiled by the research team with the support of local representatives. Most of this effort will be conducted by the researcher. HUD estimates that it will take approximately two hours per grantee and per partner organization to support the researcher during this time (e.g., pulling the appropriate files).
The burden table below shows the hours associated with each collection. The interviews with local NSP2 grantees, local partner agencies, and national NSP2 grantees are conducted with program managers and senior staff. The cost estimates assume that the average cost of labor time for such staff is $50 per hour. The collection of property-level data involves data analysts or junior staff to support data collection. The cost estimates assume that the average cost of labor time for such staff is $30 per hour. As shown in the figure, the burden estimates imply a total monetary cost of $47,240 for the grantee agencies’ staff time associated with data collection.
Costs to the federal government are discussed in more detail in Question 6 below.
Subtotal and total rows have been added to the burden summary table (see below):
|
Number of entities |
Responses per entity |
Hours per response |
Total hours |
Cost per hour |
Estimated total cost |
Interviews with local NSP2 grantees |
25 |
4 |
2 |
200 |
$50 |
$10,000 |
Interviews with local partner agencies |
50 |
4 |
2 |
400 |
$50 |
$20,000 |
Interviews with national NSP2 grantees |
4 |
2 |
2 |
16 |
$50 |
$800 |
Interview Subtotal |
79 |
2 to 4 |
2 |
616 |
|
$30,800 |
Collection of property-level data: compiled by local representative |
39 |
1 |
12 |
468 |
$30 |
$14,040 |
Collection of property-level data: compiled by HUD contractor |
40 |
1 |
2 |
80 |
$30 |
$2,400 |
Property-Level Data-Collection Subtotal |
79 |
1 |
7 (avg.) |
548 |
|
$16,440 |
Total |
79 |
3 to 5 |
3 (avg.) |
1,164 |
|
$47,240 |
There is no cost to respondents other than the time required to participate in the on-site interviews and assist researchers with collection of property-level data.
The cost to the federal government associated directly with the execution of the second round of site visits is $426,186. These costs include the following activities:
Refine the interview protocols and site summary forms for completion after each site visit;
Developing the data collection tools for the property-level data;
Developing training materials and conducting a site-visitors training for 17 staff members;
Scheduling interviews with 79 entities located in 20 counties nationwide;
Conducting on-site interviews in 20 counties with approximately 790 individuals who are involved in the design, operations, execution and management of the NSP2 program ;
All airfare, lodging, and transportation costs for staff members;
Providing typed notes for each interview;
Completing a site summary form for each NSP2 grantee that synthesizes the information from the interviews.
To complete these activities, the study team assumed the following level of effort: 40 hours to develop the interview protocols and site summary forms; 40 hours to develop and test the data collection tool; 4 hour training and prep time for all site visitors; 2-person site-visiting teams to each entity; 1 to 4 days on-site, depending on the availability of property-level data; up to one day of travel time; 40 hours total per site-visiting team to produce the interview notes, develop the site summaries, and review and submit the property-level data file.
The amount initially cited in Question 14 ($784,884) also includes several other activities related to the identification of the sample sites, which include: the purchasing of county-level administrative data (e.g., foreclosure, sales transaction, and vacancy) from private data vendors; compiling other administrative data (e.g., education and crime data) for the study areas from on-line websites, and reviewing the quality of the various administrative data to identify good candidates for the study.
This submission to OMB is a new request for approval; there is no change in burden.
The information collected from the follow-up site visits will be used in conjunction with several other data sources, including baseline site visits, to evaluate the impact of the NSP2 program on neighborhood stability. The specific use of these data sources within each evaluation component is described below.
The implementation analysis will use data from the follow-up on-site interviews (and the baseline on-site interviews conducted previously) to create tabulations that compare components of grantees’ implementation expectations to grantees’ actual implementation experiences. Data from the follow-up on-site interviews that will be used in these analyses include 1) the NSP2 eligible activities the grantees ultimately pursued; 2) the outcomes the grantees achieved; 3) the non-NPS2 funds the grantees leveraged (both federal and non-federal); 4) and the characteristics of individuals that directly benefited from the programs. In addition to these tabulations, this data will also be used in a qualitative analysis to identify and summarize common themes among each of these components of grantees’ implementation expectations.
The implementation analysis will also use cross-tabulations to examine how several factors affect the likelihood that grantees achieved their intended outcomes. Data from the follow-up on-site interviews that will be analyzed include: 1) pre-existing staff capacity; 2) training; 3) technical assistance; 4) use of innovative technology tools; 5) partnership types; 6) level of concentrated NSP2 activities; and 7) the presence and degree of other intervention in grantees’ targeted neighborhoods.
The impacts of the NSP2 program are likely to appear across a range of neighborhood characteristics. The impact analyses, therefore, examine the impact of NSP2 activities with respect to multiple neighborhood stability outcomes: home sales and prices; additional foreclosures; crime; housing tenure; and vacancy. The analyses will be conducted at the property level. In addition, for each outcome, we will conduct analyses that identify the impact of NSP2 activities on tract-level outcomes in NSP2 tracts, and contrast these with the outcomes in comparison tracts.
The property-level data on NSP2 activities collected from grantees will be used to identify NSP2 properties for this analysis, measure the intensity of the NSP2 treatment, and pinpoint the appropriate timeframe for analysis. In addition, the information obtained during the site visits regarding each grantee’s organizational capacity, partnership arrangements, and other factors that may influence the program’s success will be used in the multivariate regression.
A second line of inquiry is the tract-level analysis, which will attempt to identify the cumulative impact of NSP2 spending on tract-level outcomes. Where the property-level analysis identifies the impact of a nearby foreclosure on a home’s property value, the tract-level analysis identifies the impact of all NSP2 activities on the overall level of home prices in the tract. The analysis compares outcomes before and after NSP2 across treatment tracts that received NSP2 and comparison tracts that did not but were otherwise similar. The difference between the tract-level impact analysis and the descriptive outcome analysis is causation; here the focus is on attributing tract-level impacts to NSP2.
The tract-level analysis estimates the average difference in tract-level outcome measures between tracts with NSP2 properties and comparison tracts. The unit of analysis for this model is the census tract in each year that data is collected.
Exhibit 4 presents the research questions and analyses that will be conducted with respect to each of the neighborhood stability outcomes. The first column presents the specific research questions to be addressed. For each research question, the remaining columns show the analyses to be conducted, the key measures, and the data sources for the key measures.
The analyses presented in Exhibit 4 focus on property-level analyses. For each outcome, we will also conduct analyses that identify the impact of NSP2 activities on tract-level outcomes in NSP2 tracts and comparison tracts. The empirical approach for the tract-level analyses is similar across outcomes.
Research Question |
Analysis Approach |
Key Variables |
Data Source |
Home Prices/Sales |
|||
1. Do foreclosures impact the values of neighboring homes? |
Estimate the impact of nearby foreclosures on home prices using a hedonic model of home values |
Location of all foreclosures and dates of notice of default, notice of trustee sale, and sale of REO property; Dates and sales price for all transactions. |
DataQuick; RealtyTrac |
2. Does NSP2 activity diminish the impact of foreclosure? |
Separately estimate the spillover effect of NSP2 properties and non-NSP foreclosures |
Location of NSP2 properties and dates of activities. |
Site visits to NSP2 grantees |
3. Is the impact of NSP2 stronger in areas with concentrated activities? |
Define the NSP2 measure in the estimated hedonic regressions to reflect the concentration of NSP2 activities |
Location of NSP2 properties and dates of activities. |
Site visits to NSP2 grantees |
4. What is the mechanism for NSP2 impact--signaling stability, increasing maintenance, shortening vacancy, reducing uncertainty, etc. |
Use cross-site analysis to describe findings across counties and to identify opportunities to examine robustness to variation in the hypothesized mechanisms |
Same as previous analyses |
DataQuick; RealtyTrac; Site visits to grantees |
Additional Foreclosures |
|||
1. Does NSP2 activity impact the number of foreclosures that appear in the targeted tracts? |
Reduced form model of the impact of NSP2 activities on the likelihood of foreclosure |
Location of all foreclosures; Location of NSP2 properties and dates of activities |
DataQuick; RealtyTrac; Site visits to grantees |
Crime Rates |
|||
1. Did foreclosures (and vacancies that result from foreclosures) result in increased crime? |
Identify the impact of foreclosures on crime using a property-level model that predicts the incidence of nearby crime. Compare the effect for foreclosures that result in vacancy and those that do not. |
Location and date for each violent crime, property crime, and public order crime |
Contacting local police departments |
2. To what extent do NSP2 activities mitigate the impact of foreclosures on crime? |
Estimate an extension of the previous model that separates NSP2 activities from other foreclosures. |
Location of NSP2 properties and dates of activities. |
Site visits to NSP2 grantees |
Vacancy and Tenure |
|||
1. Do changes in vacancy and tenure status accompany foreclosures in NSP2 neighborhoods? |
Document the presence of vacancies and changes in tenure status among foreclosed properties. |
Tenure status prior to foreclosure; Tenure status during NSP2 activities; Final tenure status |
DataQuick; RealtyTrac; USPS vacancy data |
2. Did NSP2 activities affect the vacancy and tenure status of NSP2 properties relative to other foreclosures? |
Use Logit analysis to compare the likelihood of vacancy and changes in tenure status between NSP2 properties and other foreclosed properties. |
Tenure status prior to foreclosure; Tenure status during NSP2 activities; Final tenure status |
DataQuick; RealtyTrac; USPS vacancy data |
Blight |
|||
1. How do the visual conditions of NSP2 neighborhoods change during the analysis period? |
See descriptive outcome analysis section |
Presence of visual blight on the property and block face |
Visual tracking survey |
Exhibit 4: Impact Analysis Research Questions and Analysis Approach
The cross-site analysis will utilize a combination of descriptive statistics and qualitative discussion, working from and incorporating results from the implementation and impact analyses. The object of this analysis is to come to broader conclusions on how NSP2 worked in a variety of housing market settings, methods of intervention, and grantee types. The data from the baseline on-site interview and crime data will be used to categorize NSP2 programs within groupings of grantee characteristics, housing market types, and neighborhood crime levels.
Phase one began in the spring of 2012 and ended in August 2012. The follow-up site visits will be conducted in March through May 2013, followed by data analysis through the fall 2013. The final report is due in February 2014.
This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
1 See: www.hud.gov/offices/cpd/communitydevelopment/programs/neighborhoodspg/pdf/nsp2awardtotalAllocation.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Melissa O'Connor |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |