Supporting Statement for Paperwork Reduction Act Submissions
Evaluation of the Supportive Services Demonstration
OMB Control # 2528-0321
B. Collections of Information Employing Statistical Methods
Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.
The Supportive Services Demonstration (SSD) evaluation has two components: a process study, to document how treatment group properties implemented the Integrated Wellness in Supportive Housing (IWISH) program and how property staff, residents, and caregivers responded to it; and an impact study, to measure the effect of IWISH on key outcomes related to residents’ use of healthcare services and housing stability.
The process study will focus on the 40 IWISH (treatment) properties and the 40 active control properties. The main data sources for the process study are questionnaires, interviews, and focus groups with program staff, residents, and caregivers. These data collection activities do not employ statistical methods. The research team will select respondents purposively. Other data sources include data collected directly by the research team with data collected by the IWISH properties, administrative data collected by the U.S. Department of Housing and Urban Development (HUD), and public use data.
The impact evaluation will analyze administrative data obtained for residents of all three demonstration groups—treatment, active control, and passive control. The main data sources for the impact study are Medicare Fee-For-Service claims, Medicaid Fee-For-Service claims, Medicare and Medicaid encounter data, HUD administrative data, and public use data to characterize the community. These data sources are not subject to the Paperwork Reduction Act (PRA) and are therefore not part of this ICR. They will be used to estimate the impact of the IWISH model on healthcare utilization, housing exits, and transfers to nursing homes and other long-term care settings. The impact of IWISH is the difference between the average outcomes among residents at IWISH properties and the average outcomes among similar residents in the control groups, estimated using experimental and quasi-experimental analysis methods. The community data will be used to ensure the treatment and control groups are evenly matched on community characteristics and for contextual analysis.
All of the data collection in this ICR will be done by Abt Associates and its subcontractor L&M Policy Research (the “research team”). The identification of respondents for each data collection activity is described below.
Identification of Respondents for the Interviews with Resident Wellness Directors
The research team plans to interview all Resident Wellness Directors working at the 40 IWISH properties. Most properties have one Resident Wellness Directors, but a few properties have more than one. The research team expects to interview up to 54 Resident Wellness Directors.
Identification of Respondents for the Interviews with Wellness Nurses
The research team plans to interview all Wellness Nurses working at the 40 IWISH properties. Most properties have one Wellness Nurse, but a few properties have more than one. The research team expects to interview up to 42 Wellness Nurses.
Identification of Respondents for the Interviews with Service Coordinators
The research team plans to interview all Service Coordinators working at the study’s 40 active control properties. Most properties have one Service Coordinator. The research team expects to interview up to 40 Service Coordinators.
Identification of Respondents for the Interviews with Service Coordinators
The research team plans to interview representatives from all of the owner organizations that own IWISH properties. There are 20 distinct owners for the 40 IWISH properties, as many owners have multiple properties in the demonstration. The research team expects to interview up to 20 owner organization representatives.
Describe the procedures for the collection of information including:
Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.
Process Study. The data collection covered by this submission is intended to provide qualitative information to support a process study of the implementation of the SSD. The process study will qualitatively describe how IWISH was implemented at the treatment sites and how it was experienced by program staff and owners, as well as how the IWISH implementation compares to the typical service coordination provided at HUD multifamily properties serving older adults.
The research team will not draw statistical inferences from the qualitative data covered in this ICR submission; instead, the analysis of administrative data will provide the main evidence of program outcomes and impacts.
Impact Study. Prior to engagement with Abt, HUD, designed a cluster-randomized controlled evaluation and randomly assigned eligible properties to treatment and control groups to support the evaluation. The number of demonstration treatment and active control sites was limited to 40 based on preliminary power analyses and budgetary constraints. The random assignment was designed and conducted to facilitate an impact evaluation such that causal inferences can be made.
To support the impact analysis, the research team will use linked HUD and Medicare and state Medicaid administrative data for HUD-assisted residents of the 40 treatment properties, the 40 active control properties, and the 44 passive control properties. The team will analyze approximately five years of person-level administrative data—two years of data prior to the start of the demonstration (October 2015 – September 2017) and three years of data covering the duration of the demonstration (October 2017 – September 2020).
To estimate the impact of IWISH on resident outcomes, the research team will conduct an intent-to-treat (ITT) analysis by comparing outcomes of residents in the treatment group and to those of residents in the active and passive control group properties, which will be pooled into one control group.
In addition to the main impact analyses, the research team will conduct additional exploratory analysis of key outcomes to examine: non-linear trends in the cumulative effect of IWISH on healthcare utilization and spending during the demonstration; potential heterogeneity of the treatment effect across important subgroups of individuals; and the extent to which sample attrition due to deaths might bias the estimated impact of IWISH on utilization and spending.
Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.
The research team expects a 95 percent or better response rate from the staff and owners to be interviewed, because the properties in the study have entered into cooperative agreements with HUD that require their participation in evaluation activities.
Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.
The research team do not plan to do a formal pretest of the interview guides, because most of the questions have been tested in prior rounds of data collection. However, the researchers will meet after the few interviews of each type to discuss any areas of confusion and revise the guides as needed for the remaining interviews.
Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.
HUD has contracted with Abt Associates to conduct the data collection. The data collection procedures will be similar to those used in other studies conducted by Abt Associates. The HUD Contracting Officer’s Representative (COR) reviewed all the procedures and had them reviewed by other subject matter experts at HUD. If there are any questions about this submission, please call either the HUD COR, Leah Lozier (202-402-3013) or the Abt Associates co-Principal Investigators, Gretchen Locke (617-349-2373) and Sara Galantowicz (617-520-2510).
In addition, Abt Associates and HUD have established an Expert Panel to review the evaluation design, progress, and findings, to maximize the rigor of the evaluation and its value to multiple stakeholders. The full membership of the panel is presented in Part A of this ICR. The following panelists have focused on the statistical aspects of the study’s design:
Partha Deb, PhD, Professor of Economics, Hunter College, City University of New York.
Kosuke Imai, PhD, Professor of Politics and Director of Program in Statistics and Machine Learning, Princeton University.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | h03483 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |