Supporting Statement for Paperwork Reduction Act Submissions
Evaluation of the Supportive Services Demonstration
(OMB# xxxx-xxxx)
A. Justification
This supporting statement provides information on baseline data collection activities associated with the Evaluation of the Supportive Services Demonstration.
1. Explain the circumstances that make the collection of information necessary.
The U.S. Department of Housing and Urban Development (HUD)’s Office of Policy Research and Development is undertaking an evaluation of the Supportive Services Demonstration (SSD). SSD is a three-year demonstration designed to test the impact of housing-based supportive services on the healthcare utilization and housing stability of low-income adults aged 62 and over. The demonstration offered grant funding to multifamily property owners to implement the Integrated Wellness in Supportive Housing (IWISH) model over the three-year demonstration period. The IWISH model features a full-time on-site Resident Wellness Director (RWD) with a part-time Wellness Nurse (WN) at each property funded to implement IWISH. The RWD and WN work together to implement a formal strategy for coordinating services to help residents meet their long-term care needs.
HUD designed the SSD as a cluster-randomized controlled trial to allow rigorous measurement of impacts. HUD published a Notice of Funding Availability (NOFA) in January 2016 for the demonstration, received more than 700 responses, and applied screening and ranking criteria described in the NOFA to identify 185 properties across seven states as eligible for random assignment. HUD assigned properties to three groups: a treatment group that received grant funding to hire the RWD and WN and implement the demonstration; an active control group that did not receive funding for implementation but received an incentive for participating in the evaluation; and a passive control group that received neither an implementation grant nor an incentive.
The final demonstration sample is 124 properties: 40 in the treatment group (also known as IWISH properties), 40 in the active control group, and 44 in the passive control group. All properties serve households headed by people aged 62 or over, either predominantly or exclusively. Most are funded through HUD’s Supportive Housing for the Elderly (Section 202) program. The properties are located in the following states: California, Illinois, Maryland, Massachusetts, Michigan, New Jersey, and South Carolina. Each state has treatment, active control, and passive control properties.
The demonstration formally began October 1, 2017. The 40 IWISH properties began enrolling residents, conducting resident needs assessments, and developing individualized healthy aging plans in late March 2018. The demonstration ends in September 2020.
HUD contracted with The Lewin Group to manage the implementation of the SSD. Data collection associated with the implementation of the demonstration is covered under a separate Information Collection Request (ICR). The ICR is entitled “HUD Supportive Services Demonstration/Integrated Wellness in Supportive Housing (IWISH)” (reference number 201702-2528-001) and was approved on February 28, 2018.
HUD contracted with Abt Associates Inc. for the evaluation of the SSD. The SSD evaluation will help determine whether offering access to on-site wellness staff, comprehensive health and wellness assessments and planning, and evidence-based services and programming for residents in project-based assisted housing is an effective way to support aging in place and, over the long-term, to reduce the use of costly or unnecessary health care services. Key measures of the SSD’s success will be whether the intervention reduces potentially avoidable hospitalizations and ambulance trips, delays transfers to costly settings such as nursing homes and other long-term care facilities and increases the share of time that residents spend in independent housing (versus medical facilities) as they age. The evaluation will also test for impacts on housing stability (fewer exits from housing due to health reasons, eviction, or death).
To determine the impact of IWISH on healthcare utilization and housing stability, the evaluation will compare outcomes for residents at treatment properties, where IWISH is implemented, to the outcomes of residents at the active and passive control properties, which represent “business as usual” for HUD multifamily elderly-designated properties.
The evaluation of the HUD SSD will take place over four and a half years, from October 2017 through March 2022. The evaluation has a qualitative component—the process study—designed to learn how treatment group properties implemented the IWISH model and how property staff and residents responded to it, and a quantitative component—the impact study—designed to measure the effect of the intervention on key outcomes related to residents’ use of healthcare services and housing stability.
This submission requests OMB approval for baseline data collection related to the process study. The purpose of the data collection for the process study is to learn about how the IWISH model was implemented (from the treatment properties) and how the IWISH implementation differs from “business as usual” service coordination in HUD multifamily housing serving elderly adults (from the active control properties).
The other component of the evaluation—the impact study—will use administrative data obtained for residents of all three demonstration groups to measure the outcomes and impacts of the demonstration. The administrative data collection related to the impact study is not part of this ICR.
The data collected for the process study (the focus of this ICR) is designed to provide important information about IWISH implementation inform the interpretation of the outcomes and impacts measured through the administrative data. The process study data will not be used to measure outcomes.
This submission requests OMB approval for the following baseline data collection activities supporting the process study component of the SSD evaluation:
Initial questionnaires
Staff interviews
Focus groups
These baseline data collection activities will begin in October 2018 (or as soon as OMB approval is received) and will end in approximately December 2019.
We will submit a second ICR in early 2020 to cover follow-up data collection to be conducted in summer 2020. We expect the follow-up data collection to consist of final questionnaires with IWISH and active control properties and interviews with community partners.
This research is conducted under the authority of the HUD Secretary to undertake programs of research, studies, testing and demonstration related to the mission and programs of HUD (12 USC 1701z-1 et seq.).
2. Indicate how, by whom and for what purpose the information is to be used Evaluation Overview
The SSD evaluation has two components: a process study, to document how treatment group properties implemented the demonstration and how property staff, residents, and caregivers responded to it; and an impact study, to measure the effect of the intervention on key outcomes related to residents’ use of healthcare services and housing stability.
Process Study
The process study will focus on the 40 IWISH (treatment) properties and the 40 active control properties. The process study is designed to collect information on how the IWISH model was implemented and how it differs from other models of service coordination being offered at the active control properties, as well as the perceived benefits of IWISH for residents, their caregivers, and property staff.
Six research questions guide the process study:
What are the experiences of resident wellness and property management staff with implementing the IWISH model?
What are the perceived benefits, strengths, and weaknesses of the IWISH model?
Within the treatment group, were there any changes in residents’ perceptions of their health, well-being, and satisfaction with housing and services?
Was the demonstration implemented with fidelity to the IWISH model across the treatment sites?
What factors explain or contribute to the observed variation in fidelity to the IWISH model across the treatment sites?
How does the service coordination and health and wellness programming provided at the IWISH sites differ from that provided at the active control properties?
The main data sources for the process study are those included in this ICR—questionnaires, interviews, and focus groups with program staff, residents, and caregivers. We also plan to submit a second ICR to cover follow-up questionnaires with program staff and interviews with service provider partners planned for summer 2020. We will supplement the data collected directly by the study team with data collected by the IWISH properties, HUD administrative data, and public use data.
The main analytic methods for the process study are: content analysis of interview and focus group data; descriptive analysis of administrative and service data; and scoring of properties along a continuum of resident wellness services.
Impact Study
The impact evaluation will analyze administrative data obtained for residents of all three demonstration groups—treatment, active control, and passive control—and use the cluster-randomized design of the demonstration to estimate the impact of the intervention on healthcare utilization and spending (including hospitalizations), housing exits, and transfers to nursing homes and other long-term care settings. The impact of the intervention is the difference between the average outcomes among residents at treatment properties and the average outcomes among similar residents in the control groups.
Three primary research questions guide the impact study:
Does IWISH reduce utilization of Medicare and Medicaid covered healthcare services, particularly use of high-cost, low-value, or potentially avoidable services?
Does IWISH reduce housing exits and lengthen resident tenure?
Does IWISH delay transitions to long-term institutional care?
The main data sources for the impact study are Medicare Fee-For-Service claims, Medicaid Fee-For-Service claims, Medicare and Medicaid encounter data, HUD administrative data, and public use data to characterize the community. These data sources are not subject to the PRA and are therefore not part of this ICR. The data from Medicare, Medicaid, and HUD will be used to estimate the impact of the IWISH model on healthcare utilization, housing exits, and transfers to nursing homes and other long-term care settings.
The impact of IWISH is the difference between the average outcomes among residents at IWISH properties and the average outcomes among similar residents in the control groups. We plan to conduct the following types of analyses:
Intent-to-treat (ITT) analysis, which estimates the impact of offering housing-based supportive services under the IWISH model by comparing outcomes for all residents of treatment and control group properties.
Treatment-on-the-treated (TOT) analysis, which estimates the effects of participating in the IWISH model (defined as enrollment in IWISH) using quasi-experimental, Instrumental Variable methods. TOT estimates will help assess whether the impact of IWISH on the outcomes for all residents are really driven by the outcomes for residents who enrolled in the model.
Study Deliverables
The SSD evaluation will result in two main reports—an Interim Report and a Comprehensive Report—as well as site reports following the site visits and interviews and several shorter reports to supplement the Comprehensive Report.
HUD and policy makers will use the information collected through the evaluation to understand the effectiveness and outcomes of the IWISH model. The evaluation will provide insight to Congress, HUD, grantee states, and other interested parties on issues to consider in providing housing-based supportive services. It will also provide rigorous, quantitative data on the impact of housing-based supportive services on healthcare utilization and housing stability among older adults in HUD-assisted housing.
Information Collection in This ICR
This ICR covers the baseline data collection supporting the process study. All of the data collection in this ICR will be done by Abt Associates and its subcontractor L&M Policy Research. Each data collection activity is described below, followed by a summary table presenting the justification for each data collection instrument.
Prior to starting data collection, we will reach out to the Authorized Organization Representative (AOR) of all SSD properties participating in the demonstration; to preview the evaluation activities and to identify a staff member (most likely the property manager) who will serve as the main point of contact for the study.
Initial Questionnaires
The first data collection activity is the initial questionnaire with the 40 treatment and 40 active control properties. The purpose of the initial questionnaire is to obtain standardized information across treatment and active control sites prior to conducting the in-depth interviews. The initial questionnaire collects information on the characteristics of each property and its residents and the service coordination activities at the property that cannot be obtained from administrative data sources.
The questionnaire will be interviewer-administered, meaning that a member of the evaluation team will read the questions over the telephone to respondents and record their responses directly into an online survey platform. Interviewers will receive training about the IWISH demonstration and the data collection tool prior to administering the questionnaire.
At the treatment properties, we will complete the questionnaires with the RWD. We expect the RWD to be able to answer most if not all of the questions, but we have built into the burden estimates for this request the possibility that the property manager may need to supplement the information provided by the RWD.
At the active control properties, we will complete the questionnaires with the service coordinator, if the property has one, or with the property manager, if the property does not have a service coordinator. We estimate that about 70 percent of active control properties have on-site service coordinators. As with the treatment properties, we have built into the burden estimates the possibility that the property manager may need to supplement the information provided by the service coordinator.
We will send a list of topics to respondents in advance via email so that respondents are prepared or can bring in others as needed. We will also send the questions from three sections of the questionnaire—the sections on property staffing, programs, and partnerships—to respondents via email so that they can complete these three sections at their own pace and to reduce the time during the interview. Other than those three sections, however, we will administer the questions on the telephone.
There are several reasons for not sending the complete questionnaire or more of the questionnaire to respondents to complete on their own. One reason is that we want to use this interview to build rapport with the respondents, as we expect to conduct a more in-depth interview with these staff via site visits the following year. Another reason is that while many of the questions are closed-ended, we have learned through pretesting the questionnaires with six respondents (in September 2018) that respondents are eager to add qualitative information on their experiences implementing IWISH or (in the case of the active control sites) other supportive services and programming for older adults. We also learned through the pretest that respondents interpret some questions differently, particularly questions about the features of the property that facilitate or hinder aging in place and those about the role of the RWD. Having the questionnaires administered by telephone by trained interviewers helps ensure that the questions are being understood and answered consistently by the designated respondent. Finally, among the seven pretest interviews conducted in late September, two respondents provided incomplete information on the materials sent out in advance. The respondents realized during the call that they had omitted important information; suggesting that a combination of select questions provided in advance followed up by the telephone call will yield the most complete information for the study.
We estimated that the questionnaires would take 45 minutes to an hour to administer, with an additional 30 minutes for scheduling and preparing for the questionnaire. Seven pretest interviews were conducted in September 2018 support these estimates, with one interview lasting just over an hour, one lasting just half an hour, and the other four lasting about 45 minutes.
We will begin fielding the initial questionnaire as soon as OMB approval is received. We expect to complete questionnaires with the 80 properties over a three-month field period—from October 2018 through December 2018. As of October 2018, the demonstration will have been underway for one year and treatment sites will have been enrolling IWISH participants for about seven months.
Data collected through the questionnaires will inform our analysis of the extent to which the IWISH model was implemented with fidelity at the treatment sites and how service coordination and programming varied between IWISH and active control sites. The data collected through the questionnaires will also provide key background information that will enable the in-depth staff interviews (planned for 2019 and discussed below) to be more efficient.
We have developed separate questionnaires for the treatment and the active control sites. The questionnaire for the treatment sites is in Appendix A. There are two questionnaires for active control properties—one for the service coordinator if the property has one and one for the property manager if the property does not have a service coordinator. Both questionnaires for the active control sites are in Appendix B.
In-Depth Staff Interviews
We plan to interview property staff at the 40 treatment properties and 40 active control properties on a rolling basis between April 2019 and December 2019. As of April, the demonstration will have been underway for a year and a half and a year will have passed since the start of enrollment at the treatment sites. By this time, the RWDs, WNs, and housing property staff at the treatment sites will have substantial experience with the program and be able to provide input into the key research questions for the process study. Parallel data collection at the active control properties will provide information for comparing service coordination and programming between IWISH and typical HUD multifamily properties.
Treatment Properties
The interviews with staff at treatment properties will be conducted in person via site visits. We will visit each treatment property once between April and December 2019. We will schedule site visits taking into consideration multiple factors, including: availability of RWD, WN, and housing property staff; locations of treatment and active control sites; and the implementation team’s site visit schedule.
While on site, we will conduct separate in-person interviews with RWDs, WNs, and housing property staff. We anticipate each interview will last up to two hours, with an additional 30 minutes for scheduling and preparing for the interview. We will use the interviews to gather information on: the experiences of on-site staff implementing IWISH, staff perceptions of the benefits, strengths, and weaknesses of the IWISH model, the specific activities and programming the RWD and WN have put in place, and contextual factors that affect local IWISH implementation such as the property’s history with service coordination and access to health and supportive services in the community. We will send a list of topics to respondents in advance so that respondents are prepared.
Trained research staff will conduct the interviews using separate interview guides for each type of respondent. The interview guide for the interviews with RWDs at treatment sites is provided in Appendix C. The interview guide for the interviews with WNs at treatment sites is provided in Appendix D. The interview guide for the interviews with housing property staff at treatment sites is provided in Appendix E. The interview guides include skip patterns to tailor to individual respondents’ experiences. This means that respondents will typically not answer all of the questions.
Active Control Properties
The interviews with staff at active control properties will be conducted through a combination of site visits and telephone interviews. We expect to be able to visit many of the active control properties in person, but some properties will have their interviews done by telephone because of budget constraints.
At those active control sites that have service coordinators (estimated to be about 70 percent of the 40 properties), we will interview both the service coordinator and the housing property staff. Otherwise we will interview the housing property staff. These interviews will cover the same broad topics as the interviews conducted at the treatment group sites. They will last about up to two hours, with an additional 30 minutes for scheduling and preparing for the interview. We will use the interviews to collect information on the service coordination and wellness activities provided at the active control sites and any other efforts to support residents’ aging in place.
Trained research staff will conduct the interviews using separate interview guides for each type of respondent. The interview guide for the interviews with service coordinators at active control sites is provided in Appendix F. The interview guide for the interviews with housing property staff at active control sites is provided in Appendix G.
Focus Groups
Focus groups will be an important data source for understanding: the range of resident and caregiver experiences with the IWISH program (and with service coordination at the active control properties), changes in residents’ perceived health and housing satisfaction since the start of the demonstration, and differences between IWISH properties and active control sites.
We will conduct a total of 24 focus groups with four different respondent types. Twenty-one focus groups will be with residents of properties in the demonstration: 12 focus groups with residents enrolled in IWISH, three focus groups with residents of IWISH properties who chose not to enroll in IWISH, and six focus groups with residents of active control properties. In addition, we will conduct three focus groups with the caregivers of enrolled IWISH participants.
All 24 focus groups will be run by a trained facilitator using a focus group guide. We will try limit the focus groups with residents to eight participants in order to accommodate residents who may have auditory or cognitive challenges. Focus groups with caregivers may be somewhat larger, 10 to 12 participants. (For the purposes of estimating respondent burden, we have estimated an average of 10 participants per focus group.) Across all the groups, we will limit the discussion time to 90 minutes, with another 15 minutes for obtaining informed consent at the start of the focus group.
The approach to selecting the properties for focus groups and the focus group participants are discussed in Part B of this ICR.
Focus Groups with IWISH Participants
We will conduct 12 focus groups with IWISH participants. The moderator guide and consent form for the focus groups with IWISH participants are provided in Appendix H.
The focus group with IWISH participants will cover the following topics:
Features of the property that facilitate or challenge aging in place
Reasons for enrolling in IWISH
Perceived impact of IWISH on health and well-being
Perceived impact of IWISH on satisfaction with housing
Types of support provided by RWD and WN
Programming and services accessed at the property and in the community
Outstanding programming, services, and support needs
Focus Groups with Residents of Treatment Properties Not Enrolled in IWISH
The three focus groups with residents who did not enroll in IWISH are designed to provide information on the range of reasons why some participants choose not to enroll. We want to explore whether there may be systematic barriers to participating—such as language barriers—and what preferences guide decisions not to participate.
The moderator guide and consent form for the focus group with residents not enrolled in IWISH are provided in Appendix I.
The focus group with residents who did not enroll in IWISH will cover the following topics:
Features of the property that facilitate or challenge aging in place
Awareness of IWISH
Reasons for not enrolling in IWISH
Interaction with RWD and WN
Other sources of support for health-related needs
Programming and services accessed at the property and in the community
Outstanding programming, services, and support needs
Focus Groups with Caregivers
We will conduct three focus groups with the caregivers of residents enrolled in IWISH. By caregivers we mean informal caregivers—the family members and friends who provide ongoing support to older residents. The reason for conducting caregiver focus groups is that we believe that caregivers may be an important beneficiary of the IWISH program, so we want to understand how caregivers interact with and experience the program.
The moderator guide and consent form for the focus group with caregivers of IWISH participants are provided in Appendix J.
The focus group with caregivers will cover the following topics:
Features of the property that facilitate or challenge aging in place
Awareness of IWISH program
Expectations for IWISH program
Observed changes in health and well-being of care recipients since joining IWISH
Changes in caregiving experience since start of IWISH
Interactions and experiences with RWD and WN
Need for additional supports for caregivers and care recipients
Need for additional programs and services to support aging in place
Focus Groups with Residents of Active Control Properties
We will conduct six focus groups with residents at active control properties. These focus groups are designed to understand how resident experiences at non-IWISH properties with service coordinators differ from resident experiences at IWISH properties.
The moderator guide and consent form for the focus group with residents of active control properties are provided in Appendix K.
The focus group with residents of active control properties will cover the following topics:
Features of the property that facilitate or challenge aging in place
Awareness of the service coordinator
Support provided by the service coordinator
Other sources of support for health-related needs
Programming and services accessed at the property and in the community
Outstanding programming, services, and support needs
Exhibit A-1 summarizes the necessity of information collection across each data collection instrument.
Exhibit A-1: Justification of Data Collection Instruments
Instrument(s) |
Respondents, Content, and Reason for Inclusion |
Initial Questionnaire for Treatment Properties (Appendix A) |
Respondents: Resident Wellness Directors at the 40 treatment properties Content:
Reason: The questionnaire gathers standard information across treatment properties that is not available from administrative data sources. It will be used to prepare for the in-depth interviews and will inform the analysis of program implementation and program fidelity. |
Initial Questionnaire for Active Control Properties with a Service Coordinator (Appendix B, B.1) |
Respondents: Service coordinators at active control properties that have a service coordinator (expected to be about 27 of 40 properties) Content:
Reason: The questionnaire gathers standard information across active control properties that is not available from administrative data sources. It will be used to prepare for the in-depth interviews and will inform the analysis of differences between treatment and active control sites. |
Initial Questionnaire for Active Control Properties without a Service Coordinator (Appendix B, B.2) |
Respondents: Housing property staff at active control properties that do not have a service coordinator (expected to be about 13 of 40 properties) Content:
Reason: The questionnaire gathers standard information across active control properties that is not available from administrative data sources. It will be used to prepare for the in-depth interviews and will inform the analysis of differences between treatment and active control sites. |
Interview Guide for Resident Wellness Directors at Treatment Properties (Appendix C) |
Respondents: Resident Wellness Directors at the 40 treatment properties Content:
Reason: The RWD interviews will collect key information on how IWISH has been implemented at each site and the factors that influence implementation effectiveness and fidelity to the model. The interviews will gather information on how the property staff work together to engage residents, conduct person-centered interviews and assessments, use the PHL system, develop programming and build partnerships. The interviews will also explore the background and training of the RWD, which could affect implementation, and RWD opinions on the benefits and strengths and weaknesses of the model. |
Interview Guide for Wellness Nurses at Treatment Properties (Appendix D) |
Respondents: Wellness Nurses at the 40 treatment properties Content:
Reason: The WN interviews will collect key information on how IWISH has been implemented at each site and the factors that influence implementation effectiveness and fidelity to the model. The interviews will gather information on how the property staff work together to engage residents, conduct person-centered interviews and assessments, use the PHL system, develop programming and build partnerships. The interviews will also explore the background and training of the WN, which could affect implementation, and WN opinions on the benefits and strengths and weaknesses of the model. |
Interview Guide for Housing Property Staff at Treatment Properties (Appendix E) |
Respondents: Housing property staff at the 40 treatment properties Content:
Reason: The interviews with housing property staff provide insight into the experience of property managers as partners in the demonstration. Property managers play a distinct role from RWDs and WNS with respect to residents, but partnership between the property management staff and wellness staff is essential to IWISH’s success. IWISH is expected to benefit property management by providing additional support to help residents remain stably housed and to address mental health issues that might affect their tenancy. |
Interview Guide for Service Coordinators at Active Control Properties (Appendix F) |
Respondents: Service coordinators at the subset of active control properties with service coordinators (expected to be about 27 of 40 properties) Content:
Reason: The interviews with service coordinators at active control properties will provide key information on the activities and roles of service coordinators at properties with HUD service coordinator funds. To the extent possible, the interviews will seek to collect information comparable to that collected through the RWD and WN information on the support that the service coordinator provides to their residents. The interviews will be used to compare IWISH implementation to typical service coordination at active control properties, which will inform interpretation of the impact study results. |
Interview Guide for Housing Property Staff at Active Control Properties (Appendix G) |
Respondents: Housing property staff at the 40 active control properties Content:
Reason: The interviews with housing property staff provide insight into the experience of managing HUD multifamily properties for older adults and manager opinions on aging in place as a goal and what it takes to help residents age in place successfully. Some of the properties will have service coordinators and others will not. For properties with service coordinators we will learn about the relationship between service coordinators and managers and management’s views of the benefits of service coordination for residents and management. For properties without service coordinators we will learn about how managers support residents in the absence of additional resources. These interviews will be used to compare IWISH implementation to typical service coordination at active control properties, which will inform interpretation of the impact study results. |
Moderator Guide for Focus Groups with IWISH Participants (Appendix H) |
Respondents: IWISH participants Content:
Reason: The 12 focus groups with IWISH participants will be a key source of information for answering the study’s research question about changes in residents’ perceptions of their health, well-being, and satisfaction with housing and services as a result of participating in IWISH. The focus groups will also contribute to the analysis of the fidelity of the implementation of IWISH at the treatment sites by providing residents’ perspectives on the support provided by the RWD and WN and the programming and services made available to residents on and off site. |
Moderator Guide for Focus Groups with IWISH Non-enrollees (Appendix I) |
Respondents: Residents of treatment properties who choose not to enroll in IWISH Content:
Reason: The three focus groups with residents who did not enroll in IWISH are designed to provide information on the range of reasons why some participants choose not to enroll. This is important if we uncover systematic barriers to participating in IWISH that could affect its impact on particular subgroups of residents. |
Moderator Guide for Focus Groups with Caregivers of IWISH Participants (Appendix J) |
Respondents: Caregivers of IWISH participants Content:
Reason: The three focus groups with caregivers will provide additional information on the benefits of IWISH. We believe that caregivers may be an important beneficiary of the IWISH program, so we want to understand how caregivers interact with and experience the program. |
Moderator Guide for Focus Groups with Residents of Active Control Properties (Appendix K) |
Respondents: Residents of treatment properties who choose not to enroll in IWISH Content:
Reason: The six focus groups with residents of active control properties are designed to understand how resident experiences at non-IWISH properties with service coordinators differ from resident experiences at IWISH properties. The information from these focus groups will be used to compare IWISH implementation to typical service coordination, which will inform interpretation of the impact study results. |
3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology.
The research team will make every effort to reduce the burden on the respondents, by making effective use of technology to streamline data collection procedures.
Questionnaires will be conducted over the telephone and entered directly into the online survey platform Confirmit.
Interviews will be conducted either in person or over the phone. The interviewers will record responses directly into Word documents on a laptop. Use of the laptops will allow interviewers to quickly record data and continue with the interview without extended pauses or delays. Interview responses will be directly uploaded to NVivo, a qualitative data analysis software for coding and analysis.
Focus groups will be conducted in person. As with interviews, focus group note-takers will record responses directly onto a laptop before uploading to NVivo. Focus groups will also be audio recorded, if all participants consent.
4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.
HUD is not aware of any other studies for which this study represents a duplicate research effort. The IWISH model has not been implemented before, and this is the only evaluation of it to date.
Throughout the evaluation, we will obtain extracts of the Population Health Logistics (PHL) data system that all IWISH properties will use to collect and store health and service information on IWISH participants. The PHL data collection is covered under ICR 201702-2528-001 and the consent process for IWISH participants covers the transfer of PHL data to the evaluation team. We will use the PHL data wherever possible for data on program implementation. We will not duplicate the information already being collected through PHL.
5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I) describe any methods used to minimize burden.
We expect only minimal (if any) impact of this data collection on small business entities. It is possible that some property owners will participate in the interviews with housing property staff and that some of these owners may be small businesses. However, all of the property owners have entered into a cooperative agreement with HUD to participate in the evaluation.
6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.
Each data collection activity under this ICR will be conducted only once and under specific contract guidelines. Without this data collection effort, HUD will be unable to evaluate the implementation of the Supportive Services Demonstration. The qualitative data collection covered by this ICR is essential for providing context for the results of the quantitative impact analysis, particularly for understanding the difference between the service coordination and wellness services provided through IWISH and typical service coordination at other HUD multifamily properties serving the elderly. Without data collected from individuals involved in the provision of IWISH services, the study will have little insight into the process of IWISH implementation and fidelity to the IWISH model. Without data collected from residents, the study will have little insight into how residents experience the IWISH model and how that compares to their experience of typical service coordination.
Explain any special circumstances that would cause an information collection to be conducted in a manner:
The proposed data collection activities are consistent with the guidelines set forth in 5 CFR 1320.6 (Controlling Paperwork Burden on the Public, General Information Collection Guidelines). There are no circumstances that require deviation from these guidelines.
Requiring respondents to report information to the agency more often than quarterly;
Most respondents will only be involved in a one-time data collection. Some respondents, such as RWDs, service coordinators, and property manager staff may be involved in more than one data collection effort over the course of the study, but no more than one per year.
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Respondents are not required to prepare a written response as part of this data collection effort.
Requiring respondents to submit more than an original and two copies of any document;
Respondents are not required to submit more than an original and two copies of any documents as part of this data collection effort.
Requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than three years;
Respondents are not required to retain records as part of this data collection effort for more than three years.
In connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
This data collection does not involve a statistical survey. Respondents will be selected based on a purposive sample, and the data collection instruments will gather qualitative information to inform HUD on the experiences of property managers, service coordination and wellness staff, residents, and the caregivers of residents in the Supportive Services Demonstration.
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
This data collection does not involve the use of any statistical data classification that has not been reviewed and approved by OMB.
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use;
This data collection does not involve the use of a pledge of confidentiality that would deviate from statute or regulation, be inconsistent with disclosure and data security policies, or be considered as impeding the sharing of data as appropriate.
Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.
This data collection does not require respondents to submit proprietary trade secrets or confidential information.
8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB.
In accordance with 5 CFR 1320.8 (Paperwork Reduction Act of 1995), a Notice of Proposed Information Collection for publication in the Federal register has been prepared to announce the agency’s intention to request an OMB review of data collection activities for the Evaluation of the Supportive Services Demonstration. HUD published a 60-Day Notice of Proposed Information Collection in the Federal Register on February 28, 2018. The Docket No. is Docket No. 83 FR 8691 and the notice appeared on pages 8691-8693. The notice provided a 60-day period for public comments, and comments were due April 30, 2018. No public comments were received. A copy of the notice is included with this Information Collection Request (ICR) in Appendix L.
The Evaluation of the Supportive Services Demonstration was developed and is being implemented with the assistance of Abt Associates Inc., the study’s contractor. Key members of the Abt team include Project Director Jennifer Turnham; Co-Principal Investigators Gretchen Locke and Sara Galantowicz; Co-Project Quality Advisors Terry Moore and Dr. Jill Khadduri; and Co-Technical Advisors Dr. Austin Nichols and Dr. Jennifer Riggs. Staff from HUD, the Office of the Assistant Secretary for Planning and Evaluation (ASPE), and the Centers for Medicare and Medicaid Services (CMS) at the Department of Health and Human Services have collaborated on the design of the evaluation with the research team throughout all phases of the study to date.
Abt Associates and HUD have established an Expert Panel to review the evaluation design, progress, and findings, to maximize the rigor of the evaluation and its value to multiple stakeholders. The members of the expert panel are:
Mara Blitzer, Director of Housing Development, Mayor's Office of Housing and Community Development, City and County of San Francisco
Melanie Brown, Technical Director, Division of Community Systems Transformation, Disabled and Elderly Health Programs Group, CMCS, CMS
Bruce Chernof, MD, FACP, President and CEO, SCAN foundation
Partha Deb, PhD, Professor of Economics, Hunter College, City University of New York
Tim Engelhardt, Director, Federal Coordinated Health Care Office, CMS
Kosuke Imai, PhD, Professor of Politics and Director of Program in Statistics and Machine Learning, Princeton University
Cindy Mann, Partner, Mannatt Health
Sandy Markwood, CEO, National Association of Area Agencies on Aging
Janice C. Monks, Founder, President/CEO, American Association of Service Coordinators
Emily Rosenoff, Acting Director, Division of Long-Term Care Policy, Office of Disability, Aging, and Long-Term Care Policy, ASPE
Lori Simon Rusinowitz, University of Maryland
Kamillah Wood, MD, MPH, FAAP, Senior Vice President of Health and Housing, Stewards of Affordable Housing for the Future, SAHF
9. Explain any decision to provide any payment or gift to respondents, other than renumeration of contractors or grantees.
As an incentive, residents and caregivers who participate in focus groups will receive a $40 gift card. Three factors helped to determine the incentive amounts for the focus group: 1) participant burden; 2) costs associated with the amount of time that the focus group participant will commit to focus group participation; and 3) other studies of comparable populations and burden.
The proposed resident and caregiver focus groups will last from 60 to 90 minutes, with minimal travel costs since the focus groups will be held at the properties (and the caregivers can be assumed to be there frequently) and no other costs to participants. The $40 incentive is also comparable to incentives provided in OMB-approved studies with similar levels of burden. In previous PD&R studies OMB has approved the following:
$25 for Jobs Plus Pilot Program, 60-minute focus groups with public housing residents (OMB control no. 2528-0310)
$60 for Pre-Purchase Homeownership Counseling Demonstration, 120-minute focus groups with first-time homebuyers (OMB control no. 2528-0306)
$50 for ConnectHome Use and Barriers, 90-minute focus groups with public housing residents (OMB control no. 2528-0311)
10. Describe any assurance of confidentiality provided to respondents and the basis for assurance in statute, regulation or agency policy.
HUD has entered into a contract with an independent research team, Abt Associates Inc., to conduct this research effort. HUD and Abt Associates will make every effort to maintain the privacy of respondents, to the extent permitted by law. The subjects of this information collection and the nature of the information to be collected require strict confidentiality procedures. The information requested under this collection is protected and held confidential in accordance with 42 U.S.C. 1306, 20 CFR 401 and 402, 5 U.S.C.552 (Freedom of Information Act), 5 U.S.C. 552a (Privacy Act of 1974) and OMB Circular No. A-130. A Privacy Threshold Assessment (PTA) for this study was approved by HUD on March 18, 2018 and a Privacy and Civil Liberties Impact Analysis (PCLIA) was approved by HUD on May 10, 2018. All research staff working on the project have been trained to protect private information and the study has a detailed Data Security Plan governing the storage and use of the data collected through the study. Additionally, individuals will not be cited as sources of information in prepared reports.
All respondents included in the study will be informed that information they provide will be used only for the purpose of this research. For the initial questionnaires and interviews, we will ask all respondents for verbal consent before beginning the interview. For the focus groups, respondents will be asked for written consent. Hard copy written consent forms and any papers that contain participant names or other identifying information will be kept in locked areas and any computer documents containing identifying information will be protected with a password. During the interviews, Abt interviewers will record staff’s position, title, and site location, and enter interview notes on an encrypted laptop. After the site visit is completed, interviewers will transfer the interview notes to Abt Associates’ common drive, to a folder with access restricted only to staff associated with the project.
11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.
The interviews with RWDs, WNs, service coordinators, and housing property staff do not contain any sensitive questions.
Some questions in the resident focus groups may be sensitive for some respondents. The resident focus groups will be conducted with low-income, elderly individuals. Questions related to their experiences living in HUD-assisted housing, working with wellness and service coordination staff, and receiving or not receiving needed services may be sensitive for some people. As part of the consent process, all focus group participants will be informed that their answers will be private and only be used for the purposes of this study, that they may refuse to answer any questions, and that results will only be reported in the aggregate. The study team will screen all respondents for the cognitive ability to participate in the group and obtain participant consent (and the consent of legally-authorized representatives, if applicable) prior to beginning the focus group.
12. Provide estimates of the hour burden of the collection of information.
Initial Questionnaire
The estimated average burden for the initial questionnaires is 1.5 hours per person per questionnaire. The questionnaire will take up to 60 minutes to complete, with an additional 30 minutes for scheduling and preparation. There will be one to two respondents from each property. The estimated number of respondents for the initial questionnaire is 120 and the estimated burden is 180 hours. Note that there are three variants on the questionnaire to reflect the different possible respondents, but each respondent will complete one questionnaire only and the burden is the same for all the questionnaires. The questionnaires are presented in Appendix A and B.
Interviews
The estimated average burden for the interviews is 2.5 hours. The interviews will last up to two hours, with an additional 30 minutes for scheduling and preparation. There will be one to four respondents per property. The estimated number of respondents for the interviews questionnaire is 220 and the estimated burden is 550 hours. Note that there are five variants on the interview guide to reflect the different possible respondents, but each respondent will complete one interview only and the burden is the same for all the interview guides. The interview guides are presented in Appendices C through G.
Focus Groups with Residents
The estimated average burden for each resident focus group is 1.75 hours. The focus group discussion will last up to 90 minutes, with an additional 15 minutes at the start for participants to complete the consent process and orient themselves to the group. We estimate an average of 10 participants per focus group and 21 focus groups, for a total of 210 respondents and 367.5 burden hours. Note that there are three variants on the focus group guide to reflect the different types of participant, but each respondent will participate in only one focus group and the burden is the same for all the focus group guides. The resident focus group guides are presented in Appendices H, I, and K.
Focus Groups with Caregivers
The estimated average burden for the caregiver focus group is 1.75 hours. The focus group discussion will last up to 90 minutes, with an additional 15 minutes at the start for participants to complete the consent process and orient themselves to the group. We estimate an average of 10 participants per focus group and three focus groups, for a total of 30 respondents and 52.5 burden hours. The caregiver focus group guide is presented in Appendix J.
Exhibit A-2 provides the total estimated hour and cost burden of the information collection.
Exhibit A-2: Estimated Hour and Cost Burden of Information Collection
Information Collection |
Number of Respondents |
Frequency of Response |
ResponsesPer Annum |
Burden Hour Per Response |
Annual Burden Hours |
Hourly Cost Per Response |
Annual Cost
|
Initial questionnaire |
120.00 |
1 |
120.00 |
1.50 |
180.00 |
$34.04 |
$6,127.20 |
Interviews |
220.00 |
1 |
220.00 |
2.50 |
550.00 |
$34.04 |
$18,722.00 |
Focus groups with residents |
210.00 |
1 |
210.00 |
1.75 |
367.50 |
$7.90 |
$2,903.25 |
Focus groups with caregivers |
30.00 |
1 |
30.00 |
1.75 |
52.50 |
$27.70 |
$1,454.25 |
Total |
580.00 |
|
580.00 |
|
1,150.00 |
|
$29,206.70 |
The total estimated annual cost for this information collection is $29,206.70.
To estimate the cost per hour for the questionnaire and interview respondents, we use the most recent (May 2016) Bureau of Labor Statistics, Occupational Employment Statistics median hourly wage for selected occupations classified by Standard Occupational Classification (SOC) codes and added 31.7 percent to account for benefits costs. (According to the Bureau of Labor Statistics’ Employer Costs for Employee Compensation data from September 2017, benefit costs averaged 31.7 percent of employer costs for employee compensation across all job categories.)
The hourly cost for the initial questionnaire is $34.04. To estimate hourly wage rates for the questionnaire respondents, we used the occupation code Healthcare Social Workers (21-1022) with a median hourly wage of $25.85 and an estimated cost with benefits of $34.04.
The hourly cost for the interviews is $34.04. To estimate hourly wage rates for the interview respondents, we used the occupation code Healthcare Social Workers (21-1022) with a median hourly wage of $25.85 and an estimated cost with benefits of $34.04.
The hourly cost for the focus groups with residents is $7.90. Most of the properties in the SSD are funded through HUD’s Supportive Housing for the Elderly (Section 202) program. According to HUD’s Picture of Subsidized Households for 2016 (https://www.huduser.gov/portal/datasets/assthsg.html), the average household income for Section 202 residents is $13,311. Some 98 percent of households have something other than wages or welfare benefits as their major source of income, in most cases Social Security benefits. To estimate the hourly cost for the residents participating in focus groups, we translated the average monthly Social Security benefit for retired workers, which in 2017 was $1,369 (https://www.ssa.gov/news/press/factsheets/basicfact-alt.pdf) into an hourly rate of $7.90 per hour (by multiplying by 12 months and dividing by 2,080 hours).
The hourly cost for the focus groups with caregivers is $27.70. To estimate hourly costs for caregivers, we used the median annual household income from the 2016 American Community Survey, $57,617, and divided it by 2,080 hours to arrive at an hourly rate of $27.70.
13. Provide an estimate of the total annual cost burden to respondents or recordkeepers resulting from the collection of information (do not include the cost of any hour burden shown in Items 12 and 14).
This data collection effort involves no recordkeeping or reporting costs for respondents other than the time burden to respond to questions on the data collection instruments as described in item 12 above. There is no known cost burden to the respondents.
14. Provide estimates of annualized cost to the Federal government.
The current effort is being carried out under a HUD Contract with Abt Associates. The total amount of this grant, spent over a 48-month period, is $2,691,632.77. This data collection activity will cost approximately $600,000, including approximately $9,600 in incentive payments to the residents and caregivers participating in the focus groups. Exhibit A-3 summarizes the cost breakdown.
Exhibit A-3: Estimated Cost to the Federal Government
Activity |
Estimated Cost to Federal Government |
Evaluation design, analysis, and reporting activities (excluding data collection) |
$2,091,632.77 |
Data collection activities (excluding incentive payments) |
$590,400.00 |
Incentive payments |
$9,600.00 |
Total |
$2,691,632.77 |
15. Explain the reasons for any program changes or adjustments reported in Items 13 and 14 of the OMB Form 83-I.
This submission to OMB is an initial submission and does not involve any program changes or adjustments.
16. For collection of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.
The data collected for the Evaluation of the Supportive Services Demonstration will be analyzed, tabulated, and reported to HUD by the evaluation contractor, Abt Associates.
Exhibit A-4 presents an overview of the data collection and analysis schedule.
Exhibit A-4: Project Schedule
Timeframe |
Activity |
Notes |
October 2018 – December 2018 |
Initial questionnaires fielded |
Data collection will not start until OMB approval is received. |
October 2018 – January 2019 |
First round of administrative data collection |
HUD data, PHL data, Medicare data, and public use data. |
January 2019 – May 2019 |
Analysis of questionnaire data and PHL data |
Data collected through the initial questionnaire will be analyzed and tabulated for the interim report and used to inform the site visits and interviews. |
April 2019 – December 2019 |
Site visits, interviews, and focus groups |
|
June – September 2019 |
Interim report |
Descriptive analysis of administrative data and data collected through initial questionnaire and early site visits. |
January 2020 |
Supplemental ICR for final questionnaire and partner interviews |
|
January 2020 – May 2020 |
Analysis of interview and focus group data |
|
May 2020 – July 2020 |
Final questionnaires and partner interviews |
Covered under supplemental ICR. |
October 2020 – January 2021 |
Second round of administrative data collection |
HUD data, PHL data, Medicare data, Medicaid data, and public use data. |
January 2021 – June 2021 |
Impact analysis and final process study analysis |
|
July 2021 – March 2022 |
Comprehensive report |
All process study and impact analyses. |
The initial questionnaire will be administered from October 2018 through December 2018. Site visits, including interviews and focus groups, will be conducted from April 2019 through December 2019, with telephone interviews as necessary throughout the same period. Analysis in preparation for the interim report will take place from January 2019 through May 2019. The interim report is due to HUD in June 2019. The interim report will describe the characteristics of the treatment and active control properties and summarize the findings from the initial questionnaires. To the extent possible, the interim report will also provide early findings from the first set of site visits, interviews, and focus groups.
We plan to submit a second ICR for follow-up data collection to support the comprehensive report. This follow-up data collection is expected to include a final questionnaire and a set of interviews with service provider partners, both scheduled to take place in summer 2020. Analysis for the final comprehensive report will take place from January 2020 through June 2021. The draft comprehensive report is due to HUD in July 2021.The comprehensive report will present the complete results of the data collection and analysis. It will use all the data sources described in this submission, will address all of the study’s research questions, and will discuss the policy implications of the study’s findings.
17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.
All data collection instruments will prominently display the expiration date for OMB approval.
18. Explain each exception to the certification statement identified in item 19.
This submission describing data collection requests no exceptions to the Certificate for Paperwork Reduction Act (5 CFR 1320.9).
Page
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Jennifer Turnham |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |