SED Supporting Statement - Part B (clean version) 07-28-2017

Supported Employment Demonstration (SED) Project

Supporting Material for OMB Clearance

Part B. Statistical Methods

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The Supported Employment Demonstration (SED) study population consists of individuals aged 18 to 50 who apply for disability benefits alleging a mental illness and the initial decision is a denial of benefits in the past 60 days. Individuals eligible for the study must live within the area of one of 30 community mental health centers (CMHCs), 20 urban and 10 rural, distributed across the United States that will serve as demonstration sites. We will screen potentially eligible individuals for competency (ability to give consent), and confirm their allegation of a mental health impairment using the Composite International Diagnostic Interview (CIDI). We will exclude from study participation any individuals who are unable to provide informed consent; or who reside in a nursing home or other custodial institution. The SED project will enroll 1,000 participants in each of the three study arms for a total of 3,000 participants: 40 participants in each of three study arms for the 20 urban sites equaling an n of 2,400 urban site participants, and 20 participants in each of three arms for the 10 rural sites equaling an n of 600 rural site participants. SSA will provide Westat with an initial list of denied applicants (within the past 60 days prior to the start of the project) with an alleged impairment of mental illness within a defined area for each site. Subsequently, on a monthly basis, SSA will provide Westat with a list of denied applicants within these areas. Upon receipt of the files, Westat will create random Identification data (IDs) for each potential participant; sanitize the data (i.e., remove Social Security Numbers); and establish release groups from which recruitment will take place. The release groups will include up to 25 potential participants based on their distance from the demonstration site. Westat will then load the release groups into the study’s Management Information System (MIS), and assign Research Assistants off site in the local community (e.g., field interviewers) exclusive access for recruitment. As contact information for each denied applicant is loaded into the study’s management information system MIS, we will create another special ID that ties the contact information ID with the subsequent enrollment decision, other study data, and the individual’s SSN.

2. Procedures for the Collection of Information

Recruitment and Screening

Westat will first send potential participants a study invitation package (see Attachment A. Introductory Letter and Attachment B. Study Brochure). Several days following this mailing, the Research Assistants contact the potential participants by phone (see Attachment C. Sample Script for Initial Phone Call); if unable to reach them by phone, the Research Assistants will make an in-person visit. As part of this initial contact, the Research Assistants will determine interest in work by means of a single question, “Are you interested in getting a job?” The Research Assistants will then invite potential participants who express a desire to work to a recruitment information meeting. The recruitment information meeting will serve to orient potential participants about the demonstration, giving them the opportunity to weigh the merits of the demonstration while allowing Research Assistants to dispel any misconceptions and provide detailed information on the demonstration’s advantages and risks (see Attachment D. Overview and FAQs). The recruitment meetings will take place in a group or individual format. The Research Assistants will then ask those interested in joining the demonstration to give informed consent to participate in a screening for eligibility to confirm competency and allegation of a mental health impairment using the Composite International Diagnostic Interview (CIDI) (see Attachment E. Screening Consent, Attachment F. Competency and CIDI Screener, and Attachment G. Ineligibility Letter). Potential participants will receive $45 for completing the screening process, which will take approximately 40 minutes. We will invite

those who screen positive for a mental health impairment to give informed consent to enroll in the study (see Attachment H. Study Consent).

Random Assignment to Study Arms

Once enrolled, we will randomly assign each participant to one of three study arms: (1) Full-Service Treatment (2) Basic-Service Treatment; or (3) Usual Services. (see Attachment I. Randomization Letters and Attachment J. Reloadable Card FAQs), our procedures for which we described in detail in the Supporting Statement Part A, #2. Here we also discuss the costs associated with the random assignments:

• Full-Service Treatment (n=1,000). The plan for this model includes out‑of‑pocket expenses associated with prescription behavioral health medications, with an average allotment of $1,166 per participant over the three years of their participation in the study. Individual Placement and Support (IPS) supported employment also includes a modest allotment of funds for essential work-related expenses. Integrated behavioral health care also requires financial support for services and treatment not covered by the participant’s health insurance. The allotments for these additional expenses include $130 for essential work-related expenses and $1,034 for behavioral health expenses per participant over the three years of study participation.

• Basic-Service Treatment (n=1,000). The plan for this model includes allotments of $1,034 per participant for expenses associated with behavioral treatment, and $130 per participant for essential work-related expenses over the three years of study participation.

• Usual Services (n=1,000). We do not include any allotments for this model.

Evaluation Data Collection

Data collection for the evaluation of the SED will consist of the following activities: baseline in-person participant interviews; quarterly participant telephone interviews; receipt of SSA administrative record data; and collection of site-level program data. Evaluation team members will also conduct site visits involving: pre-visit environmental scans; independent fidelity assessments; key informant interviews; focus groups; and ethnographic data collection.

• Baseline Participant Interviews

Research Assistants, hired by Westat and located off site in the local community, will conduct a baseline, in-person interview with each participant enrolled in the Supported Employment Demonstration via a computer-assisted personal interview (CAPI) (see Attachment K. Baseline Interview). Participants will receive $50 for completing the baseline interview, which will take approximately 45 minutes. The baseline interview will include questions on the following topics: work history; job status; health status and quality of life; demographics and income; functional limitations; health insurance coverage; and health care utilization (doctor visits, hospitalizations, etc.). During the baseline interview, the Research Assistant will also ask participants to provide contact information for two additional individuals in the event that the participant cannot be located. At the end of the baseline interview, the Research Assistant will provide the participant with an assignment to one of the three study arms, as well as written material and guidance about moving forward in the study (see Attachment I. Randomization Letters and Attachment J. Reloadable Card FAQs).

• Quarterly Participant Interviews

Westat Telephone Research Center (TRC) interviewers will administer blind quarterly follow-up telephone interviews to participants via a computer-assisted telephone interview (CATI) (see Attachment L. Quarterly Interview). The quarterly interview will assess employment outcomes and utilization of health care and other related services. In addition, the quarterly interviews administered at the annual follow-up time periods (Quarters 4, 8, and 12), will assess health status; functioning; symptoms; medication use; presence of two or more chronic illnesses; quality of life; and recidivism. Participants will receive $25 for completing each quarterly interview (at Quarters 1, 2, 3, 5, 6, 7, 9, 10, and 11), which will take approximately 20 minutes, and $40 for each annual interview (at Quarters 4, 8, and 12), which will take an average of 30 minutes.

We will continue conducting quarterly follow-up interviews with all participants, even if they stop receiving treatment services before the end of the treatment period. Only in cases where participants formally end their relationship with the study and explicitly request no further contact will we no longer pursue quarterly follow-up interviews. We will employ several strategies to minimize attrition and loss to follow-up for the quarterly interviews: First, in the baseline and quarterly interviews, the interviewer asks the participant if he or she plans to move; reiterates the

toll-free participant number; and offers a $5 cash incentive if the participant calls to report the new address. Second, we will flag participants who initially refuse to continue their participation in a quarterly interview for further follow-up by a CATI interviewer experienced in refusal conversion. Third, for participants we cannot locate, TRC interviewers experienced in tracing will use contact the information the participant provided in the baseline interview to obtain updated contact information. If these efforts prove unsuccessful, we may engage local Westat field staff to conduct in-person tracing efforts, starting with visiting the participant’s last known address.

SSA Administrative Data

SSA will provide Westat access to administrative data on the full participant sample (N=3,000), even for those participants who exit the study or do not complete the follow-up interviews. These data will include items such as: application for disability benefits; disability decisions; award amounts; and earnings. Westat will use the administrative data SSA provides to measure benefit-related outcomes.

Site-Level Program Data

We will collect program data at the site level via the MIS and site visits. These data will include items related to the provision of services to participants. Westat will use this information largely in the participation analyses and process evaluation.

Site Visits

Evaluation of the SED will include annual site visits to each of the 30 study sites, for 120, 5-day long site visits, involving the following activities: pre-visit environmental scan; independent fidelity assessments of IPS-supported employment services and other treatment components; key informant interviews; focus groups with treatment participants; and person-centered interviews for the ethnographic data collection. Table 1 displays the data collection activities; we will carry out during the site visits, and the total number of respondents for each activity.

Table 1. Target Number of Respondents for Site Visit Data Collection Activities

Contract Year	Fidelity Assessment Participant Interviews	Fidelity Assessment Family Member Interviews	Key Informant Interviews	Participant Focus Groups	Ethnographic Observations	Ethnographic Person-Centered Interviews
1
2	180	90	120	600	60	180
3	180	90	120		60	180
4	180	90	120	600	60	180
5	180	90	120		60	180
6
Total Target Number of Respondents	720	360	480	1200	240	720

Pre-Visit Environmental Scan. In preparation for each site visit, we will scan the local environment to understand the local context in which we embed SED services. For example, we will access unemployment rates and information on general types of work available using Bureau of Labor statistics and County Business Patterns. We will also explore available community, non-profit, and charitable services through online and phone searches as well as community resources manuals. Information acquired through the environmental scans will provide a framework of available services to individuals with employment and mental health needs.

Independent Fidelity Assessments. In contract Years 2 through 5, we will conduct annual, independent fidelity assessments of IPS supported employment services and other treatment components, i.e., the Nurse Care Coordinator role and provision of Systematic Medication Management, for the purpose of assessing treatment fidelity (see Attachment M7. IPS Fidelity Assessment Guide). There will be a total of 120 independent fidelity assessments, one for each center in each implementation year. To reduce respondent burden and efficiency, we will schedule these independent fidelity assessments in conjunction with those conducted by state Mental Health/Vocational Rehabilitation staff as a part of the regular monitoring and quality improvement fidelity reviews conducted for implementation. However, we will score and interpret the findings from our fidelity assessments independently.

The treatment team at each site will select SED participants and family members involved as respondents in the Independent Fidelity Assessment. The treatment team will select potential respondents to represent the diversity of SED participants by gender, race, ethnicity, age, employment status, and employment type. We will mail or hand a letter to respondents (see Attachment M1. IPS Fidelity Assessment Participant Invitation Letter and Attachment M4. IPS Fidelity Assessment Family Member Invitation Letter) describing the IPS interview with the proposed date and time. If we mail the letter, a member of the treatment team will follow up with a phone call (see Attachment M2. IPS Fidelity Assessment Participant Phone Call Sample Script and Attachment M5. IPS Fidelity Assessment Family Member Phone Call Sample Script) to the respondent to confirm interest in attending (see Attachment M3, IPS Fidelity Assessment Participant Consent Script and Attachment M6). IPS Fidelity Assessment Family Member Consent Script provides the scripts for use in obtaining verbal informed consent for the IPS Fidelity Assessments conducted with participants and family members.

Key Informant Interviews. In contract Years 2 through 5, at each of the 30 sites we will conduct semi-structured key informant interviews with the IPS specialist, the nurse care coordinator, the case manager, and the director of the facility (see Attachment N. Key Informant Interview Administrator Initial; Attachment O. Key Informant Interview Administrator Follow Up; Attachment P. Key Informant Interview Clinician Initial; and Attachment Q. Key Informant Interview Clinician Follow Up). We expect the key informant interviews to take up to 60 minutes. Interviews will cover topics such as: agency funding streams; challenges and successes of demonstration implementation; sustainability; participant outcomes; and recommendations. In Years 2 and 3, evaluation team members will conduct these interviews in person; in Years 4 and 5 they will conduct these interviews by telephone. We will interview up to 4 key informants at each site for a total of up to 240 in-person interviews in Years 2 and 3 and 240 telephone interviews in Years 4 and 5, or 480 key informant interviews combined.

Focus Groups. In contract Years 2 and 4, at each of the 30 sites, we will conduct 2 focus groups with treatment participants: one with Full-Service Treatment participants and one with Basic-Service Treatment participants (see Attachment R3. Participant Focus Group Guide). The IPS will select potential participants to represent a diversity of participants by gender, race, ethnicity, age, employment status, and employment type. We will mail or hand a letter to participants (see Attachment R1. Participant Focus Group Invitation Letter) describing the focus group with the proposed date and time. If we mail a letter, the IPS specialist or Site Visitor will follow up with a phone call (see Attachment R2. Participant Focus Group Phone Call Sample Script) to the participant to confirm interest in attending. We expect focus group topics to take up to 60 minutes and will cover: perceived quality of services delivered; preferences among types of employment and mental health services; barriers to employment; perceived availability of community employment resources; and experiences of supported employment. We aim to include at least 6 participants and no more than 10 participants in each focus group, for a total of up to 1200 participants (20 participants x 30 sites x 2) in the focus groups combined. Each participant will receive $40 for attending the focus group.

Ethnographic Data Collection. In contract Years 2 through 5, we will conduct ethnographic data collection consisting of observations in the natural environment, and person-centered interviews with participants and non-participants (Attachment S6. Ethnographic Observation Template). With participants’ permission, evaluation team members will observe participants and non-participants in natural environments such as: workplaces; family homes; churches; community settings; treatment settings; leisure activities; and other locations important for understanding the layered contexts in which participants sought disability benefits and now seek employment. The observations will contribute to a holistic understanding of the constraints, opportunities, preferences, and assumptions that affect employment and the mental health among those applying for SSA disability benefits. We will not determine the specific locations for observation until the site visit team systematically gathers information about the environment of the treatment site. We may choose some locations for ethnographic observation the visit begins, we may not discover others until after the site visit team begins the visit. We anticipate that discussions with clinicians and other workers at the treatment site, local community leaders, and study participants will yield appropriate locations for ethnographic observations. Once we have chosen a location for observations, site visit team members will mail invitation letters to location’s gatekeepers and, or to study participants who agreed to be observed (see Attachment S1. Ethnographic Observation Gatekeeper Invitation Letter, and Attachment S3. Ethnographic Observation Participant Invitation Letter). The Site Visitors will follow up with phone calls (see Attachment S2. Ethnographic Observation Gatekeeper Phone Call Sample Script and Attachment S4. Ethnographic Observation Participant Phone Call Sample Script) to gatekeepers and participants to confirm interest in participating. Once at the location for observation, we will ask any individuals whom the Site Visitors may observe to provide verbal consent (see Attachment S5. Ethnographic Observation Ad Hoc Consent Script). For example, observing a local job fair would involve seeking consent from the organizers of the fair (the gatekeepers) and any individuals with whom the site visitors may interact. Observing the daily life of a study participant will require the consent of the participants and any individuals with whom they may interact. Evaluation team members will take field notes to systematically document their observations. We will observe up to 8 individuals at each of the 30 sites, for a total of 240 individuals observed in Years 2 through 5 combined.

As part of the ethnographic data collection, evaluation team members will also conduct person-centered interviews to better characterize participants and non‑participants of the Supported Employment Demonstration (see Attachment T5. Person-Centered Interview Initial and Attachment U. Person-Centered Interview Follow Up). As informants, we will ask interviewees to narrate their work and illness histories to describe the chain of events and decisions that led to making an SSA application. As respondents, we will ask interviews to explain how these experiences motivated them to apply for SSA disability benefits. At each site, the Research Assistant will select interviewees from participants in each demonstration arm (Full‑Service, Basic-Service, and Usual Services) and from non-participants. Non‑participant interviewees will include: SSA disability applicants recently denied benefits who chose not to participate in the study; individuals who enrolled in the study, but not engaged with treatment services; and those who enrolled in the study but withdrew from treatment services. To the extent possible, interviewees will represent diversity of age, gender, race, ethnicity, and employment status. We will mail or hand a letter to potential interviewees (see Attachment T1. Participant Person‑Centered Interview Invitation Letter and Attachment T3. Non-Participant Person-Centered Interview Invitation Letter) describing the interview with proposed dates and times. If we mail the letter, the Site Visitor or the Research Assistant will follow up with a phone call (see Attachment T2. Participant Person-Centered Interview Phone Call Sample Script and Attachment T4. Non-Participant Person‑Centered Interview Phone Call Sample Script) to the potential interviewee to confirm his or her interest in attending. We will interview up to 3 participants (1 from each study arm) and 3 non-participants at each of the 30 sites in contract Years 2 through 5, for a total of up to 720 person-centered interviews. Each individual will receive $40 for participating in the person-centered interview, which will take approximately 60 minutes.

3. Methods to Maximize Response Rates and Deal with Nonresponse

There are two types of non-response: (1) non-participation in the study and

(2) non-response to the quarterly interviews. Non-participants in the Supported Employment Demonstration are individuals who live within the area of a participating Community Mental Health Center who choose not to enroll in the study. The study approaches all eligible denied disability applicants in the project area first by sending them a letter followed by a telephone call to ascertain their interest in learning more about participating in the study. We consider those who indicate they are not interested as well as those who learn more about the study and then decide not to enroll as non-participants. We expect a few individuals contacted will be ineligible.

Social Security has a strong policy interest in knowing who elects to participate and whether or not they are different from those who choose not to participate in the SED project. Non-Participation reflects on the generalizability of the study to the defined group of denied disability applicants. The proportion of those denied applicants who decide to participate in the study represent the take-up rate. The take-up rate is a best guess at what actual participation would be if the agency were to roll out the services to all denied disability applicants. It is an imperfect measure, but can assist

policy-makers with decisions about demand for the services associated with effective treatments. Further, knowing the characteristics of participants and non-participants also contributes to a fuller understanding of demand. Social Security maintains an extensive database on their denied disability applicants. The application and disability review process requires the agency to collect applicant information. In addition to primary demographic variables, SSA collects work history, medical history, and functional limitations. Westat has access to the file information on all denied disability applicants in the defined areas of the community mental health treatment centers participating in the demonstration. Through the enrollment process, we will carefully document the decision of each individual who receives a letter seeking participation in the study. The enrollment process results in three expected classifications of individuals deemed eligible to participate in the study to whom we will send a letter, including those who are: (1) eligible participants; (2) eligible non-participants; and (3) unable to contact. Among eligible participants with whom we make contact during the recruitment process, we will document individuals who turn out to be ineligible, and, therefore, not included in analyses of non-participation. Once enrollment ends, we will conduct an analysis of the enrollment characteristics of the three groups to determine whether non-participants and those we are unable to contact are different from those who elect to participate in the study. We will also address interview nonresponse. We will implement several strategies to maximize response rates and minimize non-response to the quarterly follow-up interviews. To ensure maximum participation throughout the study, we will establish a repository of contacts for each participant – the names, addresses, and telephone numbers of individuals who are familiar to the participant and who can get in touch with them if we cannot. During the baseline interview, the Research Assistant will ask the respondent for the name and number of two such individuals to contact in the event that we cannot locate the respondent. Ideally, at least one of these alternate contacts should not live in the same household as the participant. Subsequently, as part of each quarterly follow-up interview, we will verify the contacts previously provided by the participant. Our MIS will include the data entry fields necessary for recording and updating the additional contact persons. Furthermore, with approval from OMB, we intend to establish a protocol whereby the baseline and quarterly interviews require the interviewer to: ask the participants if they have plans to move; reiterate the toll-free participant number; and offer a $5 cash incentive if the participants call to report their new address. Using these strategies described above, we expect to increase the probability of relocating participants who move following the baseline interview. We will work to minimize attrition and loss to follow-up for the quarterly interviews. We will flag participants who initially refuse to continue their participation in a quarterly interview for further follow-up by a CATI interviewer experienced in refusal conversion.

We will conduct intensive tracing efforts for participants who are lost to follow-up because they are not locatable. TRC interviewers experienced in tracing individuals will use this information as needed to obtain updated contact information for non-respondents who are not locatable. If these efforts prove unsuccessful, we will arrange for our local field staff to conduct in-person tracing efforts, starting with visiting the participants at their last known addresses. With over 50 years of experience in survey data collection, Westat has substantial knowledge of the challenges in achieving high response rates and is experienced in implementing data collection procedures to increase survey response. In addition to the hundreds of surveys we successfully fielded, and our experience with the MHTS, in which we achieved an overall survey response rate of 84 percent (without incentives for the treatment group), we are confident that we can meet a target response rate of at least 80 percent for the SED project. If we do not meet this target response rate, we will conduct a non-response bias analysis using SSA program data, study participation data, and data from the initial baseline survey.

3. Test of Procedures or Methods to be Undertaken

Westat will conduct a cognitive test of new and revised questionnaire items to assess respondent understanding of the questions, as well as pretests of the interview instruments (screener, baseline, quarterly, and annual) to confirm respondent burden and assess question flow. We will conduct both the cognitive test and pretests with no more than 8 denied disability applicants local to the Washington, DC metropolitan area. We will use the results of the tests to refine the questionnaire items and minimize burden.

3. Statistical Agency Contact for Statistical Information

Westat staff developed the data collection and analysis plans; they are also responsible for collecting and analyzing study data.