|
|
Zero Suicide in Health Systems Evaluation
Supporting Statement A
Check off which applies:
☒ New
☐ Revision
☐ Reinstatement with Change
☐ Reinstatement without Change
☐ Extension
☐ Emergency
☐ Existing
The Substance Abuse and Mental Health Services Administration (SAMHSA) is requesting clearance for a new data collection associated with the Zero Suicide Evaluation (ZSE). The Zero Suicide (ZS) initiative is part of the National Strategy for Suicide Prevention grant program that is authorized under Section 520L of the Public Health Service Act (PHSA), 42 U.S.C 290bb-43. SAMHSA’s ZS Program provides funding for community-based primary care or behavioral health care settings, emergency departments, a State mental health agency (or State health agency with mental or behavioral health functions), public health agency, a territory of the United States, or an Indian Tribe or Tribal organization to implement the ZS intervention and prevention framework for adults throughout health systems.
ZS is a commitment to suicide prevention in health and behavioral health care systems and a framework with a specific set of tools and strategies. It proposes that suicide deaths for individuals under care within health and behavioral health systems are preventable, and that a systematic approach to quality improvement in these settings is both available and necessary to identify suicidal patients and keep them safe. The approach aims to improve care and outcomes for individuals receiving services in health care systems, including behavioral health care, who are at risk for suicide. ZS represents a commitment to patient safety—the most fundamental responsibility of health care—and to the safety and support of clinical staff, who do the demanding work of treating and supporting suicidal patients (Suicide Prevention Resource Center [SPRC], 2015).
The ZSE is designed to assess the implementation and outcomes of SAMHSA’s ZS Program. Specifically, the ZSE will gather information about:
Health system implementation of the ZS model, including staff training;
Health care provider training, knowledge, practices, and confidence related to implementing the core elements of the ZS model;
Consumer experiences with services provided under the ZS model; and
Outcomes related to suicide attempts and deaths.
Clearance is being requested for data collection associated with eight instruments—specifically, Web-based surveys, inventory forms, and key informant interviews. These include the following:
Prevention Strategies Inventory (PSI)
Behavioral Health Provider Survey (BHPS)
Key Informant Interviews (KII) with grantees
Workforce Survey (WS)
Training Activity Summary Page (TASP)
Training Utilization and Preservation Survey (TUPS) Baseline, 6, & 12 month follow up
Consumer Experiences Survey (CES)
Consumer Study Interest Form (C-SIF)
Mini Consumer Study Interest Form
Consumer Key Informant Interviews (CKII)
See Exhibit 2 for a description of data collection activities.
Suicide ideation and attempts continue to be a major public health challenge in the United States. Over the last decade, rates of suicide have continued to rise, with the rate of suicide rising from 12.57 per 100,000 in 2013 to 14.21 per 100,000 in 2022 (CDC, 2024). In 2021 alone, an estimated 12.3 million American adults considered suicide. Of these 12.3 million individuals, 28.5% made a suicide plan and 13.8% attempted suicide (Substance Abuse Mental Health Services Association [SAMHSA], 2022). Although suicide is widely associated with mental illness, the CDC study found that the majority of individuals who died by suicide had no known mental health diagnosis (CDC, 2018). Rather, many of those individuals experienced a variety of life stressors prior to their deaths, such as relationship loss/issues, death of a loved one, losing their homes/evictions, and/or other crises (CDC, 2018). Other studies have shown similar results.
Ahmedani and colleagues (Ahmedani et al., 2014) reviewed health visit data from eight health care systems, specifically for patients who died by suicide across a 10-year period. Researchers found that 83% of those patients had accessed care in the year prior to their deaths, and of those individuals, more than half had no mental health diagnosis. Further, nearly 50% of patients who died by suicide made a health care visit within one month of their deaths and only 24% of those patients had a mental health diagnosis. In another study, researchers analyzed Veterans Health Administration service utilization data for male Veterans who died by suicide across an eight-year period. Findings indicated that 55% of male Veterans with substance abuse disorders had accessed care in the month prior to their deaths, and that a quarter of those Veterans accessed services the week before their deaths (Ilgen et al., 2012).
Taken together, these findings reiterate that suicide is not caused by a single factor, nor is it limited to individuals with mental health issues. Further, among suicidal individuals, simply accessing health care services was not sufficient to prevent their deaths. Studies like these show the need for suicide prevention efforts to address all factors contributing to suicide and the importance of expanding efforts beyond the behavioral health system to other environments and settings. Health care systems offer multiple points of access to a wide range of individuals and are both natural and critical environments in which to focus suicide prevention efforts.
The 2012 revised National Strategy for Suicide Prevention (NSSP), the National Action Alliance for Suicide Prevention (Action Alliance; NAASP), in conjunction with the U.S. Surgeon General, recognized that suicide prevention was not a mental health issue, but rather a health issue that needed to be addressed at multiple levels, including within the health care system (U.S. Department of Health and Human Services [HHS] Office of the Surgeon General and NAASP, 2012.
The 2024 National Strategy for Suicide Prevention Strategic Direction 2 recognized the improvement of identifying and treating the risk of suicide in health systems (U.S. HHS, 2024). Goal 8 under Strategic Direction 2 continues to call for the interaction of suicide prevention into healthcare systems and identifies 9 objectives designed to improve suicide outcomes in healthcare settings:
Objective 8.1: Implement effective services to identify, engage, treat, and follow up with individuals with suicide risk as standard care in public and private health care delivery.
Objective 8.2: Develop and implement effective standard protocols to identify, engage, treat, and follow up with individuals with elevated suicide risk in health care.
Objective 8.3: Address practice and policy barriers in order to implement effective emergency department screening, safety planning, and rapid and sustained follow-up after discharge in all emergency departments.
Objective 8.4: Promote effective continuity of engagement and care for patients with suicide risk when they transition between different health care settings and providers, especially crisis, emergency, and hospital settings, and between health care and the community.
Objective 8.5: Ensure suicide prevention competency in initial and continuing education of health professionals to achieve and maintain quality and effectiveness of suicide prevention services.
Objective 8.6: Incentivize and enable health care organizations to track suicide thoughts, attempts, and deaths in their patient and beneficiary populations to inform continuous quality improvement efforts.
Objective 8.7: Increase and leverage the use of electronic health records to track and support implementation of best practices for suicide prevention.
Objective 8.8: Implement effective health care practice strategies that encourage safe and secure storage of lethal means among people at increased risk of suicide.
Objective 8.9: Ensure that suicide prevention services include the capability to identify and address co-occurring substance use issues and ensure that substance use treatment services include the capability to identify and address suicide risk.
The ZS model posits that death by suicide is preventable among consumers of heath and behavioral health services. ZS is now led, operationalized, and continues to be expanded by the ZS Institute. Founded on the realization that suicidal patients often “fall through the cracks” of the fragmented American health care system, the ZS model surmises that a systematic approach to quality improvement is necessary for improvement (Suicide Prevention Resource Center [SPRC], 2015). As such, the ZS model is a comprehensive, multi-setting approach to suicide prevention in health systems that draws on the work of leading health care organizations (HCOs), including the Henry Ford Health System (HFHS). After applying a rigorous quality improvement process to issues like medication errors and inpatient falls, HFHS determined that the same type of process could improve mental and behavioral health care. In turn, HFHS developed the Perfect Depression Care model, an approach that integrates best and promising practices in evidence-based care and quality improvement with suicide prevention as a goal (Coffey et al., 2015). Remarkably, the approach resulted in an approximate 75% reduction in suicide rates among behavioral health care patients in their system (Coffey et al., 2015).
Seven Elements of the ZS Framework
Lead—create a leadership-driven, safety-oriented culture committed to dramatically reducing suicide among people under care; include survivors of suicide attempts and suicide loss in leadership and planning roles.
Train—develop a competent, confident, and caring workforce.
Identify—systematically identify and assess suicide risk among people receiving care.
Engage—ensure every individual has a pathway to care that is both timely and adequate to meet his or her needs. Include collaborative safety planning and restriction of lethal means.
Treat—use effective, evidence-based treatments that directly target suicidal thoughts and behaviors.
Transition—provide continuous contact and support, especially after acute care.
Improve—apply a data-driven, quality improvement approach to inform system changes that will lead to improved patient outcomes and better care for those at risk.
In FY 2023, SAMHSA announced grant funding through its ZS in Health Systems Program (ZS Program) to state and U.S. territory health agencies with mental and/or behavioral health functions, tribes/tribal organizations, community-based primary care or behavioral health care organizations, emergency departments, and/or local public health agencies to implement the ZS model throughout their health systems (Notice of Funding Opportunity (NOFO) No. SM-23-011). This was authorized through Section 520L of the Public Health Service Act. Health systems not providing direct care services can partner with agencies/organizations to implement the ZS model. Communities without well-developed behavioral health care services can implement the ZS model in Federally Qualified Health Centers or other primary care settings (SAMHSA, 2017a).
Grantees are charged with implementing suicide prevention and intervention programs, for individuals aged 25 years or older, designed to raise awareness, establish referral processes, and improve care and outcomes for individuals at risk. The grants require recipients to use their funding to support direct services primarily.
SAMHSA first provided funding for five-year grants for three ZS grantees in fiscal year (FY) 2017 (Cohort 1). Subsequent funding has included 15 grantees in FY 2018 (Cohort 2); 15 in FY 2020 (Cohort 3); 10 in FY 2021 (Cohort 4); 25 in FY 2023 (Cohort 5), and funding has been requested for 11 new grantees, which would create cohort 6 (SAMHSA, 2024) for a total of 68 unique funded ZS grantees and 11 potential additional grantees. Only Cohort 4 will participate in data collection for 2 evaluation instruments, the PSI and the TASP. Later cohorts will participate in all data collection activities.
The purpose of the ZSE is to serve as the primary source of information and to build the program’s knowledge base of effectiveness by thoroughly describing the implementation and outcomes associated with SAMHSA’s ZS Program and the impact of a program meant to reduce deaths by suicide. The ZSE aligns with the provisions for evaluation described in the Public Health Service Act as amended through P.L. 118-35 and enacted on January 19, 2024 (See Attachment A). Because this is a new program, there are no studies on its implementation or effectiveness to date. The collection of this data will enable SAMHSA to report on key outcome measures relating to the grant program.
Team Aptive, formed by Aptive Resources, LLC and its partners, ICF and the Education Development Center (EDC), is the government contractor that will coordinate data collection for the evaluation and provide support for its local-level implementation. Each grantee is required by the cooperative agreement and grant to conduct a self-evaluation and to participate in the ZSE. In this partnership, the government contractor provides training and technical assistance (TTA) regarding data collection and design for the evaluation. In addition, the contractor directly collects data, receives data from grantee data collection efforts, monitors data quality, and provides feedback to grantees. Data gathered through the ZSE will continue to be used for grantee-specific and national program assessments.
SAMHSA is requesting OMB approval for a new data collection, the ZS Evaluation. We request approval for three years of data collection associated with the proposed design, which represents SAMHSA’s desire to support the design, implementation, and dissemination of findings of a national evaluation of the ZS Program.
The ZSE is a multimethod, multisite design that aligns with SAMHA’s desires to analyze implementation, outcomes, and impact of its ZS Program. The evaluation includes four studies: Systems Change, Workforce, Consumer Experience, and Impact. The design considers required and allowable activities; variation in the partnerships, and health care systems/organizations; existing data systems and grant infrastructures to support implementation; and evaluation participation. The evaluation design is intended to monitor progress and outcomes across the grant funding period.
SAMHSA is interested in determining the extent to which strategies employed by grantees and/or their partnering health care systems are consistent with the ZS model, assessing the feasibility of implementing the ZS model, and determining the outcomes associated with implementation.
Specifically, the ZSE will gather information on:
Health care system implementation of the ZS Framework;
Health care provider training, knowledge, practices, and confidence related to implementing the core elements of the ZS Framework;
Consumer experiences with services provided under the ZS Framework; and
Outcomes related to suicide attempts and deaths.
Further, the ZSE will draw on extant Medicaid claims data to compare outcomes for consumers receiving services from grantees that are HCOs and/or HCOs partnering with grantees (hereafter, “participating HCOs”) implementing ZS and those receiving services from non-ZS HCOs.
The overarching evaluation questions are presented below for each study in Exhibit 1.
Exhibit 1: Overarching Evaluation Questions
Overall Evaluation Question for each Study |
Systems Change Study |
EQ1. To what extent are grantees implementing the ZS Program in accordance with the ZS Framework, adoption of core activities, and indicators of sustainable systems change? EQ2. What have been the challenges and barriers to implementation? Are there lessons learned in overcoming these challenges? EQ3. What is the impact of emerging technologies utilized by the grantees such as chat, text, and use of social media? EQ4. How do grantees effectively assess the impact among populations at risk from marginalized communities such as AI/AN, Black, and LGBTQIA+ where there may not be enough to analyze mortality? EQ5. What is the cost of implementing the framework per consumer? What is the cost of implementing per outcome? |
Workforce Study |
EQ7. Are the staff at ZS-participating HCOs well-trained and prepared to address suicide risk among their consumers? EQ8. Are HCO staff aware of suicide prevention protocols and care management plans generally? EQ9. Are ZS staff aware of the ZS activities in particular? EQ10. To what extent are providers utilizing the interventions on which they’ve been trained? (Implementation Evaluation) EQ11. Are gains in identification and management of individuals with suicide thinking, planning, and attempts sustained at 1 year from exposure? (Outcome Evaluation) EQ12. Do individuals [in the health system workforce or in the community in the areas served by training of the grant report greater knowledge, confidence, and an improvement in demonstrable skills in addressing suicide (identification and management)? Is this a short-term gain or long-term gain (e.g., at 12 months)? (Outcome Evaluation) |
Consumer Experience Study |
EQ13. What has been the experience of consumers of ZS-participating HCOs, from screening to care and care transitions? E14. How satisfied are consumers with their experience? Do consumers have particular areas of concern? Have consumer experiences changed over time with the implementation? EQ15. What specific component of the grant program do individuals report as most helpful, both immediately and long term? EQ16. To what extent are providers utilizing the interventions on which they’ve been trained? EQ17. Do consumer experiences differ by setting, services received, or demographic factors? EQ18. How do clinical behavioral health outcomes (i.e., suicide risk assessment scores, depression symptoms, anxiety symptoms, suicide attempt/death rates) change over time among ZS grantees? |
Impact Study |
EQ19. What is the overall impact of the ZS program on reducing suicide morbidity and mortality? |
Approval is being requested for eight data collection activities that compose the ZSE. A description of each instrument, the methods used, and respondents is provided in Exhibit 2.
Exhibit 2. Data Collection Instruments
Instrument and Acronym |
Description |
Prevention Strategies Inventory (PSI)
Attachment B |
The PSI (Prevention Strategy Inventory) is an online tool used every quarter for the ZS Program. It collects detailed information about the suicide prevention strategies and resources that are being implemented. This includes the kinds of strategies and resources, their target groups, and how much money is being spent on each category. The PSI is organized into main strategy categories, each with sub-categories for more specific details about the activities or resources and their targeted populations. PSI gathers data on three key areas: the use of new technologies, the application of proven practices for suicide prevention, and efforts in cultural adaptation and promoting health equity. The budget section captures expenditures across major categories (e.g., outreach and awareness, training, screening programs, etc.). The PSI-ZS takes approximately 1 hour to complete and is completed by one grantee staff person each quarter via the Web-based data collection and management system, the ZS Data Center (ZSDC), for an average of 40 respondents annually. |
Behavioral Health Provider Survey
Attachment C
|
The BHPS (Behavioral Health Provider Survey) is an annual online survey taken by administrators of healthcare organizations (HCOs) participating in the ZS program. This survey, initially based on a measure from the 2017 SAMHSA ZSE and refined through Ohio’s Department of Mental Health and Addiction Services ZS grant, aims to understand how patients' experiences (like the services they receive and their views on care) relate to the characteristics and practices of their healthcare organizations. This includes things like staff training, location, and ZS policies. In year 2 for cohort 5 and years 1 and 3 for cohort 6, one administrator/director from every location that HCO operates within each ZS grant will fill out the BHPS. When the evaluation starts, grantees will provide the names of these administrators. Before the survey begins, Aptive will email the administrators with instructions and a survey link. The survey will be open for 30 days, with reminder emails sent after 15 and 25 days. (NOTE: Full administration will be years 2 and 5 for cohort 5 and years 1,3,5 for cohort 6, but not all administrations fall within the OMB request) |
Key Informant Interviews
Attachment D
|
HCO Case Studies: Key Informant Interviews (KIIs) conducted with a sample of grantees (up to five grantees per cohort) with a maximum of 10 interviews per grantee of various workforce levels within a ZS-participating HCO. Grantee participation will be selected based on representation from the various grantee types (i.e., tribal, urban, rural). Information gathered will include successes and challenges in implementation, contextual information around the extent of implementation of the ZS Framework, including collaboration and partnerships, policies and protocols, and facilitators and barriers that impact program implementation. |
Key Informant Cost Study Interviews Attachment E |
Cost Sub-Studies: A subset of case studies (up to five grantees) will focus on the costs of ZS implementation, which will support nuanced understanding of costs incurred beyond the scope of grant activities or funding. These may include lost revenue when clinical staff are taken “offline” for training, as well as costs associated with improving screening and assessment processes, implementing evidence-based treatments, enhancing follow-up and aftercare services, and integrating data collection and analysis systems. |
Workforce Survey (WS)
Attachment F
|
The Workforce Survey (WS) is a web-based survey that collects information from staff at ZS-participating HCOs. Data will include knowledge of the ZS model and its core elements, receipt of training on and use of ZS practices, and self-efficacy around implementing the core elements of the ZS model. Appropriate respondents are clinical staff who have direct contact with patients (e.g., psychiatrists, internists, nurse practitioners, registered nurses, therapists, etc.) and non-clinical staff. The survey will be administered in year 2 for cohorts 5 and in years 1 and 3 for cohort 6; and will take 10-15 minutes to complete. Up to 600 staff from each grantee will participate, for an average of 9,400 respondents per year. (NOTE: Full administration will be years 2 and 5 for cohort 5 and years 1,3,5 for cohort 6, but not all administrations fall within the OMB request.) |
Training Activity Summary Page
Attachment G
|
The Training Activity Summary Page (TASP) is a web-based survey completed by grantees to collect key information about the completed training event, including the number of training participants, the professional role of the participants, type of training and resources offered, cultural adaptations of the training, training setting, and training location ZIP code. This form consists of 16 questions and should be completed for every training activity implemented as part of the ZS program. This data will provide important characteristics and insights of completed ZS training and activities to identify gaps and inform future suicide prevention training. Grantees are asked to submit TASP forms on an ongoing basis after each training event they hold. It is estimated that grantees will spend about 15 minutes each quarter completing TASPs. |
Training Utilization and Preservation Survey (baseline, 6, and 12 month)
Attachment H and I |
The Training Utilization and Preservation Survey (TUPS) is a web-based training follow-up survey of staff who participate in ZS funded trainings. Staff who completed the ZS-related training will complete the survey at the end of each training course and at the 6th and 12th- month mark after the training date. Each participant will provide their contact information and consent to be re-contacted to complete the survey. Data collected in the survey include trainee characteristics (e.g., provider role, education level, demographics); changes in awareness, knowledge, confidence, and skills in addressing suicide; self-reported application of training skills (e.g., identification and management). There will be 10,000 baseline, a sample of 756 6-month, and an expected 567 12-month. Respondents to the 6-month and 12-month follow—ups will receive a $10 gift cared. |
Consumer Experience Survey
Attachment J
|
The web-based Consumer Experience Survey (CES) will be completed by individuals receiving services through a ZS grantee when they enroll in services and six months after they enroll in services. The CES includes questions on mental health status, crisis experiences, safety planning, care transitions, satisfaction with care, and behavioral outcomes, as well as suicide-specific assessment, screening, and services received. The survey will be administered to adults (aged 25 years and older) who receive behavioral health services from a ZS organization and have suicide care management plans. To enable conclusions about changes in consumer experiences and clinical outcomes throughout the course of ZS implementation, CES recruitment will be limited to individuals receiving care from grantees funded through the most recent cohorts (i.e., Cohort 5 and later). After enrolling a consumer into a Zero Suicide care management plan, Zero Suicide grant staff will provide consumers with information about the CES via the Consumer Study Interest Form (C-SIF, described below). Consumers who are interested in study participation will complete the C-SIF with their provider, which will be submitted directly to the ZSDC. Consumers will receive a follow-up email inviting them to review a consent form for the CES and complete the baseline survey. Consenting consumers will receive a similar emailed survey invitation six months after their baseline survey. Respondents will be provided with a $15 gift card after completing the baseline survey and a $15 gift card after completing the six-month follow-up survey in appreciation of their time and participation. There will be approximately 4,500 C-SIF completions,3,375 baseline CES completions and 2,530 6-month CES completions. |
Consumer Study Interest Form
Attachment K
|
The Consumer Study Interest Form (C-SIF) will be used to guide participant recruitment into the study. The C-SIF provides a brief recruitment script and collects information about the best contact methods for survey invitations, suicide risk, and care status (i.e., suicide care management plan status) for those interested in study participation. Suicide risk will be assessed through the Columbia Suicide Severity Rating Scale (C-SSRS), which has been validated with a myriad of clinical populations (Posner et al., 2011) and is recommended as a screening tool by the ZS Toolkit (EDC, n.d.). The C-SIF will be completed by grantees when individual consumers enroll in a suicide care management plan with their provider organization. Grantees will have the option of incorporating these questions into an intake interview by verbally asking each question to participants, or by offering participants to answer questions directly in a self-report format (i.e., via tablet, laptop, or other device) at the close of the interview. Similar to the CES, C-SIF implementation will be limited to grantees in Cohort 5 and later. A $5 gift card will be provided to participants who complete the full C-SIF with their provider. There will be 4,500 C-SIF respondents. (125 per grantee) |
Consumer Key Informational Interview (C-KII) Attachment L
|
A series of virtual key informant interviews with consumers will shed additional light on perceptions and outcomes of care within a ZS system. During the course of the one-hour interview, participants will answer questions about their experiences with components of care embedded within the ZS Framework, including elements of the Identify, Engage, Treat, and Transition domains. Participants will also be asked to share the life experiences that were most directly connected to engaging with their provider agency, outcomes of their care, and any barriers or facilitators they experienced related to care engagement. Participants for C-KIIs will be recruited from a pool of CES participants who indicate interest in a follow-up interview. Potential participants will be stratified by their reported suicide risk level, as measured by the C-SSRS, at baseline to ensure that interviews represent experiences with care across a range of consumer needs. A total of 15 interviews will be conducted throughout each year of data collection. A $30 gift card will be provided to interview participants. |
The instruments above are all part of the 4 studies that comprise the ZSE. Each of these studies has unique evaluation questions and approaches.
Systems Change Study
The Systems Change Study will seek to evaluate the adoption and fidelity of the ZS Framework within HCOs, a comprehensive suicide prevention setting that has not yet been extensively examined. The study will explore the challenges and barriers to implementation, the role of organizational characteristics, and the impact of emerging technologies on healthcare delivery. Additionally, the study will examine the service delivery and engagement strategies for marginalized populations, such as AI/AN, Black, and LGBTQIA+ communities, and the economic evaluation of the framework’s cost-effectiveness. The Systems Change Study is designed to understand how grantees are implementing the ZS Program in accordance with the ZS Framework, the core activities accomplished, and indicators of sustainable systems change (i.e., policy and practice changes, infrastructure changes, organizational culture).
Workforce Study
The Workforce Study is designed to document staff awareness and perceptions associated with the ZS activities implemented by ZS-participating Healthcare Organizations (HCOs). This study also seeks to understand the utilization, outcomes, and sustainment of training programs intended to increase the knowledge, confidence, and skills among staff to address suicide, both in the short and long term.
Consumer Experience Study
The Consumer Experience Study is designed to examine the relationship between ZS activities and key clinical outcomes (i.e., suicide risk, depression), along with consumer perceptions of care, access to care, services received, and treatment adherence.
Impact Study
SAMHSA's ZS program aligns with the broader national strategy for suicide prevention, acknowledging the need for a unified and systematic response to mitigate the impact of suicide on individuals and communities. The Impact Study will investigate the overall impact of the ZS program on reducing suicide morbidity and mortality by using secondary data and quasi-experimental designs to develop a control group and estimate the causal impact of the ZS Program on suicide morbidity and mortality.
Information collected through the ZSE will address current strategic initiatives and goals in SAMHSA’s 2023-2026 Strategic Plan, which aims to better meet the behavioral health needs of the United Sates and provide a roadmap to improve and advance public health and service delivery efforts that promote mental health, prevent substance misuse and overdose, and provide resources that foster recovery while also ensuring equitable access and better outcomes (SAMHSA Strategic Plan, 2023) See Attachment L. The ZSE aligns with the plan’s new mission and vision that are supported with five priority areas (Preventing Substance Use and Overdose, Enhancing Access to Suicide Prevention and Mental Health Services, Promoting Resilience and Emotional Health for Children, Youth and Families, Integrating Behavioral and Physical Health Care, and Strengthening the Behavioral Health Workforce) and four core principles (Equity, Trauma-Informed Approaches, Recovery, and Commitment to Data and Evidence).
The information collected through the ZSE will also answer process and outcome questions related to the implementation of the ZS model, respond to recommendations by the 21st Century Cures Act Interdepartmental Serious Mental Illness Coordinating Committee (ISMICC), advance the field of suicide prevention, and meet government performance reporting act (GPRA) measures. The information gathered through the ZSE will be essential to SAMHSA and others in helping communities and decision-makers at all levels of government to make informed funding decisions and contribute to the evidence base around suicide prevention program effectiveness.
SAMHSA continues to invest in key suicide prevention efforts such as ZS programs. The ZS Program aligns with the second priority in SAMHSA’s Strategic Plan 2023-2026, “Enhancing Access to Suicide Prevention and Mental Health Services,” which aims to enhance access to suicide prevention and crisis care as crucial elements of the mental health continuum of care, so that people experiencing suicidal ideation and other behavioral health crises can receive the care they need and want in order to thrive and achieve well-being (SAMHSA Strategic Plan, 2023). Specifically, the ZS Program Addresses Objective 2.1 and 2.2 (see below) of SAMHSA Strategic Plan 2023-2026:
2.1 Improve access to suicide prevention services.
Suicide prevention services must also be embedded through the broader public health and healthcare systems. To reinforce these programmatic attributes, states and territories are required to report on their systemic suicide prevention activities in the Community Mental Health Block Grant application. An example of this work can be seen in the ZS grant program, which supports the implementation of the ZS intervention and prevention model for adults throughout a health system or systems (SAMHSA Strategic Plan 2023-2026, 2023).
2.2. Improve the quality and effectiveness of suicide prevention services.
SAMHSA will enhance suicide prevention services by supporting training standards and promoting the adoption of practices that are evidence-based, evidence informed, or promoted through expert consensus. This type of work is supported through the ZS grant program that supports the implementation of the ZS intervention and prevention model for adults throughout a health system or systems (SAMHSA Strategic Plan 2023-2026, 2023).
The ZS program aims to intercept individuals at risk for suicide within health care systems and refer them to evidence-based treatments, with an overall goal of reducing the likelihood of suicide. The ZSE also will assist SAMHSA in answering process and outcomes questions related to the ZS Program by assessing the extent to which grantee strategies are consistent with the ZS model, assessing the feasibility of implementing the ZS model in real world settings, and determining the outcomes associated with implementation meant to reduce deaths by suicide.
The ZS Program addresses a recommendation made to Congress, by ISMICC. The 21st Century Cures Act (Public Law 114-255) and the Federal Advisory Committee Act authorized the ISMICC, a federal advisory committee that includes government and public members, to coordinate across federal agencies and address the needs of adults with serious mental illness (SMI) and children and youth with serious emotional disturbances (SED) and their families (see Attachment M). In 2017, the ISMICC released a set of recommendations to Congress, including one calling for the adoption of ZS across all federal agencies. SAMHSA’s ZS Program addresses recommendation 3.7 (see below).
3.7 Advance the national adoption of effective suicide prevention strategies. All federal departments, including the Veteran’s Administration and Department of Defense, should adopt ZS as a model for suicide reduction, and agree to develop and implement strategic plans with achievable and transparent targets for progress. Consider ways to widely disseminate and universally apply these strategies in the public health system (ISMICC, 2017).
Since 2021, SAMHSA was among the ten federal agencies that support programs that address behavioral needs of individuals with SMI and SED. SAMHSA is addressing this recommendation through its funding of the ZS Program. In addition, SAMHSA is helping to establish “achievable and transparent targets for progress” through the ZSE. In particular, the ZSE will document the implementation of ZS policies and practices in a range of health care systems, identify successful strategies for implementation, determine the feasibility of implementing ZS in real world settings, and provide measures of ZS outcomes.
Results of the ZSE will help answer the call by those in suicide prevention to identify and establish evidence for effective interventions to address the mounting epidemic of suicides in this country. To establish this evidence, the process- and outcomes-focused evaluation will gather information on the implementation of ZS policies and practices across a range of health care systems; determine the feasibility of ZS model implementation in real world settings; assess health care provider knowledge, practices, and confidence to implement ZS practices; and examine consumer experiences, satisfaction, and outcomes related to the receipt of services through ZS heath care systems. Analysis of ZSE data will identify variations in ZS model implementation across participating HCOs and assess consumer outcomes (suicide attempts and deaths) associated with those variations in implementation. The evaluation also will generate evidence of the ZS model’s real-world success, provide a roadmap for implementing the ZS model in health care systems, and begin to establish realistic benchmarks for consumer outcomes. Further, the ZSE will help answer the ISMICC recommendation that all federal agencies “develop and implement strategic [suicide prevention] plans with achievable and transparent targets,” while helping other agencies and health systems to do the same.
The findings will add value to SAMHSA and its partners, other federal agencies, the Action Alliance, ZS grantees, legislators, the field of suicide prevention, and consumers and the communities in which they live. Results will inform SAMHSA’s decision making around continued and future funding of the ZS Program and other suicide prevention initiatives, such as the Garrett Lee Smith (GLS) Suicide Prevention Program and NSSP grantees, as well as other federal agencies engaged in ZS work. The ZSE will allow SAMHSA, the Action Alliance, and other federal agencies to expand the evidence base for suicide prevention initiatives; address factors contributing to suicide deaths and attempts; and establish standards for developing, implementing, and evaluating suicide prevention programs.
The ZSE will allow Federal and local officials to determine whether the ZS programs implemented have an impact on the prevention of suicide; their effectiveness on identification, referral, and provision of evidence-based interventions to individuals identified as at risk; and whether grantee programs are meeting ZS Program goals. SAMHSA also will use the data collected to provide objective measures of its progress toward meeting targets of key performance indicators put forward in its annual performance plans as required by law under the Government Performance Requirements Act (GPRA). Grantees currently submit Infrastructure Development, Prevention, and Mental Health Promotion (IPP) Indicators including Training-suicide risk assessment trainings (TR3), Screening (S3), Referral (R3), Access (AC1), Types/Targets of practice-suicide deaths (T7), and Types/Targets of practice-attempts deaths (T8).
Every effort was made to reduce the burden on individual respondents who participate in the ZSE. The data system is designed with the user in mind, making it intuitive and easy to navigate. All data collection instruments will be administered via the Web. Data collection will be possible via a computer or mobile web-enabled device to allow grantees to submit data at the best time and place for them. In addition, features of the system will assist grantees in tracking their data entry and responses.
Every effort has been made to reduce the burden on individual respondents who participate in the ZSE while maintaining accuracy and efficiency through technology. Team Aptive will integrate the proposed ZSDC into the existing Suicide Prevention Data Center (SPDC), which is being developed for the evaluation of the GLS Youth Suicide Prevention Program. Our team will integrate the ZSDC using the latest Microsoft .NET Framework version that SAMHSA’s cloud supports, along with Amazon RDS for SQL Server RDBMS. A custom URL will bring users to the ZS-branded landing page. Role-based permissions tied to login credentials will ensure that only ZS users can access the ZS data.
The web based ZSDC system will support data collection, management, and dissemination activities associated with the ZS evaluation, including communication between grantees and the evaluation team, secure data transmission and storage, data quality monitoring that triggers corrective action when necessary, and updates around evaluation activities and performance. It will serve as the portal for data collection forms, surveys and data upload tools supporting the ZS Evaluation, providing the flexibility for the loading and processing of new data collection protocols in addition to the importing or linkage of data from existing sources. The system will provide a series of tools that allow the monitoring of response rates in real time, support the data reporting and analytical needs of the program, and allow the downloading of response data in real-time throughout the data collection cycle, including production of data sets at least twice a year for delivery to grantees and final data sets once data collection has been completed. Core functionality will include the functions detailed in Exhibit 3.
Exhibit 3. Zero Suicide Data Center (ZSDC) Core Functionality
Function |
Security Level (Role Based Access) |
Description |
User Administration |
Grantees: Manage own user profile and add subordinate users with same or limited permissions
SAMHSA: Manage own user profile
Team Aptive: Full access to administration functions
|
Allow a user with the proper permissions to add a new user, edit a user profile, deactivate/delete a user, and assign permissions. A minimum amount of identifying information must be collected from each user to ensure the system can uniquely identify each individual and successfully contact them. The system should only collect personally identifiable information (PII) that is critical to meeting this requirement, so the risk of loss and exposure of sensitive data is minimized. The password specifications for each user of the system should follow Federal Information Processing Standard Publication 140-2 (FIPS 140-2) standards, which is a U.S. government computer security standard used to approve cryptographic modules. |
Data Collection and Storage |
Grantees: Access to data collection forms, surveys, and data import tools.
Team Aptive: Access to grantee data collection forms and surveys; manage and send data collection reminders |
Provide web-based data collection forms, surveys, and data import tools; automated data collection reminders; text-based data entry reminders. Web-based forms support data collection using online forms directly within the Web browser and will utilize responsive design to support varying screen sizes across a wide spectrum of modern-day devices (e.g., desktop, laptop, tablet, and phone). The Web-based forms require a constant Internet connection and will only operate online.
|
Data Management |
Team Aptive: Role-based access to raw data |
Secure storage of grantee level data allowing for efficient aggregation of data for analysis in SAMHSA requested formats. |
Response Monitoring |
Grantees: Track own progress and completeness
SAMHSA: View high-level and grantee-level data submission metrics
Team Aptive: View high-level and grantee-level data submission metrics and detailed completion information |
Monitoring reports will be available at several levels of detail, allowing users to view real time aggregate level data and drill down into more detailed submission statuses. Depending on level of access, a user can view high-level data submission metrics across all grantees in the system, or they can view detailed completion information for a particular grantee. Progress and completeness can be tracked at an overall report level or by categories or major sections within each individual data collection tool. |
Quality Control Tools |
Grantees: Respond to real-time validations during data entry or upload
Team Aptive: Data review and editing tools; communicate with grantees for error resolution
|
Use a combination of client-side validations in the web form interface and server-side validations that are run against data already stored in the database, to ensure all validation rules are implemented. Front-end validations prevent invalid or incomplete data from being saved into the database. These types of validations can be warnings that allow a user to proceed or hard errors that must be corrected before proceeding. Using validations and data reports, the system will allow for quick identification of inaccuracies and anomalies in the data and allow for corrective action. Data review and editing tools used by the data collection and data management teams to ensure that the evaluation data provided by grantees are complete, clean, consistent, and usable for analysis and reporting. |
Data Set Download |
Grantees: Access and download own data sets
SAMHSA/ Team Aptive: Access and download aggregated and individual data sets |
Users with the appropriate permissions can access and download data sets for an individual grantee and/or across all grantees. Users can select entire data sets or select and filter on a subset of data. |
Data Reporting |
Grantees: Access to canned grantee-specific reports
SAMHSA: Access to canned aggregate and grantee-specific reports
Team Aptive: Access to full set of reports
|
Provide automated data reporting capabilities, including grantee-level progress data collection participation reports, program monitoring reports, data issues, assessments, and findings. Reports will provide users with access to all pre-programmed data reports. These reports may include static and dynamic data reporting capabilities on data issues, data quality, and program monitoring and evaluation information. The reports will be developed as canned reports, meaning that they have a defined, static format, but the data populated within them is dynamic based on the access level of the user, the selected filters, and the most up-to-date data available in the database. |
Evaluation Resources |
Grantee: Access to general and targeted resources
SAMHSA/Team Aptive: Access to all resources
|
Maintain a repository of evaluation related documents (e.g., data dictionaries, codebooks, user manuals). This repository allows for retrieval of important documents and links to websites and other information. All users with appropriate permissions can download documents and click on shared links. Downloads and link clicks should be tracked by user and by date. These metrics will be stored in the database and are used to determine the usefulness of each resource that is posted. |
As the ZSDC will be built inside of the existing infrastructure of the GLS National Outcomes Evaluation Suicide Prevention Data Center, a System of Records Notice (SORN) and a U.S. HHS Privacy Impact Assessment (PIA) form will be completed when then system is further into development.
Before developing data collection activities for the ZSE, we reviewed the literature to avoid duplication in data collection activities and similar information. Specifically, we reviewed existing evaluation studies and the efforts of other Federal initiatives designed to evaluate the implementation of the ZS model in health systems. In addition, we reviewed and incorporated instruments already required and implemented by grantees. Two instruments in particular are already required of ZS grantees. The BHPS and the WS are both based on instruments from the Zero Suicide Institute (ZSI) (the Organizational Self-Study and the WS). Grantees will receive data reports from our evaluation team that they will be able to utilize to submit required data to the ZSI.
To date, there is a dearth of literature on the implementation and outcomes of the ZS model as implemented in health care systems, though research efforts are ongoing (see Section 4.iii.). The ZSE is the first national evaluation of a grant-funded program to implement the ZS framework in health care systems. SAMHSA’s ZS grantees are charged with implementing suicide prevention and intervention programs—for individuals 25 years and older—that are designed to raise awareness of suicide, establish referral processes, and improve care and outcomes for adults at risk for suicide. The ZSE process- and outcomes-focused design draws on existing resources in place for the evaluation of SAMHSA’s GLS Suicide Prevention Program. Those instruments have been tailored for use with the ZSE (for health care systems implementing the ZS framework for adults), and findings will not be duplicative of the GLS NOE.
Currently, SAMHSA, the Indian Health Services (IHS), and the National Institutes of Health (NIH) National Institutes of Mental Health (NIMH) are implementing other efforts related to ZS. These are described below.
From 2014 to 2016, SAMHSA awarded twelve National Strategy Grants (NOFO Number: SM-14-016) to states to support their implementation of the 2012 NSSP goals and objectives focused on preventing suicide and suicide attempts among individuals aged 25 years and older (SAMHSA, 2016). From 2017 to 2019, SAMHSA awarded fourteen additional National Strategy Grants (SM-17-007) (SAMHSA, 2017a). The program funds states to implement services or practices with a demonstrated evidence base that are appropriate for working-age adults from 25 to 64 years old and aims to reduce the overall suicide rate and number of suicides in the U.S. nationally (SAMHSA, 2017b). National Strategy grantees are not eligible for ZS Program awards and vice versa. While the National Strategy Grants cooperative agreements require grantees to conduct a local evaluation, the program currently does not have a national evaluation.
From 2020 to 2022, SAMHSA awarded fifteen grants (NOFO Number: SM-20-014) to continue to bolster the NSSP and its state and community-focused efforts to address suicide in individuals aged 25 and older (SAMHSA, 2020). In 2023, SAMHSA awarded five NSSP grants (NOFO Number: SM-23-017) to health care and behavioral health care sites focused on adult suicide prevention, particularly those treating individuals in rural areas, older populations, and American Indian and Alaska Native adults (SAMHSA, 2023).
Beginning in 2014, IHS funded eight tribal sites/tribal health systems through its ZS Initiative to help address high suicide rates in Indian Country. The ZS Initiative aims to improve the system of care for those at risk for suicide by implementing a comprehensive, culturally informed, multi-setting approach to suicide prevention in Indian health systems (U.S. HHS IHS, 2017). The awards support the implementation of the ZS model within federal, Tribal, and urban Indian health care facilities and systems that provide direct care services to AI/AN individuals to raise awareness of suicide, establish integrated systems of care, and improve outcomes for such individuals. The program represents a continuation of IHS’s efforts to implement the ZS approach in Indian Country. In 2021, IHS funded eight tribal/urban tribal organizations to implement one element of the SPDC ZS framework, with emphasis on the element Improve (U.S. HHS IHS, 2022). IHS's previous efforts focused on TTA and consultation for several pilot American Indian/Alaska Native ZS communities, which resulted in an understanding of the unique opportunities and challenges of implementing ZS in Indian Country. Currently, the ZS Initiative does not have a national evaluation.
In 2016, the NIH NIMH funded three ZS studies focused on adult- and youth-based suicide prevention practices in health care settings. These efforts include improving the quality of behavioral health care to reduce suicide risk, testing the efficacy of a suicide prevention approach, and identifying youth at risk for suicidal behavior (NIMH, 2016). In 2017, the NIMH also funded a cross-site evaluation of six health systems implementing the ZS model. Each of these four studies is described below.
Study 1: Improving Behavioral Health Care
Researchers at Columbia University partnered with the New York State Office of Mental Health to compare quality improvements in suicide prevention practice across 145 outpatient state licensed clinics (representing 85 New York state agencies and including 1,490 clinical providers that reach over 80,000 adult clients) (NIMH, 2016). The study team randomly assigned some clinics to additional training, tracking, and other infrastructure support, to learn of the best ways to improve suicide screening and safety planning.
Study 2: System of Safety (SOS) in Multiple Types of Care Settings
The SOS study builds on findings from the Emergency Department Safety Assessment and Follow-up Evaluation (ED-SAFE), a study of adults at risk for suicide who visited an ED for care (Boudreaux et al., 2022; NIMH, 2016). The ED-SAFE study assessed outcomes related to improved brief suicide-risk screening, providing outpatient suicide prevention discharge resources, and follow-up telephone counseling for the patient and a significant other. The ED-SAFE and the ED-SAFE 2 used a continuous quality improvement approach to implement, monitor, and enhance the interventions during routine clinical care. Through SOS, researchers from the University of Massachusetts Medical School are extending the ED-SAFE CQI approach to additional care settings in the local health care system (representing 6 ED units, 25 inpatient units, and 8 primary care clinics and an estimated 310,000 patients aged 12 and older). The SOS uses an innovative phased roll-out study design to test the effectiveness of its suicide prevention approach as compared to standard care. The team is also assessing the cost effectiveness of SOS.
Study 3: ‘Stepped’ Care for Youth Suicide Prevention
The ‘Stepped’ Care for Youth Suicide Prevention study focuses on youth receiving services through the Kaiser Permanente Northwest (KPNW) health system who are at risk for suicide (NIMH, 2016). Researchers from the Kaiser Permanente Center for Health Research are assessing outcomes for 300 at-risk youth aged 12 to 24 years. Building on previous NIMH research on dialectical behavior therapy and internet cognitive therapy, the study compares outcomes for youth receiving two care approaches: (1) the KPNW system’s ZS practices and (2) a stepped care treatment approach that includes ZS practices and matches intensity of treatment to severity of risk. Researchers are exploring which group benefits more in terms of reduced suicide attempts and other patient outcomes. The KPNW effort also includes a cost effectiveness component.
Study 4: An Evaluation of the ZS Model Across Learning Healthcare Systems
Researchers from the HFHS and Kaiser Washington Health Research Institute are implementing a cross-site evaluation of six health systems implementing specific components of the ZS model to determine separate and cumulative benefits. Delivery system leaders will guide the selection and implementation of specific ZS components in each system. The goals of the study include (1) measuring the fidelity of ZS components implemented, (2) examining outcomes (suicide attempts and deaths) within and across the various models, and (3) developing electronic health record metrics to assess care processes and quality improvement targets tailored to local implementation of the ZS model.
Some HCOs partnering with ZS grantees are state/local behavioral health agencies or private health care systems. While most HCOs have been public agencies or large organizations to date, it is possible that some small entities may qualify in the future. Although respondents to the BHPS and WS may be employed by small businesses or other small entities, these data collections will not have a significant impact on the agencies or organizations.
SAMHSA is interested in determining the extent to which strategies employed by grantees are consistent with the ZS model, assessing the feasibility of implementing the ZS model in real world settings, and determining the outcomes associated with the model’s implementation. As such, the ZSE and its evaluation questions are aligned with the foci of the ZS Program. The rigor of the ZSE design and its ability to answer the primary evaluation questions are dependent on the frequency of the data collected. Thus, the frequency with which data collection activities are administered is critical to SAMHSA’s overall assessment of the program. Exhibit 4 describes the consequences if data are collected less frequently.
Exhibit 4: Consequences and Rationale if Data are Collected Less Frequently for Each Study
Activity |
Rationale |
PSI |
Grantees will be required to complete the PSI quarterly over their grant periods. Collecting this information quarterly is necessary to track progress toward meeting suicide prevention goals, to provide information on the development of strategies and products and delivery of services within ZS programs, and on budget expenditures. The consequences of collecting the PSI less frequently include losing the ability to track progress over time, not collecting information related to the program, not capturing budget expenditures over time, and inhibiting our ability to fully understand the different ZS elements that have been implemented by grantees. |
BHPS
|
The BHPS is administered twice during the evaluation for cohort 5 grantees and bi-annually for cohort 6 and will inform SAMHSA about the implementation of ZS practices and consumer outcomes. The BHPS is administered annually and will inform SAMHSA about the implementation of ZS practices and consumer outcomes. Information collected through the BHPS is critical to understanding the practices and policies implemented by grantees and their partner HCOs. If the BHPS were administered less than once per year, it would affect SAMHSA’s ability to determine whether the practices implemented are consistent with the ZS model, whether it is feasible to implement the ZS model in real world settings, whether implementation changes over time, and outcomes associated with implementation. |
KII |
KIIs are collected on an ongoing basis over the evaluation. If data collected for the KII’s for the ZSE were collected less frequently, it would specifically jeopardize the robustness of responses to key evaluation questions. For instance, questions related to the adoption and fidelity of the ZS Framework rely on continuous monitoring to accurately track the implementation stages and sustainability of the framework's elements over time. Infrequent data collection could lead to incomplete or outdated insights into the challenges, barriers, and successes in implementing and maintaining these elements. Furthermore, questions regarding the penetration and coverage of services and costs of implementation would also be affected. Without regular data, it would be challenging to assess the reach and impact of the program among marginalized communities effectively or to accurately calculate the ongoing costs associated with implementing the framework. This would significantly hinder SAMHSA’s ability to evaluate the program's effectiveness, adaptability, and overall value comprehensively. |
WS |
The WS is administered twice during the evaluation for cohort 5 grantees and bi-annually for cohort 6 to assess staff knowledge, practices, and confidence related to the core elements of the ZS Framework. The WS will inform SAMHSA about the extent to which the ZS model has been adopted by HCOs and the feasibility of implementing the ZS model in real world settings. Collecting this information less frequently would affect SAMHSA’s ability to assess workforce adoption and implementation of the ZS model and progress/change over time across grantees. |
TASP |
A TASP is completed for each training event. Because staff training is a requirement of ZS grantees, aggregate basic information about trainings, trainee types, and trainee roles is necessary for SAMHSA to understand how grant funds are being utilized in support of training and to understand impact of ZS training on providers’ confidence, knowledge, and skills. |
TUPS |
The baseline TUPS is administered to staff who participated in ZS funded training immediately before the training. This survey will provide information to evaluate changes in confidence levels, knowledge, practices, and utilization of ZS skills over time. Collecting this information less frequently would affect the ability to assess any of these changes or progress and miss out on important data describing the ZS Framework's effectiveness on providers’ confidence, knowledge, and skills. |
CES |
The CES is administered on an ongoing basis and will inform SAMHSA about consumer experiences with entities implementing the ZS Framework, including the components of the model that clinicians implemented with consumers, consumer satisfaction with the services provided, and self-reported outcomes associated with the services received. Not collecting these data, collecting them less frequently, or collecting them with fewer consumers will negatively impact SAMHSA’s ability to report on key outcome measures of the grant program and understand consumers’ experience of ZS participating HCO’s. |
C-SIF |
The C-SIF is administered on an ongoing basis and serves as a recruitment tool for the CES and provides SAMHSA with baseline information about suicide risk and suicide care management plan enrollment within ZS HCOs. This data is integral to understanding whether and to what extent ZS activities lead to improved behavioral health outcomes. Not collecting these data or collecting them with fewer consumers will limit SAMHSA’s ability to assess the impact of grant services on consumers and identify short- and long-term components that benefit individuals of the ZS program. |
The data collection fully complies with the requirements of 5 CFR 1320.5(d) (2).
For the Prevention Strategies Inventory data collection instrument (Attachment B), we request exemption from utilizing the extended SPD (Figure 1). The Prevention Strategies Inventory (PSI) collects information about all the strategies grantees are implementing due to their Zero Suicide funding. For each strategy added to the inventory, there is a series of follow-up questions about the strategy. One of these follow-up questions asks if the strategy targets a specific race/ethnic group and which group.
This question about race/ethnicity is about the population in the grantee community and not one individual person. Therefore, we do not think respondents will be able to respond (without increased burden) to the additional details as shown in Figure 1 of the Federal Notice. We request to use Figure 3 for the race/ethnicity questions on the PSI.
SAMHSA published a notice in the Federal Register on September 11, 2024 (89 FR 73666), soliciting public comment on this study. No public comments were received.
Consultation on the design, instrumentation, and statistical aspects of the evaluation has occurred with individuals outside of SAMHSA. As part of the ZS Health Evaluation, an evaluation advisory Expert Advisory Panel (EAP) established in 2023 and convened in 2024, provided input and guidance on the ZSE design and instruments—including the PSI, BHPS, KII’s, WS, TASP, TUPS, and CES—prior to submission to OMB. EAP representatives included leaders in the field of suicide prevention program implementation, research, and evaluation. In addition, a panel of local evaluators from ZS funded grantees reviewed these instruments and provided feedback. In 2023, the ZSE team reviewed the recent literature and the ZS toolkit prior to tailoring and adjusting the PSI, BHPS, KII, WS, TASP, TUPS, CES, and C-SIF for use with the ZSE.
Finally, the Action Alliance ZS Advisory Group, which provides expert guidance to behavioral health care settings seeking to implement the ZS Initiative, developed the WS as part of the ZS toolkit to help organizations gain a general understanding of their ability to address issues related to suicide.
The ZSE will require participation from health care administrators, clinicians/client-facing staff, and consumers. Remuneration is suggested for respondents not directly affiliated with the grantees’ ZS programs at the time of their participation in appreciation of their time, potential inconvenience and burden of participation, and any related costs (e.g., mobile phone data, compensation for time). We recommend the following remuneration at the time of participation: C-SIF ($5 per participant) CES ($10 per baseline and $15 per follow up), Consumer KII ($30 per response), and TUPS ($10 per response at the 6 and 12 month follow up). Respondents to other data collection activities are primarily staff of the ZS grantees or close affiliates. Therefore, no remuneration is planned for those activities.
Data will be kept private to the extent allowed by law. To ensure the confidentiality of data collected during the ZS Evaluation, and to ensure the protection of human subjects, evaluation data collection protocols and instruments will be reviewed and approved by ICF’s Institutional Review Board (IRB) prior to its collection. This review ensures compliance with the spirit and letter of U.S. Department of Health and Human Services (HHS) regulations governing projects that collect data from human subjects. The ICF IRB holds a Federal wide Assurance (FWA00002349 Exp. October 13, 2025) from the HHS Office for Human Research Protections (OHRP). In addition, the ZSE will apply for a SAMHSA certificate of confidentiality (CC). The CC protects grantees and contractors from legal requests for names or other information that would personally identify participants in the evaluation of a grant, project, or contract.
All protected data will be stored in the ZSDC which will be supported by SAMHSA’s cloud environment and in the manner described in the Data Management and Storage Plan submitted as a section in the Evaluation Plan submitted to and approved by SAMHSA for the ZS Evaluation on April 8, 2024. In addition, ZSDC will facilitate data entry and management for the evaluation.
All ZSE data will be collected electronically. Personal identifying information (PII) is not collected as a part of any data collection activity other than as required to administer surveys or distribute incentives. Specifically, contact information required for survey administration and/or incentive distribution are entered into password-protected databases that are accessible only to a limited number of individuals who require access, such as data analysts and administrative staff who distribute incentives. These individuals have signed privacy, data access, and data use agreements. PII collected to facilitate the administration of surveys will not be stored with survey responses and all datasets will be stripped of any PII prior to use.
Specific procedures to protect the privacy of respondents for activities that require PII for administration and/or incentive distribution are described in Exhibit 5.
Exhibit 5. Procedures to Protect Respondent Privacy
Activity |
Privacy Procedures |
PSI & TASP |
Grantee staff will enter information directly into the ZSDC to complete the PSI and TASP. To access the system, respondents receive an individual username and password to protect their privacy. No PII is requested on these instruments. |
BHPS & WS |
It is necessary to collect respondent contact information to administer and to distribute incentives for the BHPS and WS. Respondent contact information will be limited to agency affiliations, names, and e-mail addresses and will be entered into a password-protected database separate from survey responses. PII required for annual BHPS and WS administrations will be maintained on Aptive’s secure servers. Although PII will not be used in any reports or datasets, reports and datasets may contain the names of agencies/organizations and the information provided about them. As such, respondents will be informed during the consent process that it is possible individuals may be identifiable when reporting results. |
KII |
Key Informant Interviews (KIIs) for the HCO Case Studies will involve a select number of grantees. To ensure privacy, interviewees will be assigned unique identifiers that disconnect their personal identity from their responses. These identifiers, along with any necessary contact information, will be securely stored in a password-protected database, separate from the interview data. PII will be requested during the interviews themselves. Cost Sub-Studies. For the Cost Sub-Studies, it will be necessary to collect limited PII from grantees for effective communication and data collection. This information, likely including names and email addresses, will be entered into a separate, secure database distinct from the data collected regarding the costs of ZS implementation. As in other studies, the PII will be used solely for administrative purposes and will not be included in any public reports or datasets. The identities of individual participants or specific organizational details will not be disclosed in any reports or analysis, maintaining confidentiality. In both the HCO Case Studies and the Cost Sub-Studies, participants will be informed during the consent process about the specific use of their data and the measures in place for privacy protection. This will ensure that all participants are fully aware of how their information will be used and safeguarded, in compliance with the privacy protocols established by ICF’s IRB and relevant regulatory guidelines. |
TUPS |
Healthcare organizational staff who completed the ZS-related training will be invited to complete the survey at the end of each training course as well as the 6th and 12th month mark after the training date. The baseline assessment will ask each participant to provide their contact information and consent to be re-contacted to complete the survey. The contact information that each participant will provide includes their work and personal emails and contact numbers and will only be used to distribute survey invitations at the 6th and 12th month mark after the training date. All survey responses will be kept confidential and maintained in a secure and password-protected database. |
CES |
After enrolling a consumer into a Zero Suicide care management plan, Zero Suicide grant staff will provide consumers with information about the CES via the Consumer Study Interest Form (C-SIF), described below. Consumers who are interested in study participation will complete the C-SIF with their provider, which will be submitted directly to the ZSDC. Consumers will receive a follow-up email, using the contact information provided through the C-SIF, inviting them to review a consent form for the CES and complete the baseline survey. Consenting consumers will receive a similar emailed survey invitation six months after their baseline survey. |
C-SIF |
The C-SIF will be used to guide participant recruitment into the study. The C-SIF provides a brief recruitment script and collects information about the best contact methods for survey invitations, suicide risk, and care status (i.e., suicide care management plan status) for those interested in study participation). The C-SIF will be completed by grantees when individual consumers enroll in a suicide care management plan with their provider organization. Grantees will have the option of incorporating these questions into an intake interview or other appointment by verbally asking each question to participants, or by offering participants to answer questions directly in a self-report format (i.e., via tablet, laptop, or other device) at the close of the interview/appointment. As part of the C-SIF, consumers will complete a consent-to-contact form so that they may receive survey invitations for the 6-month follow-up survey. The consent-to-contact request will ask consumers to provide identifying information (i.e., name, email, phone number) necessary to distribute survey invitations at each timepoint. Contact information will be used only to distribute survey invitations and will be maintained in a secure, password-protected database. Contact data will be stored separately from de-identified CES and C-SIF data. |
The CES includes questions that are potentially sensitive. For example, certain questions request current mental health and health status, the types of behavioral health services received, and health outcomes. These questions are central to SAMHSA’s goal of learning whether/how health systems are implementing the seven elements of the ZS model, consumer experiences with those services, and outcomes related to the services received. Given the sensitive nature of some CES questions, the following procedures are in place:
CES web-based consent form explicitly advises potential respondents about the sensitive nature and content of the data collection protocol, as well as the voluntary nature of all data collection activities. In addition, the CES consent contains contact information for the 988 Suicide & Crisis Lifeline.
CES participants are reminded of their ability to skip questions, pause the survey, or stop the survey throughout the survey, including a dedicated screen that will display if responses indicate an active crisis or moderate/high crisis risk. These prompts and instructions include contact information for the 988 Suicide & Crisis Lifeline.
We will report any unanticipated or negative consequences immediately to the ICF IRB. In these situations, the principal investigator and project director will consult with appropriate clinical professionals to immediately determine if the participant presents a risk to themselves or others and make appropriate referrals.
CES respondents will complete a brief consent-to-contact form after completing the baseline survey to allow distribution of future surveys. Participants will be informed that their contact information will be stored separately from survey responses in a password-protected database, accessible only to authorized project staff, and used only for the purposes of distributing survey invitations. Contact information to the close of the study, or as otherwise required by the ICF IRB.
C-SIF uses a common clinical suicide risk screening tool. This screening data will be collected by grantee or HCO staff and then submitted to the ZSDC through a web-based form. The C-SIF will be completed with grantee or partnering HCO staff during the course of routine clinical care and/or the collection of other grant-required performance data. Grantee/HCO staff will be asked to follow the crisis procedures of their organization if consumer express discomfort or active crisis risk during the course of data collection.
This is a new data collection. Clearance is being requested for three years of data collection for the ZSE. Exhibit 6 below describes the burden and costs associated with ZSE data collection activities. Burden was calculated for 10 ZS grantees from Cohort 4, 25 ZS grantees from Cohort 5, and the 11 potential grantees for cohort 6. Annualized counts of respondents per grantee and total respondents were calculated for all studies, as presented in Exhibits 6 and 7, are derived from these cumulative counts and rounded up. These calculations are derived from the number of grantees that will participate in ZSE activities during each year of the data collection and the activities that the cohort will participate in (Cohort 5 grantees will participate in all aspects of the ZSE data collection activities while Cohort 4 grantees will only participate in the PSI and the TASP). The cost was calculated based on hourly wage rates for appropriate categories presented in the Bureau of Labor Statistics (BLS) Occupational Employment Statistics (OES) May 2022 National Industry-Specific Occupation Employment and Wage Estimates. Exhibit 7 shows an annualized summary of burden hours by respondent type.
Exhibit 6. Estimated Annualized Burden Hours & Costs Across the 3-Year Clearance Period
Type of Respondent |
Form |
Number of Respondents per year |
Responses per Respondent |
Total Number of Responses |
Burden per Response (hours) |
Annual Burden (hours) |
Hourly Wage Rate |
Total Cost |
Project Evaluator 1 |
PSI |
40 |
4 |
160 |
1 |
160 |
$61.53 |
$9,845 |
Grantee/HCO administrator 2 |
BHPS |
47 |
1 |
47 |
0.5 |
24 |
$61.53 |
$1,477 |
Grantee/HCO administrator 2 |
KII-Case Studies |
7 |
1 |
7 |
1 |
7 |
$61.53 |
$431 |
HCO Staff 3 |
KII-Case Studies |
27 |
1 |
27 |
1 |
27 |
$26.81 |
$724 |
Grantee/HCO administrator 2 |
KII-Cost Sub studies |
2 |
1 |
2 |
1 |
2 |
$61.53 |
$123 |
HCO Staff 3 |
WFS |
9,400 |
1 |
9,400 |
0.25 |
2,350 |
$26.81 |
$63,004 |
Project Evaluator1 |
TASP |
40 |
10 |
400 |
0.25 |
100 |
$36.67 |
$3,667 |
HCO Staff 3 |
TUPS-Baseline |
3,334 |
1 |
3,334 |
0.25 |
834 |
$26.81 |
$22,360 |
HCO Staff 3 |
TUPS-6 month |
252 |
1 |
252 |
0.5 |
126 |
$26.81 |
$3,378 |
HCO Staff 3 |
TUPS-12 month |
189 |
1 |
189 |
0.5 |
95 |
$26.81 |
$2,547 |
Clinicians |
C-SIF |
180 |
8.3 |
1,494 |
0.25 |
374 |
$57.21 |
$21,397 |
Consumer |
CES - Baseline |
1,128 |
1 |
1,128 |
0.4 |
451 |
$7.25 |
$3,270
|
Consumer |
CES – 6-month |
843 |
1 |
843 |
0.4 |
337 |
7.25 |
$2,443
|
Consumer |
C-KII |
15 |
1 |
15 |
1 |
15 |
7.25 |
$109 |
Total |
15,504 |
|
17,298 |
|
4,902 |
|
$134,773 |
|
Abbreviation: HCO= Healthcare Organization
1BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Survey Researchers (code 19-3022); https://www.bls.gov/oes/cuSee trrent/naics5_541720.htm
2BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Medical and Health Services Managers (code 11-9111); https://www.bls.gov/oes/current/oes119111.htm
3BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Community and Social Service Occupations (code 29-1000); https://www.bls.gov/oes/current/oes210000.htm
4BLS OES May 2022 National Industry-Specific Occupation Employment and Wage Estimates average annual salary for Health Diagnosing and Treating Practitioners (code 29-1000); https://www.bls.gov/oes/current/oes_nat.htm#29-0000
5BLS OES May 2022 Characteristics of minimum wage workers, 2022; https://www.bls.gov/opub/reports/minimum-wage/2022/home.htm#:~:text=In%202022%2C%2078.7%20million%20workers,wage%20of%20%247.25%20per%20hour.
Exhibit 7. Annualized Summary Burden by Respondent Type
Respondents |
Number of Respondents |
Responses/ Respondent |
Total Responses |
Hours per Response |
Total Annualized Hour Burden |
Grantee/HCO administrator |
56 |
3 |
168 |
0.59 |
99 |
HCO Staff |
13,202 |
5 |
66,010 |
0.26 |
17,163 |
Project Evaluator |
80 |
14 |
1120 |
0.46 |
515 |
Consumer |
1,986 |
3 |
5958 |
0.40 |
2,383 |
Clinicians/ Providers |
180 |
8 |
1,440 |
0.25 |
360 |
Total |
|
|
74,696 |
|
20,520 |
Grantees are collecting most of the required data elements as part of their normal ZS Program grant operations. Grantees will maintain this information for their own program planning, quality improvement, and reporting purposes. Therefore, there are no additional capital or start-up costs associated with the ZSE. There will be some additional burden on record keepers to provide potential respondent lists for data collection activities. However, these operation costs will be minimal. As part of the overall cooperative agreement award, each grantee has received funding to hire an evaluator and for costs related to carrying out the requirements of the ZSE. Therefore, no cost burden is imposed on the grantee by this additional effort.
CMHS has planned and allocated resources for the management, processing, and use of the collected information in a manner that shall enhance its utility to agencies and the public. Including the Federal contribution to local grantee evaluation efforts, the contract with Aptive Resources, and government staff to oversee the evaluation, the annualized cost to the Government is estimated at $3,023,033. These costs are described below.
Each grantee is expected to fund an evaluator to conduct the self-evaluation and to satisfy the requirements of the ZSE. It is estimated that ZSE participation will require 0.25 full-time equivalent (FTE) to collect information, enter information into the Web-based data collection and management system, and conduct analyses at the local level. Assuming an annual evaluator salary of $66,440 based on the BLS May 2022 data for the Survey Researcher category, 25 percent effort for one grantee would be $16,610. With 56 grantees expected to participate in the ZSE, the total grantee cost would be $931,060.
A contract has been awarded to Aptive Resources for evaluation of the Zero Suicide Program. The current evaluation contract with SAMHSA is funded to conduct the ZSE with 35 grantees over the next 5 years with a value of $8,171,243. The estimated average annual cost of the contract will be $1,634,249. An additional $568,931 estimated average per year will need to be awarded to include an additional 11 grantees expected for Cohort 6 in addition to approximately $20,404 to provide incentives to this cohort as part of this evaluation. This amount covers expenses related to developing and monitoring the ZSE, including but not limited to, developing the evaluation design and instrumentation, developing TTA resources (e.g., manuals, training materials, etc.), conducting in-person and/or virtual TTA, monitoring of grantees, traveling to grantee sites and relevant meetings, and analyzing and disseminating data. In addition, these funds will support the enhancement of an existing Web-based data collection and management system for the ZSE and fund staff support for data collection.
It is estimated that CMHS will allocate 0.30 of a full-time equivalent each year for Government oversight of the evaluation. Assuming an annual salary of $117,962 for a GS-13 step 1 payscale, these Government costs will be $35,389 per year.
This is a new data collection.
The time schedule for implementing the cross-site evaluation is summarized in Exhibit 8. SAMHSA is requesting a three-year clearance for this project.
Exhibit 8. Time Schedule
Activities |
Timeframe |
Evaluation Year 1 09/11/2024-9/10/25 |
|
Evaluation Year 2 09/11/2025-9/10/26 |
|
Evaluation Year 3 09/11/2026-9/10/27 |
|
Evaluation Year 4 09/11/2027-9/10/28 [not included in OMB coverage or burden estimates] |
|
Team Aptive prioritizes the dissemination of the evaluation findings to SAMHSA and key stakeholders, including grantees, and the broader field of behavioral health for key insights that shape consumer service delivery. Dissemination of findings includes but is not limited to development of publications, conference abstracts, and conference panel and/or poster presentations. We collaborate with SAMHSA to develop a dissemination strategy that meets the immediate needs of the agency, stakeholders, and consumers and is also responsive to the evaluation findings. To ensure our dissemination products are accessible and meaningful to diverse audiences, we ensure accessible communications (including plain language text and 508 compliance), pilot early versions of the materials with our expert panel and a group of consumers when possible and promote equity of access through multiple modes of communication (electronic and print versions). This requires the creation of materials in a variety of mediums for ease of consumer use, such as infographics, briefings, and presentations in addition to reports and publications, an endeavor Team Aptive has successfully executed in large-scale evaluations. Team Aptive collaborates with the COR to follow the appropriate clearance channels prior to publication of materials. We follow SAMHSA’s Office of Communication policies to ensure that we are exceeding SAMHSA’s communication standards throughout the development and review process.
In the event that SAMHSA wants to publish findings from the ZSE, potential manuscript topics will include findings related to priority areas, such as how health systems implement the ZS model; clinician/client-facing staff knowledge, practices, and confidence to implement the ZS model; consumer experiences with ZS services; and the effectiveness of the ZS model at reducing suicide attempts and deaths among consumers in the health care system.
All publications will be submitted to the Contracting Office Representative (COR) in draft form for review and approval prior to submission to the selected journal. Examples of journals that will be considered as vehicles for publication include the following:
American Journal of Public Health
American Psychologist
American Journal of Diseases of Children
Child Development
Crisis
Evaluation Review
Evaluation Quarterly
Journal of the American Academy of Child and Adolescent Psychology
Journal of Applied Development Psychology
Journal of Child and Family Studies
Journal of Clinical Child and Adolescent Psychology
Journal of Consulting and Clinical Psychology
Journal of Health and Social Behavior
Journal of Mental Health Administration
Psychological Reports
Social Services Review
Suicide and Life-Threatening Behavior
Data collected through the ZSE will be analyzed to address key evaluation questions and related sub-questions. Analysis plans for each study are described below.
Systems Change Study
The Systems Change Study documents, provides insights, and assesses the specific components of the ZS model as implemented at HCOs through a combination of qualitative and quantitative elements. Analysis for this study will consist of a combination of descriptive statistics, tests to measure change over time, and qualitative techniques to ensure a comprehensive understanding of the collected data.
Our adoption and fidelity measures will be answered through our two primary instruments, the PSI and the BHPS. Analysis will consist of descriptive statistics to understand what grantees are implementing, how the HCOs are being affected by the implementation of ZS in their organization, and if grantees vary in their implementation as a function of HCO or community characteristics. Longitudinally, this data will be analyzed using techniques, such as Autoregressive Integrated Moving Average (ARIMA) or mixed models, to understand changes and variances over time. The BHPS scoring rubric will assess HCOs across 18 domains, such as staff, suicide risk assessment, and safety planning, within the seven components of the ZS model. This scoring allows us to standardize the assessment and evaluate the depth of adoption in suicide prevention practices. In addition to these quantitative approaches, further analysis relies on thematic analysis of our KIIs that are part of our case studies. Building off the work of Porter et al. (2022) and Ross et al. (2021), we will analyze perceptions of ZS adoption. The thematic analysis, guided by coding rubrics developed for Consolidated Framework for Implementation Research (CFIR) and Consolidated Framework for Collaboration Research (CFCR), will clarify how organizational features influence adoption and fidelity.
The services coverage and cost evaluation questions will be addressed using a combination of secondary and primary data analyzed using qualitative and quantitative analytic approaches. To understand service coverage, we will utilize the ACS and CHRR data to analyze population-level trends in the areas served. This will highlight discrepancies, if any, between the characteristics of clients served and the general population. Based on PSI questions, we will identify grantees that utilize social media and other technologies as part of their ZS programs. Those grantees that utilize social media and are part of our case studies will be asked to share social media metrics. The examination of these metrics will help us assess how technology influences the reach of ZS programs.
The cost analysis for our costing case studies will include a quantification cost beyond grant activities, like training expenses and system integration costs. Grantees in this case study will be asked to share cost/budgetary information. This data will be utilized, along with outcomes data from the Consumer Experience Study, in an activity-based cost model that will calculate cost per outcome, providing a clear picture of financial efficiency in service delivery. As much as possible, insights from all instruments and evaluation questions will be combined to paint a holistic picture that will allow us to understand not only statistical trends, but also the underlying narratives and context.
Workforce Study
The Workforce Study is quantitative, aimed at evaluating the effectiveness of the ZS training programs and anchored by longitudinal mixed effects regression models. These models are particularly crucial in addressing training sustainment, where we will compare responses across various demographic factors, roles, and settings, utilizing data from the TUPS. Mixed models will allow us to handle the complexities and variations in the data, such as differences in organizational cultures and training environments. This approach will enable a nuanced comparison across different groups and account for intra- and inter-organizational variabilities to understand differences in responses among different groups within the workforce (type of role in the organizations, demographics etc.)., thereby providing insights for tailoring future training programs.
Training utilization and sustainment will be addressed utilizing data from our TUPS series of instruments. To answer these questions, which concentrate on the extent of training utilization in practice and the sustainability of acquired skills and knowledge, we will again employ longitudinal mixed effects regression models allowing examination of changes in provider practices and skill levels at multiple post-training intervals by comparing baseline data with follow-up data to measure training retention and application over time. Additionally, regression analyses will be employed to identify predictors of successful training utilization and skill retention, considering variables like training intensity, provider background, and practice settings. This integrated statistical approach will robustly assess both the immediate and long-term impacts of the ZS training, guiding future enhancements in training methods and content.
Building on the comprehensive quantitative analysis framework, the Workforce Study will also incorporate a detailed Longitudinal Analysis. This aspect of the study will utilize data from the TUPS to track and analyze changes over time in knowledge retention and skill application among healthcare providers. The use of mixed effects regression models in this longitudinal analysis allows examination of variations within and across different organizational contexts enabling us to differentiate the impact of the ZS training from other influential factors, such as organizational culture, policies, and staff composition. By employing these models, we can effectively chart both individual and collective growth trajectories, gaining insights into how knowledge and skills evolve post-training. This approach offers flexibility in managing diverse data structures and enhances the generalizability of our findings.
Consumer Experience Study
We will use descriptive statistics to summarize consumer mental health outcomes, service engagement, and satisfaction with care. This includes 1) the number and characteristics (e.g., demographics, history of suicidal behavior) of participating consumers; 2) the number and proportion of consumers who receive suicide-specific elements of care, including risk assessment, safety planning, clinical care, follow-up support, and care transitions; 3) types of services received by consumers, including treatment adherence, common discussion topics and other supports received from provider; 4) crisis experiences within the past 12 months; 5) general satisfaction with and results of services received, including treatment alliance with provider and most helpful elements of care; and 6) the outcomes of care, including changes in depression symptoms, anxiety symptoms, and suicide risk. We will also provide summary statistics (e.g., mean, standard deviation, change scores between time points) to help further characterize the experiences of consumers as they navigate ZS organizations.
Longitudinal client-level outcome measures, including anxiety symptom scores, depression symptom scores, and suicide risk assessment scores will be assessed through a series of linear mixed effects models. Linear mixed effects models are particularly useful in this context as they address data non-independence (e.g., hierarchical structures), preserve statistical power, flexibly work with missing data, and allow for the addition of random effects (Krueger & Tian, 2004; Matuschek et al., 2017). These models will account for both between-consumer and within-consumer residual variances, and will explore the association between outcomes (e.g., clinical score change, readiness to change, self-efficacy to avoid suicide), services received, organizational implementation characteristics (obtained through the BHPS), and perceptions of care. We will assess and address the impact of potential covariates, as needed, during analysis. We will also employ a variety of other inferential analyses (i.e., correlation, t-tests, ANOVA) to provide nuanced answers to key evaluation questions. For example, we will explore provider utilization of training materials as a predictor of client outcomes through a linear mixed model, while a series of correlations will highlight differences between trainings offered to providers and those recognized by consumers as part of their care. Data analysis will also be conducted to understand experiences related to health equity and outcome disparities based on consumer race, ethnicity, gender, sexual orientation, and perceived cultural competency of HCO staff. Data from the Consumer Experience Study will also be used in the quasi-experimental comparison study described in the Impact Study section.
Impact Study
Because we cannot randomly assign consumers identified at risk for suicide to services received from either a ZS or non-ZS provider, all analyses will involve establishing a control condition that shares key characteristics with the ZS sample, including demographic characteristics, suicide-related diagnoses, previous suicide attempts, and use of mental health services. The establishment of the control group will allow the Team to assess the relationship between implementation of ZS Program activities and consumer outcomes from a quasi-experimental causal perspective.
Using the ‘potential outcome’ framework (Rubin, 1974), we define the impact of the ZS intervention on the units exposed as the difference between the values of the outcomes that were observed following the start of the intervention and the values that would have been observed in the absence of the intervention. To estimate these ‘counterfactual’ outcomes, we take advantage of the available ‘control’ information, particularly values of the outcome from units and periods that were not exposed to the intervention. In addition to the history of the outcome, we consider other characteristics that may be predictive of the outcome.
Individual-level Morbidity Impact: The purpose of the individual-level morbidity impact analysis is to compare suicide attempts among consumers enrolled in Medicaid following contact with a ZS provider with the suicide outcomes these individuals would have experienced had they contacted a non-ZS provider instead. To estimate what would have happened to individuals served by a ZS provider had they been served elsewhere, we will use suicidal behavior among individuals served by other providers who are otherwise comparable in terms of demographics and service use. We will identify these comparable individuals using propensity score-based techniques, including propensity score matching (PSM) (Rosenbaum & Rubin, 1983).
We will use a Medicaid claims-based retrospective cohort design to compare consumer outcomes, particularly nonfatal suicide attempts requiring medical attention, the year following a contact with a ZS-participating provider with the outcomes of consumers, under similar circumstances, following contact with non-ZS providers. We will compare rates of suicide attempts before and after implementation of the ZS Program to determine whether there is a larger decrease in suicide attempts and deaths among patients served by ZS-participating providers. We will estimate the impact of the ZS Program at a provider and at a county level. The approach will draw on a comparison of suicidal deaths in non-ZS counties. The analysis team will compile historical data on the suicide attempts and deaths, as well as demographics and other county-level characteristics in each ZS County and all other potential comparison counties. To address the question of the impact of the program on the specific subgroups, we will consider spatiotemporal models for impact evaluation. To ensure that the analysis results are attributable directly to the program, we will conduct several procedures to assess the robustness of our approach, including placebo studies and different definitions of the intervention. We will conduct in-time and/or across-site placebo checks. To address the question of whether specific elements of the ZS Framework have a greater influence on outcomes than other elements, we will use CNA to identify the causal paths that lead from the presence or absence of combinations of implementation conditions (e.g., screening, assessment and referral training, partnerships) to outcomes (e.g., self-reported suicidal thoughts, plans, and attempts; inpatient hospitalization and ED discharges; and mortality) thereby establishing the necessary and sufficient implementation conditions that lead to positive impact.
Provider-level Morbidity and Mortality Impact. We will estimate the impact of the Zero Suicide Program at a provider level. Medical claims data can be used to create suicidal attempt rates at the provider level. Using this compiled information, it is possible to construct a control group that closely resembles the trajectory of the outcome variables before the implementation of Zero Suicide. To the extent that the control groups are successful in closely reproducing the trajectory of the outcome variable before the intervention, differences after the initiative can be directly interpreted as estimates of the impact of the Zero Suicide Program.
We are currently considering three different approaches to make this estimation. Synthetic difference-in-difference (SDID, Arkhangelsky et al., 2021) builds on the classical difference-in- difference approach but attempts to address some of its limitations by incorporating weights that make control units more comparable to treated units and pre-treatment periods more comparable to post-treatment periods. Synthetic control for staggered adoption (SC, Ben-Michael et al., 2021), on the other hand, builds on the synthetic control methodology (Abadie et al., 2003, 2010, 2015), originally developed for a single treated unit, and extends the methodology for multiple units as well as multiple starting periods. While the two procedures start in quite different places, they end up being comparable in many respects, including the use of weights to make units more comparable, and the attempt to avoid making assumptions about the underlying data generation mechanism. Causal Bayesian Additive Regression Trees (BART, Hahn et al., 2020; Hill, 2011), relies on an ensemble of regression trees to model both the response surface and the assignment mechanism. Prior distributions on the branching process and the parameters of the terminal nodes prevent overfitting and support full Bayesian inference. Rather than avoiding modelling assumption all together, this approach attempts to build very flexible models, taking advantage of machine learning tools.
All data collection instruments will display the expiration date of OMB approval.
This collection of information involves no exceptions to the Certification for Paperwork Reduction Act Submissions.
SS
Part A ZS Evaluation
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2024-11-23 |