988 Evaluation Part B_03202025_Clean

SAMHSA 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation

Supporting Statement Part B

2. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

Exhibit 1 displays the expected number of respondents for each data collection activity across the three-year OMB period. The number of participating organizations/respondents was calculated based on the reported number of crisis service agencies serving both children and adults presented in the NRI (2023) State Mental Health Agency Profiles, including crisis contact centers (n = 544), mobile crisis teams (n = 1,287), and crisis stabilization facilities (n = 237). Estimates represent the number of these respondents who will be asked to participate based on the sampling criteria for each data collection activity (i.e., the use of a recruited sample of agencies for client-level components).

Exhibit 1. Total Annualized Number of Respondents by Data Collection Activity (Estimated)

Instrument	Estimated Participating Organizations	Estimated Respondents per Organization	Total Respondents
SIS	73	1	73
CCPS	1,034	1	1,034
KII-CS	4	8.75	35
KII-CS-CSS	1.25	10.40	13
CCDF	517	11.61	6,000
CCDF Parent Supplement	517	3.02	1,560
CES – Baseline	517	11.61	6,000
CES – 3-month	517	2.90	1,500
CES – 6-month	517	0.73	375
CES – 12-month	517	0.18	94
C-KII-DC	30	1	30
C-KII-TPC	10	1	10

Due to the novelty of the 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation, using these estimated numbers of respondents will capture variation across types of grantees and other participating organizations to help establish benchmarks for progress. The respondent universe and sampling methods for all data collection activities are described in Exhibit 2.

Exhibit 2. Universe and Sampling Methods for Instruments

Instrument	Universe and Sampling Method
System Implementation Survey (SIS)	The SIS will collect data on crisis system structures, partnerships supporting 988 Lifeline and Crisis Services, collaboration levels between partners, funding sources, Tribal community engagement efforts, and implementation barriers and facilitators. The evaluation team will designate a representative from each state, territory, or Tribal community receiving funding through SAMHSA-funded 988 State/Territory, 988 Tribal nation, or other relevant grants to complete the SIS on an annual basis.
Crisis Continuum Provider Survey (CCPS)	Behavioral health providers from BHCSC organizations (crisis centers, mobile crisis teams, or crisis stabilization facilities) and those serving Tribal communities will be invited to participate in the CCPS on a biennial basis. Respondents will be identified through the SIS, SAMHSA Treatment Locator, and mobile crisis system reports required of 988 state/territory grantees. One representative from each participating organization will respond to the CCPS and receive a financial incentive to reduce the burden.
Key Informant Interviews: Case Study (KII-CS)	KIIs will provide an in-depth picture of 988 successes, barriers, funding, and collaboration efforts at the national, state/territory, and provider levels. The team will employ stratified purposive sampling upon selecting case study locations. Specifically, we will categorize potential case study sites based on key criteria, such as geographic location (urban, rural, tribal) and client demographics . From this stratification, we will select a maximum of 10 interviews, either virtual or in-person, with incentives provided for each participant to reduce the burden. Annually, the team will recruit up to 5 staff members from the 988 Lifeline Administrator, totaling 20 participants over 4 years. This group will comprise 1-2 members from various teams, including organizational leadership, the 988 data analytics team, the standards, training, and practices team, and the partnership and other relevant program teams. Also, the team will recruit up to 10 988 State/Tribal grantees from 10 distinct states annually, amounting to over 120 participants over four years. These participants will include 1-2 representatives from the state or territory agency overseeing crisis services, 1-2 from the 911 system, 2 from crisis contact centers, 2 from the mobile crisis system, 1-2 from crisis stabilization or similar providers, and 1-2 from outpatient behavioral health or similar providers. At least 2 Tribal locations will be included to gather sufficient data providing insights into the integration of local resources, and federal support in crisis care by Tribal communities.
Key Informant Interviews: Case Study-Cost Case sub-Study (KII-CS-CSS)	A subset of case studies will focus on the costs of 988 implementation, which will support nuanced understanding of costs incurred beyond the scope of SAMHSA grant activities or funding. Up to 5 states, territories, or Tribal nations, selected from the initial case studies, will undergo additional interviews. This selection will ensure the inclusion of at least one urban and one rural setting, facilitating a comprehensive understanding of the challenges and successes experienced by organizations in varying geographical and demographic contexts.
Client Contact Disposition Form (CCDF)	Crisis staff will complete a CCDF for clients who agree to participate at the close of contact or during a follow-up contact with the same crisis services provider. The CCDF will capture information about the crisis contact and contact information that will be used to facilitate CES distribution. Those under 18 will also be asked to provide contact information for their parent or guardian, who will receive the Client Contact Disposition Form: Parent Supplement.
Client Contact Disposition Form: Parent Supplement (CCDF-PS)	Parents and guardians of all potential CES participants who 1) complete the CCDF indicating their (the youth’s) interest in CES participation and 2) are under the age of 18 will complete the CCDF-PS. This supplement will be emailed to parents/caregivers and contain the CES consent form for youth and a small subset of CCDF contact information questions.
Client Experience Survey (CES)	The CES will be administered to individuals engaging with 988 Lifeline and the BHCSC when they enroll in services and 3, 6, and 12 months after they enroll in services. The CES will be used to assess behavioral health outcomes, services engagement, perceptions of care, and pathways through care. To reduce the burden, the survey will be administered through the CSPDC. Power calculations for the CES indicate a comparison of 3 groups (crisis contact support, mobile crisis service participants, and crisis stabilization services participants) over four waves of data collection (enrollment, 3, 6, and 12-months) using a mixed factorial analysis of variance assuming a medium effect size (Cohen’s f = 0.25), a non-sphericity correction of 0.75, alpha = 0.05, and power = 0.80 would require a total sample size of 267 (89 per group) after attrition. As crisis contact centers typically serve a higher volume of clients than mobile crisis or crisis stabilization services, we will aim to recruit enough participants to yield an additional 89 responses from crisis contact centers at 12-month follow-up. Assuming 75% attrition between waves based on previous studies in similar populations (Gould et al., 2017; Witte, et al., 2010), that would require an enrolled sample baseline sample of 24,000, or 12,000 each from 988 Lifeline contact centers and community crisis response providers (24,000 * 0.75 = 18,000, 24,000-18,000 = 6,000; 6,000 * 0.75 = 4,500, 6,000-4,500 = 1,500; 1,500 * 0.75 = 1,125, 1,500-1,125 = 375). Thus, our proposed sampling approach will aim to ensure at least 356 responses at the 12-month follow-up to fully address evaluation questions, after accounting for up to 75% attrition between waves based on previous studies in similar populations. Incentives will be distributed to clients participating in individual-level data collection.
Client-Key Informant Interviews: Direct Contacts (C-KII-DC)	A series of C-KII-DCs will be carried out to gather insights on client experiences with navigating the 988 Lifeline and BHCSC, satisfaction with services and providers (such as addressing concerns), and factors influencing service engagement. Participants for C-KII-DCs will be recruited via the CCDF. Interview participants will be stratified by presenting concern (e.g., risks of suicide, violence toward others, and/or overdose) and point of entry into the BHCSC (e.g., 988 chat, text, or call, or other BHCSC service) and then randomly selected from the pool of those indicating interest in participation.
Client Key Informant Interviews: Third Party Contacts (C-KII-TPC)	A series of C-KII-TPCs will be carried out with third-party contacts to gather insights on experiences navigating the 988 Lifeline and BHCSC, satisfaction with services and providers, and any known information about outcomes for the individual at-risk. Participants for C-KII-TPCs will be recruited via the CCDF. Interview participants will be stratified by presenting concern (e.g., risks of suicide, violence toward others, and/or overdose) and point of entry into the BHCSC (e.g., 988 chat, text, or call, or other BHCSC service) and then randomly selected from the pool of those indicating interest in participation.

2. Information Collection Procedures

Information collection procedures for the 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation instruments are described in Exhibit 3.

Exhibit 3. Information Collection Procedures

Instrument	Procedures
SIS	Beginning in 2025, the SIS will be administered annually through the CSPDC to states, territories, and Tribal nations receiving funding through SAMHSA-funded 988 State/Territory or 988 Tribal grants. Respondents will receive an invitation to complete the SIS via CSPDC notification. Respondents will create their own username and password on the CSPDC to access and submit the SIS. Additionally, the SIS will gather contact details (such as name, contact number, and email) of crisis organizations constituting the BHCSC within the state, territory, or Tribe. These details will facilitate the distribution of the CCPS.
CCPS	Providers identified through the SIS will be invited to complete the CCPS in the 2nd (2026) and 4th (2028) years of the evaluation. TTA will conduct outreach to the identified CCPS participants to verify information and invite respondents to complete the instruments. After obtaining consent to participate, respondents will create an account on the CSPDC to securely access and submit the CCPS.
KII-CS	Potential interviewees will be invited to participate in the KIIs via notification through the CSPDC. Following agreement, technical assistance liaisons will contact prospective participants via email and phone to arrange interviews. Interviews will be conducted via web conferencing or in-person. Before the interview starts, participants will verbally consent to participate. The expected duration of the interview will be 30-45 minutes.
KII-CS-CSS	The procedures for the cost sub-studies are similar to the case study procedures. In total, the qualitative interviews for the evaluation will involve many participants across various groups and settings, which will ensure a thorough data collection process
CCDF	Crisis staff from participating agencies will recruit clients aged 13 and older to participate in this study during their index contact with a 988 Lifeline crisis contact center (via call, chat, or text), mobile crisis service, or crisis stabilization service. Clients will be asked about their interest in participating at the contact's close to ensure that counselors can establish rapport and effectively intervene in crisis events. If a client agrees to participate, crisis staff will complete a CCDF for that client, which will be used to distribute the CES.  The CCDF will ask crisis counselors (or other providers from a participating organization) to provide contact information for clients interested in study participation and information about the crisis services provided to the client. The questions will include the client’s presenting concerns, contact disposition, characteristics of the contact (e.g., risk assessment, suicide attempt in progress, emergency rescue, mobile crisis intervention, referrals received), along with contact information that the client has agreed to let the study team use for recruitment efforts. Those under 18 will be asked to provide contact information for their parent or guardian, who will receive the Client Contact Disposition Form: Parent Supplement. These forms will be submitted through the CSPDC. Data will only be submitted for clients who have provided consent to share this information as part of their interest in study participation.
CCDF-PS	Parents and guardians of all potential CES participants who 1) complete the CCDF indicating their (the youth’s) interest in CES participation and 2) are under the age of 18 will complete the CCDF-PS. This supplement will be emailed to parents/caregivers and contain the CES consent form for youth and a small subset of CCDF contact information questions. Like the CCDF, these forms will be submitted to the CSPDC. The CCDF-PS must be completed, and parental consent obtained, before potential participants under the age of 18 receive the CES.
CES	This survey will be administered to individuals aged 13 years and older who engage with 988 Lifeline and the BHCSC and have completed the CCDF. Participants will provide consent/assent immediately prior to completing the survey. One week after their index contact with a participating 988 Lifeline center and/or the BHCSC, clients will receive a survey invitation via email (or text message, in alignment with contact preference indicated on the CCDF) to complete a baseline CES through the CSPDC. Participants will receive similar survey invitations 3, 6, and 12 months after completing their baseline CES. Participants will receive up to 4 reminders to complete each survey over a 4-week period.
C-KII-DC	C-KII-DC participants will include those that indicate interest in the CCDF. Technical assistance liaisons will contact prospective participants via their preferred contact method to arrange interviews. Interviews will be conducted through online tele-video conference software (e.g., Microsoft Teams, Zoom), which allows participants to join through phone or web application. Before the interview begins, participants will verbally consent to participate. The expected duration of the interview will be 45-60 minutes.
C-KII-TPC	The procedures for the third-party contact KIIs are similar to the direct contact KII procedures. In total, the qualitative interviews for the evaluation will involve many participants across various groups and settings, which will ensure a thorough data collection process. C-KIIs will further provide a holistic picture of client outcomes during and following crisis contacts, including client and third-party experiences navigating the crisis care continuum.

3. Methods to Maximize Response Rates

988 Suicide & Crisis Lifeline Contact Centers and BHCSC providers will be expected and encouraged to participate in 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation activities, regardless of their SAMHSA funding status. Participation will include completing required instruments and assisting with respondent identification and recruitment. The team will implement measures to minimize the burden on grantees and client contacts to ensure the efficiency of grantee participation, including a web-based system for data collection and submission, as well as provide training and technical assistance for instrument completion and participant recruitment.

The evaluation team will provide training and technical assistance to maximize response rates for the other data collection activities by hosting web trainings, distributing procedural manuals, and conducting onsite training visits as appropriate. Specific methods to increase the response rates are provided in Exhibit 4.

Exhibit 4. Methods to Maximize Response Rates

Instrument	Methods to Maximize Response Rates
SIS	Participating organization staff will serve as respondents to the SIS. To minimize the burden on local programs and to ensure timely completion of the SIS, staff will complete this instrument on the CSPDC, a web-based data collection system, and will have training and technical assistance (TTA) to complete the activity.
CCPS	BHCSC providers and those serving Tribal communities will serve as respondents to the CCPS. Efforts to maximize response rates to the CCPS will include using a web-based data collection system for administration, providing TTA for completing the survey, and offering incentives. The evaluation team has earmarked $20 gift cards distributed to up to 1,000 participants total, or 500 per survey administration.
KII-CS & KII-CS-CSS	The evaluation team will provide grantees with TTA, preparing them with the tools needed to identify potential interview respondents. Additionally, TTA will help to coordinate and arrange interviews with the participants and communicate any necessary logistical information.
CCDF	The CCDF is a study recruitment tool that will be completed by crisis counselors and potential participants during a crisis contact. Crisis agency staff will briefly describe the CES and C-KII to potential participants and collect preferred contact information contact (text, email) from those who express interest. Those under 18 will be asked to provide contact information for their parent or guardian, who will receive the Client Contact Disposition Form: Parent Supplement. The team will provide grantees with evaluation TTA, preparing them with the tools needed to recruit participants through the CCDF.
CCDF-PS	This supplement contains the CES consent form for youth and a small subset of CCDF contact information questions that will be completed via email. Reminder emails will be sent to parents to prompt their completion of this form, which is necessary before those under age 18 receive the CES.
CES	To offset any costs associated with their support of data collection (e.g., staff time), a selection of crisis contact centers, mobile crisis, and crisis stabilization providers will receive a stipend of $3,000 as an incentive/cost offset for participation. The evaluation team will work with crisis service agencies to recruit clients for the CES via the CCDF. This support will include providing grantees with TTA around patient recruitment. Respondents will complete the CES online through the CSPDC at the time of their choosing. In appreciation of their time and participation, CES respondents will receive up to $80 total for completing all 4 surveys (i.e., baseline, 3-month, 6-month, and 12-month), or $20 for each survey. This will be electronically transmitted to the respondents as a card after completion of each survey.
C-KII-DC & C-KII-TPC	TTA will help to coordinate and arrange interviews with the participants and communicate any necessary logistical information. Participants will receive a $50 gift card after completion of the interview in appreciation of their time and participation.

4. Tests of Procedures

As new measures were developed, standard instrument development procedures include a review of the literature, item development, and content review by individuals from SAMHSA and other experts. Drafts of the instruments were developed and reviewed by evaluation team members, survey methodologists, representatives from SAMHSA, and content experts in suicide prevention. More specifically, we conducted the following tests of procedures:

• In 2024, the 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation Expert Advisory Panel reviewed all proposed instruments and provided feedback. Team Aptive incorporated proposed administration protocol and instrument content improvements prior to submitting the materials for OMB approval.

• In 2024, Team Aptive held multiple 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation Local Evaluator meetings with providers and staff along the BHCSC and received feedback on the evaluation studies.

• To develop the CES, the team conducted a literature review to pinpoint existing surveys and pertinent elements evaluating consumer perspectives on community mental health systems and behavioral health outcomes. The CES comprises items identified from this review, encompassing measures evaluating satisfaction with care, suicide risk, readiness to change, and recovery from suicide, all of which have established reliability and validity among mental health consumers.

• To enhance question accuracy and determine administration time, instruments underwent pilot testing. Team Aptive used a convenience sampling approach to recruit participants directly through community-based organizations and other partnerships to ensure that pilot participants reflected the intended respondent pool for each instrument. 9 individuals were invited to participate in pilot testing for each instrument. A total of 5 people participated in pilot testing for the SIS, while a further 4 completed the CCPS; 4 individuals with lived experience of behavioral health crisis completed pilot testing for the CES. Pilot-tested surveys were administered electronically via web-based survey software. Participants were asked to review the instrument that best fit their role in the crisis system and provide feedback on the clarity and conciseness of instructions/survey items, as well as the time required to respond. Participants were also asked to identify any spelling or grammar errors, confusing items, or contradictory response choices. Estimates of time to complete the SIS ranged from 10 to 60 minutes, likely reflecting differences in time needed to gather relevant data before beginning the survey. Estimates for the CCPS, which has a variable length depending on how many crisis services an agency provides, were similarly different and ranged from 35 to 60 minutes. Completion time estimates for the CES ranged between 20 and 60 minutes, with an average of approximately 40 minutes. Changes were made to the instruments in response to feedback about instructions, questions, response options, and time to complete, including reductions that shortened the CES by approximately 40 items. Burden estimates included in this OMB package reflect the final, post-pilot instruments that are available for review in attachments B-J.

5. Statistical Consultants

Team Aptive has full responsibility for the development of the overall statistical design and assumes oversight responsibility for data collection and analysis. Training, technical assistance, and monitoring of data collection will be provided by the 988 Suicide & Crisis Lifeline and Crisis Services Program Evaluation team. The individuals responsible for overseeing data collection and analysis are:

Christine M. Walrath, PhD, Principal Investigator

ICF

630 3rd Avenue 11th floor

New York, NY 10017

Phone: (212) 941-5555

Email: christine.walrath@icf.com

Taylor Moore, PhD

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: (765) 412-2459

Email: taylor.moore@icf.com

The following individuals will serve as statistical consultants to this project:

Bob Stephens, PhD

ICF

2635 Century Parkway Suite 1000

Atlanta GA 30345

Phone: 404-320-4494

E-mail: bob.stephens@icf.com

Lucas Godoy-Garraza, MA

Los Nardos 444 PH5

La Paz, Bolivia

Email: Lucas.GodoyGarraza@icf.com

The agency staff person responsible for receiving and approving contract deliverables is:

Oumou K Barry

Contracting Officer Representative

Office of the Assistant Secretary for Mental Health and Substance Use, SAMHSA

5600 Fishers Lane

Rockville, MD 20857

Phone: (240)-276-1966

Email: oumou.barry@samhsa

Mindy Ault, Alt. COR
Center for Behavioral Health Statistics and Quality, SAMHSA
5600 Fishers Lane
Rockville, MD 20857
Telephone: (240)-276-1262
Email: mindy.ault@samhsa.hhs.gov

Attachments

1. Public Health Services Act Section 520 L

2. System Implementation Survey (SIS)

3. Crisis Continuum Provider Survey (CCPS)

4. Key Informant Interviews: Case Study (KII-CS)

5. Key Informant Interviews: Cost Case Sub-Study (KII-CS-CSS)

6. Client Contact Disposition Form (CCDF)

7. Client Contact Disposition Form: Parent Supplement (CCDF-PS)

8. Client Experience Survey (CES)

9. Client Key Informant Interviews - Direct Contact (C-KII- DC)

10. Client-Key Informant Interviews: Third Party Contact (C-KII-TPC)

References

Abadie, A., & Gardeazabal, J. (2003). The economic costs of conflict: A case study of the Basque Country. American Economic Review, 93(1), 113-132.

Abadie, A., Diamond, A., & Hainmueller, J. (2015). Comparative politics and the synthetic control method. American Journal of Political Science, 59(2), 495-510.

Barber, J. G., Blackman, E. K., Talbot, C., & Saebel, J. (2004). The themes expressed in suicide calls to a telephone help line. Social Psychiatry and Psychiatric Epidemiology, 39, 121-125.

Beck, A. T., Kovacs, M., & Weissman, A. (1979). Assessment of suicidal intention: The Scale for Suicide Ideation. Journal of Consulting and Clinical Psychology, 47(2), 343–352. https://doi.org/10.1037/0022-006X.47.2.343

Beck, A. T., Weissman, A., Lester, D., & Trexler, L. (1974). The measurement of pessimism: The Hopelessness Scale. Journal of Consulting and Clinical Psychology, 42(6), 861–865. https://doi.org/10.1037/h0037562

Ben-Michael, E., Feller, A., & Rothstein, J. (2021). The augmented synthetic control method. Journal of the American Statistical Association, 116(536), 1789-1803.

Breiman, L., & Ihaka, R. (1984). Nonlinear discriminant analysis via scaling and ACE. Davis One Shields Avenue Davis, CA, USA: Department of Statistics, University of California.

Calancie, L., Frerichs, L., Davis, M. M., Sullivan, E., White, A. M., Cilenti, D., Corbie-Smith, G., & Hassmiller Lich, K. (2021). Consolidated Framework for Collaboration Research derived from a systematic review of theories, models, frameworks and principles for cross-sector collaboration. PloS ONE, 16(1), e0244501.

Casey, C. G. (2022). Adolescent Behaviors and Experiences Survey--United States, January-June 2021 [Supplemental material]. MMWR. Morbidity and Mortality Weekly Report, 71(3). Centers for Disease Control and Prevention.

Centers for Disease Control and Prevention/ Agency for Toxic Substances and Disease Registry/ Geospatial Research, Analysis, and Services Program [CDC]. CDC/ATSDR Social Vulnerability Index 2020 Database: US. (2022). Retrieved April 2, 2024, from https://www.atsdr.cdc.gov/placeandhealth/svi/data_documentation_download.html.

Centers for Disease Control and Prevention [CDC]. (2024). Web-based Injury Statistics Query and Reporting System (WISQARS). U.S. Department of Health and Human Services. Retrieved from https://www.cdc.gov/injury/wisqars/index.html

Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022a). The updated Consolidated Framework for Implementation Research based on user feedback. Implementation Science, 17(1), 1-16.

Damschroder, L. J., Reardon, C. M., Widerquist, M. A. O., & Lowery, J. (2022b). Conceptualizing outcomes for use with the Consolidated Framework for Implementation Research (CFIR): the CFIR Outcomes Addendum. Implementation Science, 17(1), 7.

De Los Reyes, A., Makol, B. A., Racz, S. J., Youngstrom, E. A., Lerner, M. D., & Keeley, L. M. (2019). The Work and Social Adjustment Scale for Youth: A measure for assessing youth psychosocial impairment regardless of mental health status. Journal of Child and Family Studies, 28(1), 1–16. https://doi.org/10.1007/s10826-018-1238-6

Forbes, D., Alkemade, N., Mitchell, D., Elhai, J. D., McHugh, T., Bates, G., Novaco, R. W., Bryant, R., & Lewis, V. (2013). Utility of the Dimensions of Anger Reactions–5 (DAR-5) Scale as a Brief Anger Measure. Depression and Anxiety, 31(2), 166–173. https://doi.org/10.1002/da.22148

Gould, M. S., Chowdhury, S., Lake, A. M., Galfalvy, H., Kleinman, M., Kuchuk, M., & McKeon, R. (2021). National Suicide Prevention Lifeline crisis chat interventions: Evaluation of chatters’ perceptions of effectiveness. Suicide and Life‐Threatening Behavior, 51(6), 1126-1137.

Gould, M. S., Lake, A. M., Galfalvy, H., Kleinman, M., Munfakh, J. L. H., Wright, J. D., & McKeon, R. (2017). Follow-up with Help seekers to the National Suicide Prevention Lifeline: Evaluation of Help seekers’ Perceptions of Care. Suicide and Life-Threatening Behavior, 48(1), 75–86. https://doi.org/10.1111/sltb.12339

Gould, M. S., Lake, A. M., Port, M. S., Kleinman, M., Hoyte-Badu, A. M., Rodriguez, C. L., Chowdhury, S. J., Galfalvy, H., & Goldstein, A. (2024). National Suicide Prevention Lifeline (now 988 Suicide & Crisis Lifeline): Evaluation of crisis call outcomes for suicidal callers. [Manuscript in preparation]. Department of Psychiatry, Columbia University.

Grunewald, W., Perkins, N. M., Jeon, M. E., Klonsky, E. D., Joiner, T. E., & Smith, A. R. (2024). Development and Validation of the Fearlessness About Suicide Scale. Assessment, 31(6), 1189–1203. https://doi.org/10.1177/10731911231200866

Held, L., & Paul, M. (2013). Statistical modeling of infectious disease surveillance data. Infectious Disease Surveillance, 535-544. https://doi.org/10.1002/9781118543504.ch43

Hill, R. M., Rey, Y., Marin, C. E., Sharp, C., Green, K. L., & Pettit, J. W. (2014). Evaluating the Interpersonal Needs Questionnaire: Comparison of the reliability, factor structure, and predictive validity across five versions. Suicide and Life-Threatening Behavior, 45(3), 302–314. Portico. https://doi.org/10.1111/sltb.12129

Hom, M. A., Joiner Jr, T. E., & Bernert, R. A. (2016). Limitations of a single-item assessment of suicide attempt history: Implications for standardized suicide risk assessment. Psychological Assessment, 28(8), 1026.

Imbens, G. W., & Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences: An Introduction (1st ed.). Cambridge University Press. https://doi.org/10.1017/CBO9781139025751

Lopes, L., Kirzinger, A., Sparks, G., Stokes, M., & Brodie, M. (2022). KFF/CNN Mental Health in America Survey. KFF.https://www.kff.org/mental-health/report/kff-cnn-mental-health-in-america-survey/

Matthews, S., Cantor, J. H., Brooks Holliday, S., Bialas, A., Eberhart, N. K., Breslau, J., & McBain, R. K. (2023). National preparedness for 988-the new mental health emergency hotline in the United States. Preventive medicine reports, 33, 102208. https://doi.org/10.1016/j.pmedr.2023.102208

Marcotte, D. E., & Hansen, B. (2023). The re‐emerging suicide crisis in the U.S.: Patterns, causes and solutions. Journal of Policy Analysis and Management, 43(2). https://doi.org/10.1002/pam.22526

Martinez-Ales, G., Hernandez-Calle, D., Khauli, N., & Keyes, K. M. (2020). Why Are Suicide Rates Increasing in the United States? Towards a Multilevel Reimagination of Suicide Prevention. Current Topics in Behavioral Neurosciences, 46, 1–23. https://doi.org/10.1007/7854_2020_158

Modi, H., Orgera, K., & Grover, A. (2022). Exploring barriers to mental health care in the U.S.. Washington, DC: AAMC. https://doi.org/10.15766/rai_a3ewcf9p

Mundt, J. C., Marks, I. M., Shear, K., & Greist, J. H. (2002). The Work and Social Adjustment Scale: A simple measure of impairment in functioning. British Journal of Psychiatry, 180, 461-464. https://doi.org/10.1192/bjp.180.5.461

Murthy, V. H. (2022). The mental health of minority and marginalized young people: An opportunity for action. Public Health Reports, 137(4), 613-616. https://doi.org/10.1177/00333549221102390

NRI (2023). State Behavioral Health Crisis Services Continuum, 2022. https://www.nri-inc.org/media/jlqocgys/2022-profiles-smha-the-bh-crisis-continuum-april-2023.pdf

Oei, T. P., & Green, A. L. (2008). The Satisfaction With Therapy and Therapist Scale--Revised (STTS-R) for group psychotherapy: Psychometric properties and confirmatory factor analysis. Professional Psychology: Research and Practice, 39(4), 435.

Patel, J. S., Oh, Y., Rand, K. L., Wu, W., Cyders, M. A., Kroenke, K., & Stewart, J. C. (2019). Measurement invariance of the patient health questionnaire‐9 (PHQ‐9) depression screener in U.S. adults across sex, race/ethnicity, and education level: NHANES 2005–2016. Depression and Anxiety, 36(9), 813–823. https://doi.org/10.1002/da.22940

Phalen, P. L., Rouhakhtar, P. R., Millman, Z. B., Thompson, E., DeVylder, J., Mittal, V., Carter, E., Reeves, G., & Schiffman, J. (2018). Validity of a two-item screen for early psychosis. Psychiatry Research, 270, 861–868. https://doi.org/10.1016/j.psychres.2018.11.002

Paul, M., & Held, L. (2011). Predictive assessment of a non‐linear random effects model for multivariate time series of infectious disease counts. Statistics in medicine, 30(10), 1118-1136.

Pisani, A. R., Gould, M. S., Gallo, C., Ertefaie, A., Kelberman, C., Harrington, D., Weller, D., & Green, S. (2022). Individuals who text crisis text line: Key characteristics and opportunities for suicide prevention. Suicide and Life-Threatening Behavior, 52(3), 567-582. https://doi.org/10.1111/sltb.12872

Purtle, J., Chance Ortego, J., Bandara, S., Goldstein, A., Pantalone, J., & Goldman, M. L. (2023). Implementation of the 988 Suicide & Crisis Lifeline: Estimating State-Level Increases in Call Demand Costs and Financing. The Journal of Mental Health Policy and Economics, 26(2), 85–95.

Raftery, A. E., Gneiting, T., Balabdaoui, F., & Polakowski, M. (2005). Using Bayesian model averaging to calibrate forecast ensembles. Monthly Weather Review, 133(5), 1155-1174. https://doi.org/10.1175/MWR2906.1

Riske-Morris, M., Hussey, D. L., Olgac, T., Bassett, J., & Holt, G. (2024). Commentary on the Revision of SAMHSA's National Outcome Measures (NOMs) for Discretionary Programs. The Journal of Behavioral Health Services & Research, 51(2), 302–308. https://doi.org/10.1007/s11414-023-09868-x

Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5), 688–701. https://doi.org/10.1037/h0037350

Rutter, L. A., & Brown, T. A. (2016). Psychometric properties of the Generalized Anxiety Disorder Scale-7 (GAD-7) in outpatients with anxiety and mood disorders. Journal of Psychopathology and Behavioral Assessment, 39(1), 140–146. https://doi.org/10.1007/s10862-016-9571-9

Santillanes, G., Axeen, S., Lam, C. N., & Menchine, M. (2020). National trends in mental health-related emergency department visits by children and adults, 2009–2015. The American journal of emergency medicine, 38(12), 2536-2544. https://doi.org/10.1016/j.ajem.2019.12.035

Saunders, J. B., Aasland, O. G., Babor, T. F., De la Fuente, J. R., & Grant, M. (1993). Development of the alcohol use disorders identification test (AUDIT): WHO collaborative project on early detection of persons with harmful alcohol consumption‐II. Addiction, 88(6), 791-804. https://doi.org/10.1111/j.1360-0443.1993.tb02093.x

Scaccia, J. P., Cook, B. S., Lamont, A., Wandersman, A., Castellow, J., Katz, J., & Beidas, R. S. (2015). A practical implementation science heuristic for organizational readiness: R= MC2. Journal of community psychology, 43(4), 484-501. https://doi.org/10.1002/jcop.21698

Skinner, H. A. (1982). Guide for using the drug abuse screening test (DAST). Toronto: Centre for Addiction and Mental Health.

Substance Abuse and Mental Health Services Administration [SAMHSA]. (2020). National Guidelines for Behavioral Health Crisis Care–A Best Practice Toolkit. U.S. Department of Health and Human Services.

Substance Abuse and Mental Health Services Administration [SAMHSA]. (2023a). Strategic plan: Fiscal year 2023-2026 (Publication No. PEP23-06-00-002). National Mental Health and Substance Use Laboratory, U.S. Department of Health and Human Services.

Substance Abuse and Mental Health Services Administration [SAMHSA]. (2023b). Key substance use and mental health indicators in the United States: Results from the 2022 National Survey on Drug Use and Health (HHS Publication No. PEP23-07-01-006, NSDUH Series H-58). Center for Behavioral Health Statistics and Quality, U.S. Department of Health and Human Services. https://www.samhsa.gov/data/report/2022-nsduh-annual-national-report

Substance Abuse and Mental Health Administration [SAMHSA]. (2024, April). Lifeline Performance Metrics. U.S. Department of Health and Human Services. Retrieved June 24, 2024, from https://www.samhsa.gov/find-help/988/performance-metrics.

Substance Abuse and Mental Health Services Administration [SAMHSA]. (2024, April). 988 Suicide & Crisis Lifeline. U.S. Department of Health and Human Services. Retrieved July 9, 2024, from https://www.samhsa.gov/find-help/988

U.S. Department of Health and Human Services [HHS]. (2024, April).National Strategy for Suicide Prevention. Washington, DC: HHS.

Vahratian, A., Blumberg, S. J., Terlizzi, E. P., & Schiller, J. S. (2021). Symptoms of Anxiety or Depressive Disorder and Use of Mental Health Care Among Adults During the COVID-19 Pandemic - United States, August 2020-February 2021. MMWR. Morbidity and mortality weekly report, 70(13), 490–494. https://doi.org/10.15585/mmwr.mm7013e2

Van Spijker, B. A., Batterham, P. J., Calear, A. L., Farrer, L., Christensen, H., Reynolds, J., & Kerkhof, A. J. (2014). The Suicidal Ideation Attributes Scale (SIDAS): Community‐based validation study of a new scale for the measurement of suicidal ideation. Suicide and Life‐Threatening Behavior, 44(4), 408-419.

Vinson, S. Y., & Dennis, A. L. (2021). Systemic, racial justice–informed solutions to shift “care” from the criminal legal system to the mental health care system. Psychiatric Services, 72(12), 1428-1433. https://doi.org/10.1176/appi.ps.202000735

Walker, T. J., Brandt, H. M., Wandersman, A., Scaccia, J., Lamont, A., Workman, L., ... & Fernandez, M. E. (2020). Development of a comprehensive measure of organizational readiness (motivation × capacity) for implementation: a study protocol. Implementation Science Communications, 1, 1-11. https://doi.org/10.1186/s43058-020-00088-4

The White House. (2022, May 31). Fact sheet: Biden-Harris Administration highlights strategy to address the national mental health crisis. The White House. Retrieved July 9, 2024, from https://www.whitehouse.gov/briefing-room/statements-releases/2022/05/31/fact-sheet-biden-harris-administration-highlights-strategy-to-address-the-national-mental-health-crisis/

Witte, T. K., Gould, M. S., Munfakh, J. L. H., Kleinman, M., Joiner, T. E., & Kalafat, J. (2010). Assessing suicide risk among callers to crisis hotlines: A confirmatory factor analysis. Journal of Clinical Psychology, 66(9), 941–964. https://doi.org/10.1002/jclp.20717

Zabelski, S., Kaniuka, A. R., A Robertson, R., & Cramer, R. J. (2023). Crisis Lines: Current Status and Recommendations for Research and Policy. Psychiatric services (Washington, D.C.), 74(5), 505–512. https://doi.org/10.1176/appi.ps.20220294

SS Part B 988 Evaluation 9