GDTA Needs Assessment SS-B_07-8-15

Grantee Data Technical Assistance

Training Needs Assessment Survey for SAMHSA Grantees

Supporting Statement

B. Collections of Information Employing Statistical Methods.

B1. Respondent Universe and Sampling Methods

The respondent universe consists of one grantee (Program Director, or PD) or a designated representative chosen by the PD from each of the 2,670 SAMHSA-funded discretionary grants served by the GDTA contract. Tabular presentations of grantee counts by program are presented for each grant-making (or programmatic) Center (CSAP, CSAT, CMHS) below.

CSAP	PD Count
CBI	27
DFC CADCA	2
DFC Mentoring Program	21
DFC Support Program	679
Idaho SPF SIG	1
Minority SA-HIV Prev. Initiative	2
MSI CBO	50
Partnerships for Success	4
PFS II SEOW Supplements	15
Ready-To-Respond Initiative	36
SA HIV AIDS Prevention New media	26
SPF-PFS	36
SPF-PFS II	15
SPF-SIG	33
STOP Act Grants	98

CSAT	PD Count		PD Count
ATCC	7	PDMP	9
ATHM	27	PPW	34
ATR	28	RCSP	3
ATTC	14	RCSP-SN	10
ATTC-COE	1	SABG-TA	1
CABHI	16	SBIRT	18
CABHI-States	11	SBIRT Medical Resident	7
Children Affected by Meth	8	SBIRT Training	25
EHR and PDMP Data Integration	7	SAMHSA TDC-AJF	100
GBHI-SSH	26	SAMHSA TDC-TJ	17
HBCU-CFE	1	SAT-ED	13
HCV Screening & Referral	3	State Youth Treatment	11
Joint adult drug court	43	Teen Court Program	10
MAI-COC Pilot	34	TCE-HIT	19
NITT-MFP-AC	2	TCE-HIV	62
OJJDP-JDC	2	TCE-HIV Minority Women	39
ORP	29	TCE-PTP	21
OTP-CoC	2	TCE-TAC	58
PCSS-MAT	1	Vietnam H-ATTC	2
PCSS Opioid	2

CMHS	PD Count		PD Count
Adult Treatment Court Collaboratives	16	NITT-MFP-Y	5
CABHI-States	7	NITT-Healthy Transitions	17
Campus Suicide Prevention Grant	87	NITT-AWARE-SEA	20
Child Mental Health Initiative	39	NITT-AWARE_LEA	100
Circles of Care	14	NTAC	1
Consumer and CSTA Center	5	PBHCI	93
Crisis Center Follow Up	12	Prevention Practices in Schools	21
Crisis Counseling	3	Project LAUNCH	45
Early Diversion	3	Services in Supportive Housing	18
Emergency Response	4	SOC Expansion Implementation	32
Healthy Transitions	7	SOC Implementation	22
Jail Diversion and Trauma Recovery	5	SOC Planning	9
Lifeline	1	SSHS State Program	7
Lifeline Supplement	1	State Tribal Youth Suicide Prevention	82
MAI-TCE	11	Statewide Consumer Network	25
Mental Health Transformation	20	Statewide Family Network	35
Minority Fellowship Program	6	Statewide Peer Networks RR	9
Native Connections	20	SPRC Supplement 14	1
National Strategy Grants	4	Supported Employment Program	7
NCTSI- Category I	1	System of Care expansion planning	7
NCTSI- Category II	20	TSA Supplements	1
NCTSI-Category III	58	YVP-RC	1

This represents a total respondent universe of 2,670 grantees.

SAMHSA’s proposed approach for the data collection for the Grantee Data Technical Assistance Needs Assessment is to submit a questionnaire to be filled out by a respondent for each grantee -- in other words, to conduct a census of grantees or, put another way, to attempt to collect data from 100% sampling of grantees.

SAMHSA believes this is justified by the need to determine whether there are program-specific training or technical assistance requirements. Were SAMHSA instead to use a stratified sampling approach with strata for each program, the small number of grantees in some programs, combined with the effect on the sampling mathematics from the use of finite population correction within strata, it would necessitate a very high sampling fraction for statistically projectable results. In effect SAMHSA would be nearing 100% sampling in any event, with the added complications of robust statistical calculations proceeding from the use of stratification. SAMHSA believes that on balance the use of 100% sampling, with corrections for undercounts or nonresponse as necessary, will produce a dataset of greater overall quality.

The “PD Counts” in the tables above represent the entities (persons, representing grantees) in the universe covered by the collection and in the corresponding sample, as SAMHSA is proposing 100% sampling. While SAMHSA is not using stratification per se, SAMHSA is proposing reporting results by program, and so each cell listing a grant program funded by one of the Centers represents a unit by which results of interest may be broken out.

SAMHSA anticipates response rates of 90 percent or greater for the collection as a whole. SAMHSA discusses SAMHSA’s means for maintaining high response rates in Section B.2 below.

No prior response rates are available, as this is a novel data collection.

B2. Information Collection Procedures

• Statistical methodology for stratification and sample selection

As noted above, SAMHSA proposes to use 100% sampling on a universe consisting of all SAMHSA grantees for all three programmatic Centers (CSAP, CSAT, CMHS) that are served by the GDTA contract.

• Estimation procedure

Estimation will consist of direct reporting of responses (summarized as descriptive statistics for proportions data) by question, by Center and by program, with adjustments (described in B.3 below) as necessary for nonresponse.

• Degree of accuracy needed for the purpose described in the justification

Data should be sufficiently accurate to allow SAMHSA to determine whether certain types of technical assistance are preferred to others by program grantees. A margin of error of +/- 5% at a 95% level of confidence is acceptable.

• Unusual problems requiring specialized sampling procedures

No specialized sampling procedures are required.

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden

The language of the contract requires that a needs assessment be performed annually, with a survey-based assessment (the one for which SAMHSA is currently seeking approval) in the second year of the period of performance. The first year’s data collection has been performed via key informant interviews, small focus groups of SAMHSA GPOs reflecting on their needs and those of their grantees.

B3. Methods to Maximize Response Rates

SAMHSA addresses the issue of response rates in two different ways – first, via the brevity of the instrument itself. SAMHSA anticipates that a grantee can complete the questionnaire in 7 minutes or less, and all questions on the instrument are either prepopulated from information in the outreach database (such as grant numbers and funding centers) or relate directly to the impressions of the respondent, without any need for desk research or other time-consuming activities in order to produce the requested data. SAMHSA expects a very, very low rate of attrition or abandonment of questionnaires. SAMHSA is also providing multiple means of responding: while the principal data collection method involves a web-based questionnaire in research.net, with a personalized link sent via e-mail, respondents can also request and fill out a paper questionnaire to be entered into the system by SAMHSA’s research assistant staff. SAMHSA is also prepared to enter data directly for respondents who wish to provide their answers via phone.

SAMHSA’s second approach to maximizing response rates is to provide for a number of periodic reminders to respondents throughout the time allotted for answering the questionnaire (anticipated to be 1 month). SAMHSA is employing an outreach database with contact information for each grantee, as well as grant number and funding Center information, in order to e-mail and partially prepopulate the questionnaires. SAMHSA will use the same database (cross-referenced against data on returned questionnaires in research.net) to e-mail reminders 3 days, 1 week, 2 weeks, and 3 weeks after distribution – as well as 48 hours prior to the conclusion of data collection – for any outstanding questionnaires.

Finally, but less formally, the preamble for the questionnaire makes clear the purpose of the data collection – to better target the GDTA resources SAMHSA is making available to grantees – which aligns sufficiently with grantee self-interest to motivate them to respond promptly and accurately.

Regarding nonresponse, SAMHSA would differentiate potential nonresponse rates from nonresponse error or bias, as SAMHSA has no reason to expect a priori that the phenomenon of nonresponse is in any way related to respondent characteristics or the data that nonrespondents would provide (in other words, SAMHSA believes influences on survey participation are not shared with influences on the survey variables). SAMHSA will report nonresponse rates by program.

There are a number of acceptable means¹ of estimating nonresponse bias, not all of which are applicable in SAMHSA’s circumstances:

• Benchmarking results for respondents against an accurate auxiliary source of information

• Using information in the sampling frame to determine whether there is a systematic empirical relationship between sample member characteristics (observable for both respondents and nonrespondents) and response rates

• Using information from a prior wave or from screening questions, or using level of effort measures (number of required attempts, etc.) on difficult respondents in order to estimate results for nonrespondents

It would be difficult to estimate nonresponse bias from the first and third methods due to a lack of auxiliary sources of information similar to ours or from prior tests or waves (this is a novel data collection); SAMHSA is also not using any screening questions.

It is, however, both possible and straightforward to use the observable characteristics in the outreach database (grant program, funding Center, and, indirectly, grantee type such as state government, unit of local government, etc.) to determine whether nonrespondents were statistically significantly different in any observable characteristic, to provide an estimate of at least one measure of nonresponse bias that can be applied to statistics from respondents. This method can be applied to subsets of nonrespondents as well (such as refusers), at least in principle.

SAMHSA can also track those respondents who required multiple attempts at outreach, or respondents who provided their data only in the final days of the data collection window, to measure whether nonrefusal-nonrespondents might be statistically distinct from respondents.

All of these methods require auxiliary assumptions but can help to indicate whether nonresponse rates indicate the actuality, rather than the risk, of nonresponse bias.

B4. Tests of Procedures

SAMHSA’s team has conducted internal cognitive testing on the questionnaire, and the instrument has been reviewed by SAMHSA staff as well as SAMHSA’s own experienced TA providers and trainers, who understand the context and language in use by its grantees.

B5. Statistical Consultants

SAMHSA’s statistical consultant for the design is Dr. Steven Sullivan of Cloudburst Consulting Group. Dr. Sullivan is an econometrician with a history of successful study design and implementation for SAMHSA, including the data collection and evaluation planning for the Co-Occurring Disorders Integration and Innovation (CODI) contract. Dr. Sullivan is also the lead for the data collection efforts (using Surveymonkey’s research.net service) and the analysis of needs assessment data from the survey.

Additional consultants include Rachael Kenney and Laura Gillis of the Center for Social Innovation and Tim Mayo of SAMHSA. Ms. Kenney is the Project Manager overseeing this OMB submission. Ms. Gillis is the Project Director for the GDTA contract. Tim Mayo is the SAMHSA Project Officer overseeing the GDTA contract.

Steven T. Sullivan, Ph.D.

Senior Director and Subject Matter Expert

Cloudburst Consulting Group, Inc.

8400 Corporate Drive, Suite 550

Landover, MD 20785

301-385-6693

steven.sullivan@cloudburstgroup.com

Rachael R. Kenney, M.A.

Associate

Center for Social Innovation

200 Reservoir Street, Suite 202

Needham, MA 02494

rkenney@center4si.com

Laura M. Gillis, R.N., M.S.

Vice President

Center for Social Innovation

200 Reservoir Street, Suite 202

Needham, MA 02494

lmgillis@center4si.com

Tim Mayo

HHS/SAMHSA

1 Choke Cherry Road

Rockville, MD 20850

tim.mayo@samhsa.hhs.gov

List of Attachments

1. GDTA Training Needs Assessment Survey for SAMHSA Grantees

2. Email invitation to Project Directors

3. Screen shots of GDTA Training Needs Assessment Survey in web application

1 Montaquila JM, & Olson KM. (2012, April 24). “Practical Tools for Nonresponse Bias Studies.” In SRMS/AAPOR Webinar Series. Retrieved from http://www.amstat.org/sections/srms/webinarfiles/NRBiasWebinarApril2012.pdf.

9