Supporting Statement MHFA Part B 1 25 17

Supporting Statement MHFA Part B 1 25 17.docx

Evaluation of the Mental Health First Aid Program

OMB: 0930-0371

Document [docx]

Download: docx | pdf

Evaluation of the Mental Health First Aid Program

Supporting Statement B

1. Respondent Universe and Sampling Methods

MHFA/YMHFA Training Surveys: SAMHSA will use a population (census) approach for administering the three MHFA/YMHFA trainee surveys. SAMHSA expect near-full participation (80%) but will monitor response rates over time.

Qualitative Interviews: As described earlier, SAMHSA will conduct interviews with 10% of grantee sites (i.e., at least 16 school districts from the SEA and LEA programs and seven community grantees from the community program).

Detailed estimates for each data collection type are outlined in Table 1.

Table 1. Background and Estimated Number of Respondents

	Characteristic	AWARE-SEA	AWARE-LEA	AWARE-C	Total
Background	Grantee unit	State departments of education (and three school districts in each state)	School districts	Community organizations
	Number of grantees	20	31	71	122
	Target respondents	School personnel, adults who interact with school-aged youth	Teachers, counselors, and other school personnel, as well as emergency responders, caregivers, parents, and other youth-serving adults	Teachers and a broad array of professionals who interact with youth through their programs at the community level, including parents, law enforcement, faith-based leaders, and other adults
Estimated Number of Respondents	Estimated total number of MHFA/YMHFA pre-training survey respondents enrolled over a roughly 10 month long recruitment period (October 2016– August 2017, contingent upon timing of OMB approval)¹	6,000 (125 per school district in a one year period, 80% of which is 100; three school districts per state, 20 states)	3,100 (125 per school district per year, 80% of which is 100; 31 school districts)	7,100 (125 per community organization per year, 80% of which is 100; 71 community organizations. However, unlike the two other grants, this figure will vary by AWARE-C grantees, based on ratio of adults to be trained relative to the size of the population of focus for effective saturation of the geographic catchment area)	16,200
	Estimated total number of MHFA/YMHFA post-training survey respondents enrolled over a roughly 10 month long recruitment period (October 2016– August 2017, contingent upon timing of OMB approval)²	6,000 (125 per school district per year, 80% of which is 100; three school districts per state, 20 states)	3,100 (125 per school district per year, 80% of which is 100; 31 school districts)	7,100 (125 per community organization per year, 80% of which is 100; 71 community organizations. However, unlike the two other grants, this figure will vary by AWARE-C grantees, based on ratio of adults to be trained relative to the size of the population of focus for effective saturation of the geographic catchment area)	16,200
	Estimated total number of MHFA/YMHFA follow-up survey respondents enrolled over a roughly 14 months (January 2017 – February 2018, contingent upon timing of OMB approval)³	9,600 3 month: 125 per school district in a one year period, 85% sample retention and 80% response rate which yields 95 respondents; three school districts per state, 20 states 6 month: 125 per school district per year, 75% sample retention. 80% response rate which yields 90 respondents; three school districts per state, 20 states	4,960 3 month: 125 per school district per year, 85% sample retention, 80% response rate, which yields 95 respondents; 31 school districts 6 month: 125 per school district in one year period, 75% sample retention, 80% response rate, which yields 90 respondents; 31 school districts	11,360 3 month: 125 per community organization per year, 85% sample retention, 80% response rate, which yields 95 respondents; 71 community organizations. However, unlike the two other grants, this figure will vary by AWARE-C grantees, based on ratio of adults to be trained relative to the size of the population of focus for effective saturation of the geographic catchment area) 6 month: 125 per school district in one year period, 75% sample retention, 80% response rate, which yields 90 respondents; 71 community organizations. However, unlike the two other grants, this figure will vary by AWARE-C grantees, based on ratio of adults to be trained relative to the size of the population of focus for effective saturation of the geographic catchment area)	25,920
	Interviewees⁴	8 interviewees, each from a distinct school district	8 interviewees, each from a distinct school district	7 interviewees, each from a distinct community organization	23
	Total respondents (duplicated across trainee surveys)	21,608	11,168	25,567	58,343⁵

2. Procedures for the Collection of Information

a. Statistical Methodology for Stratification and Sample Selection

A census approach will be used for the MHFA/YMHFA trainee surveys (see estimates in table above). The minimum detectable effect (MDE) ranges from 0.14 to 0.19 under the assumption of an intra-class correlation coefficient of 0.05–0.10. This range is well within (actually lower than) the expected effect sizes for MHFA reported in prior research. Hadlaczky, Hökby, Mkrtchian, Carli, and Wasserman (2014)⁶ conducted a meta-analysis including 15 studies (12 studies were conducted in Australia, 2 in Sweden, and 1 in Canada; 4 were randomized controlled trials; 3 used YMHFA). Examination of three outcomes—knowledge, attitudes, and helping behaviors—found that average effect sizes for MHFA/YMHFA were 0.56 for knowledge, 0.28 for attitudes, and 0.25 for behaviors. Therefore, although we are not trying to make any claims about the effectiveness of MHFA, we should be more than adequately powered to produce trustworthy results (avoiding a Type II error). A sampling approach will be used for qualitative interviews. SAMHSA expect to include approximately 10% of the grantees in the interview study (i.e., at least 16 school districts from the SEA and LEA programs and 7 community grantees from the community program). These grantees will be randomly selected from the full population of grantees based on two strata: grant program, and urban/suburban/rural context. The decision to interview 10% of grantees was made in light of recognition that a sufficient number of grantees should participate in interviews to allow the investigators to capture enough variance between grantees, while minimizing burden to the extent possible. Ten percent made sense as a balance point. At each selected site, SAMHSA will interview the individual who is most knowledgeable about the MHFA/YMHFA plans and activities. Selected grantees can voluntarily participate in or decline the interviews, in which case SAMHSA will randomly select interviewees from the next grantee on the basis of the identified strata.

b. Estimation Procedure

The estimates for responses provided in this OMB package are based on the following points:

MHFA/YMHFA trainee surveys: SAMHSA expect that all MHFA/YMHFA trainees will complete surveys as part of the “business as usual” of the trainings.
Interviews: Please see above (under “Statistical methodology for stratification and sample selection”)

c. Degree of Accuracy Needed

To compare implementation of MHFA/YMHFA across the three AWARE grant programs, it will be extremely important to have accurate information for each of our data collection activities:

MHFA/YMHFA trainee surveys: trainees must accurately report changes in their mental health literacy over time. To maximize validity of recall, trainees will be asked to complete surveys immediately before and after training, and in a timely fashion at three and six month intervals. In addition, the anonymity of the data should mitigate any reluctance to provide honest self-reports for any of the items (e.g., attitudes about mental illness).
Interviews: it is essential to receive accurate input from grantees about facilitators and challenges encountered during implementation of MHFA/YMHFA. Accuracy will be enhanced by selecting interviewees that are well positioned in the district or community (e.g., as key informants) to speak to the real strengths and barriers encountered.

3. Methods to Maximize Response Rates and Deal with Nonresponse

MHFA/YMHFA surveys are voluntary, yet encouraged. Even in the absence of a requirement, incentives won’t be used. SAMHSA expect a sufficient response given that SAMHSA are providing support, and disseminating a tool that that will provide grantees with useful information that can be integrated into their local evaluations. Activities to encourage response rates will include messaging about TTA events and reminder emails to ensure that surveys are received in a timely manner. Additional data collection activities are also voluntary.

4. Tests of Procedures or Methods to be Undertaken

We are not planning to undertake tests of procedures or methods as part of the work described in this OMB package. However, past testing indicates that the MHFA/YMHFA survey has Cronbach alphas ranging from .69 to .95, indicating that, overall, the items in the instrument have acceptable internal consistency. Assessment of survey validity is being assessed as part of Georgetown University’s ongoing program of research. The qualitative interview protocol is based on the Interactive Systems Framework (ISF) for Dissemination and Implementation,⁷ a conceptual model for how communities, organizations, or any other host settings change to effectively begin innovations, and/or to improve the quality of the services they provide. We have no current plans to field test or cog lab the protocol; however, the concepts and principles reflected in the ISF have been well operationalized in the literature, which we reviewed as part of developing of the qualitative protocol. Qualitative interviewers are currently receiving training on use of the protocol.

5. Consultants on Statistical Aspects of the Design and People who will Collect and Analyze the Information

The MHFA/YMHFA evaluation design was developed under contract to the American Institutes for Research in conjunction with Dr. Laura Jacobus-Kantor (SAMHSA COR) and the technical work group. The members of the evaluation team are listed below.

Bruno Anthony, Ph.D.

Center for Child and Human Development, Georgetown University

202-687-5086

bja280@gmail.com

My Banh, Ph.D.

Center for Child and Human Development, Georgetown University

202-944-5394

My.K.Banh@gunet.georgetown.edu

Elizabeth Chagnon