Form Annual Performance Annual Performance Annual Performance Assessment

OMB No. 0930-0xxx

Expiration Date:  xx/xx/xx

Attachment 3: NRC Annual Performance Assessment Survey

 

Public Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0930-0xxx.  Public reporting burden for this collection of information is estimated to average of 30 minutes per respondent, per year, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to SAMHSA Reports Clearance Officer, 5600 Fishers Lane, Room 15E57-B, Rockville, Maryland, 20857.

Note: The Center Assessment Team (CAT) at Community Science developed a measurement plan to assess the training and technical assistance (T/TA) provided by the National Resource Center for Youth Violence Promotion (NRC) to SAMHSA’s Safe Schools/Healthy Students (SS/HS) and Project LAUNCH grantees. Annually, the CAT distributes the online survey instrument toward the end of the fiscal year (mid-September) to key grantee staff. For the SS/HS program, surveys are sent to (1) all current SS/HS project coordinators and (2) their local education agency representatives. For Project LAUNCH, surveys are sent to Project LAUNCH grantee staff and three sets of local partners: (1) young child wellness experts; (2) young child wellness partners; and (3) young child wellness coordinators. The survey may also be distributed to other [local education agency representatives (SS/HS) or local community partners (Project LAUNCH)], who have accessed or received services from the NRC in the last fiscal year.

If necessary, the survey can be distributed in hard copy to grantees, who may experience technological challenges or limited English proficiency. In addition, if needed, AIR will translate the paper survey into a language other than English. The CAT then requests from AIR a current list of grantee contact names and email addresses to send the survey email link. The 30-minute online survey is fielded for two to three weeks. Survey invitees first receive an email announcement from the NRC to encourage participation. They receive weekly reminders for two weeks from the CAT, and then every other day until the fielding period has ended. Other follow-up measures may be implemented to follow up on the administration of paper surveys. After data collection ends, the CAT generates a database of survey responses and conducts data cleaning. The CAT analyzes the data and reports the findings to AIR as part of the annual performance assessment report.

INVITATION LANGUAGE

Invitation Email Script (sent by NRC leadership)

Dear [insert name]:
 
I am writing you on behalf of the National Resource Center for Mental Health Promotion and Youth Violence Prevention (NRC) Assessment Team at Community Science.

You are receiving this email because you are a grantee state project coordinator or [local education agency SS/HS) or local community partner (Project LAUNCH)] who has received services from the NRC. The NRC’s services have most likely been provided to you by the NRC’s resource specialist or communications specialist in person, online, or electronically. NRC services do not include services provided by government project officers. We are requesting that you complete an important survey that asks about your experiences as a participant in the grant program receiving training and technical assistance services from the NRC. The survey will take 30 minutes or less to complete.

Your feedback will help ensure that the NRC is providing you with the best services needed to support your work, and it will inform the services that will be provided by NRC staff in the coming years. Your participation is voluntary, and your individual responses will be de-identified to maintain confidentiality. The findings from this survey will be incorporated into the NRC’s annual report to SAMHSA and may be included in published articles or presented at professional conferences.

We appreciate you taking time out of your busy schedule to complete the questionnaire. If you have any technical difficulties accessing the questionnaire, feel free to contact the survey manager TBD. If you have any additional questions about this instrument and how it pertains to your grant, please feel free to contact your SS/HS resource specialist.

You will receive an email from noreply@qemailserver.com with a link to the survey. If you do not receive this email within one day of receiving this email, please check your junk or spam folders or contact TBD for another link to the survey. Please complete and submit your responses by XX/XX/XX.
 
Thank you!
 
 
[NRC leadership signature]

Reminder Email Script (sent by Community Science)

Dear [insert name]:
 
You have recently received an email asking you to complete an important survey that asks about your experiences receiving training and technical assistance services from the National Resource Center for Mental Health Promotion and Youth Violence Prevention (NRC) as a participant in the [SS/HS or Project LAUNCH] grant program funded by SAMHSA.

You are receiving this email because you are a grantee state project coordinator or [local education agency SS/HS) or local community partner (Project LAUNCH)] who has received services from the NRC. The NRC’s services have most likely been provided to you by the NRC’s resource specialist or communications specialist in person, online, or electronically. NRC services do not include services provided by federal or government grant officers. We are requesting that you complete an important survey that asks about your experiences as a participant in the grant program receiving training and technical assistance services from the NRC.

Your feedback will help ensure that the NRC is providing you with the best services needed to support your work, and it will inform the services that will be provided by NRC staff in the coming years. Your participation is voluntary, and your individual responses will be de-identified to maintain confidentiality. The findings from this survey will be incorporated into the NRC’s annual report to SAMHSA and may be included in published articles or presented at professional conferences.

We appreciate you taking time out of your busy schedule to complete the survey. If you have any technical difficulties accessing the survey, feel free to contact survey manager TBD. If you have any additional questions about this instrument and how it pertains to your grant, please feel free to contact your resource specialist.
 
You should have received an email from noreply@qemailserver.com with a link to the survey. If you have not received this email, please check your junk or spam folders or contact TBD for another link to the survey. Please complete and submit your responses by XX/XX/XX.
 
Thank you!
 
TBD

ANNUAL PERFORMANCE ASSESSMENT SURVEY DATA COLLECTION INSTRUMENT

National Resource Center for Mental Health Promotion and Youth Violence Prevention (NRC)

[Safe Schools/Healthy Children (SS/HS) or Project LAUNCH] Annual Performance Assessment Survey

OMB No:

Exp. Date:

The National Resource Center for Mental Health Promotion and Youth Violence Prevention (NRC) has developed this online survey to obtain your feedback about the training and technical assistance (T/TA) services provided to you by the NRC. Your feedback will help ensure that the NRC is providing you with the best services needed to support your work, and it will inform the services that will be provided by NRC staff in the coming years.

Your participation is voluntary, and your individual responses will be de-identified to maintain confidentiality. The findings from this survey will be incorporated into the NRC’s annual report to SAMHSA and may be included in published articles or presented at professional conferences.

[NEXT PAGE]

1. (a) Please select your role on the SAMHSA grant. [DROP-DOWN SELECTION, FORCED RESPONSE]

• State project coordinator

• [Local education agency/community program manager (SS/HS) or local community partner (Project LAUNCH)

• Other (please specify) [OPEN ENDED]

1. (b) Please select your [Project LAUNCH] cohort. [DROP-DOWN SELECTION, FORCED RESPONSE]

• Cohort 4

• Cohort 5

• Cohort 6

• Expansion Cohort

• Other (specify) __________________

2. In the last 12 months, have you received, used, or accessed the following NRC training and technical assistance (T/TA) services? Check all that apply. [MULTIPLE SELECTION, FORCED RESPONSE]

   • NRC bi-annual grantee meeting or summit

   • TA Gateway

   • NRC public website

   • NRC Office Hours

   • NRC grantee site visits

   • Grantee-specific technical assistance (e.g., from resource specialists or communication specialists)

   • Technical assistance tools and materials (e.g., manuals, worksheets, guides)

   • Online learning events (OLEs; e.g., live webinars)

   • Self-guided online learning modules (OLMs; e.g., prerecorded webinars)

   • Alumni mentoring and peer networking opportunities

   • Community of practice (CoP) discussions

   • Other: _____________________________

   • None of the above

[NEXT PAGE]

For each set of T/TA topics listed in the rows below, please check in the columns for each row the NRC services you received, used, or accessed in the last 12 months. Check all that apply. [CARRY FORWARD ITEMS SELECTED IN Q2 ACROSS MATRIX COLUMNS, FORCED RESPONSE]

Did not use or receive services in this T/TA topic

NRC annual grantee conference

TA Gateway

NRC public website

NRC Office Hours

NRC grantee site visits

Grantee-specific technical assistance (e.g., resource specialists or communication specialists)

Technical assistance tools and materials (e.g., manuals, worksheets, guides)

Online learning events (e.g., webinars)

Self-guided online learning modules (e.g., prerecorded webinars)

Alumni mentoring and peer- networking opportunities

Community of practice discussion

Marketing and communications

□

□

□

□

□

□

□

□

□

□

□

□

Environmental scans

□

□

□

□

□

□

□

□

□

□

□

□

Planning (including SS/HS comprehensive plans)

□

□

□

□

□

□

□

□

□

□

□

□

Use of a public health approach (including policy and systems change, and the Strategic Prevention Framework)

□

□

□

□

□

□

□

□

□

□

□

□

Grant sustainability

□

□

□

□

□

□

□

□

□

□

□

□

Assessment of state needs and resources, external environment, and readiness of grantees

□

□

□

□

□

□

□

□

□

□

□

□

Selection and use of evidence-based practices (EBPs; including use of evaluation results to adapt EBPs)

□

□

□

□

□

□

□

□

□

□

□

□

Grant implementation with quality and fidelity to the SS/HS grant model

□

□

□

□

□

□

□

□

□

□

□

□

Data use and other data-driven practices

□

□

□

□

□

□

□

□

□

□

□

□

Cultural and linguistic competence (CLC)

□

□

□

□

□

□

□

□

□

□

□

□

Provision of a continuum of cross-system care

□

□

□

□

□

□

□

□

□

□

□

□

Workforce training

□

□

□

□

□

□

□

□

□

□

□

□

Interdisciplinary, multilevel partnerships and systems-level collaboration

□

□

□

□

□

□

□

□

□

□

□

□

Health disparity reduction strategies

□

□

□

□

□

□

□

□

□

□

□

□

Mental health promotion and violence prevention

□

□

□

□

□

□

□

□

□

□

□

□

Diffusion and spread of effective program strategies

□

□

□

□

□

□

□

□

□

□

□

□

[NEXT PAGE]

The following questions ask you to provide information on your utilization of services provided by the NRC in the last 12 months and the perceived usefulness of these services. [MATRIX, FORCED RESPONSE]

Overall, how useful were each of the following NRC T/TA services to you in the last 12 months? If you did not receive, use, or access the listed service, check the “Did not receive or participate” box.

	Did not receive or participate 0	Not at all useful 1	A little bit useful 2	Somewhat useful 3	Very useful 4	Extremely useful 5
NRC annual grantee conference	□	□	□	□	□	□
TA Gateway	□	□	□	□	□	□
NRC public website	□	□	□	□	□	□
NRC Office Hours	□	□	□	□	□	□
NRC grantee site visits	□	□	□	□	□	□
Grantee-specific technical assistance (e.g., assistance provided by resource specialists)	□	□	□	□	□	□
Technical assistance tools and materials (e.g., manuals, worksheets, guides)	□	□	□	□	□	□
Online learning events (OLEs; e.g., live webinars)	□	□	□	□	□	□
Self-guided online learning modules (OLMs; e.g., prerecorded webinars)	□	□	□	□	□	□
Alumni mentoring and peer-networking opportunities	□	□	□	□	□	□
Community of practice (CoP) discussions	□	□	□	□	□	□

[NEXT PAGE]

Please help us understand how the following NRC T/TA services could be improved.

30. How could the NRC annual grantee conference be improved? [DISPLAY IF Q19 = 1-3, OPEN-ENDED]

30. How could the TA Gateway be improved? [DISPLAY IF Q20 = 1-3, OPEN-ENDED]

30. How could the NRC public website be improved? [DISPLAY IF Q21 = 1-3, OPEN-ENDED]

30. How could the NRC grantee site visits be improved? [DISPLAY IF Q22 = 1-3, OPEN-ENDED]

30. How could the grantee-specific technical assistance (e.g., assistance by resource specialists) be improved? [DISPLAY IF Q23 = 1-3, OPEN-ENDED]

30. How could the technical assistance tools and materials (e.g., manuals, worksheets, and guides) be improved? [DISPLAY IF Q24 = 1-3, OPEN-ENDED]

30. How could the online learning events (OLEs; e.g., live webinars) be improved? [DISPLAY IF Q25 = 1-3, OPEN-ENDED]

30. How could the self-guided online learning modules (OLMs; e.g., prerecorded webinars) be improved? [DISPLAY IF Q26 = 1-3, OPEN-ENDED]

30. How could the alumni mentoring and peer-networking opportunities be improved? [DISPLAY IF Q27 = 1-3, OPEN-ENDED]

30. How could the community of practice (CoP) discussions be improved? [DISPLAY IF Q28 = 1-3, OPEN-ENDED]

Please indicate the level of contribution the NRC made to build your state’s capacity in the following areas in the last 12 months. Check only one response option. [MATRIX, FORCED RESPONSE]

	Made no contribution 1	A small contribution 2	A moderate difference that led to better results 3	A major contribution 4	Could not have achieved results without their assistance 5
Increase state capacity to use and select evidence-based practices (EBPs) (Increase in your state’s capacity to select and use evidence-based best practices that match the needs of the population[s] of focus)	□	□	□	□	□
Implement grant with quality and fidelity to the grant model (Your state’s capacity to implement your services and programs according to 5 SAMHSA Strategic Approaches and 7 SAMHSA Guiding Principles)	□	□	□	□	□
Be data driven (Your state’s capacity to collect and use data to make decisions)	□	□	□	□	□
Cultural linguistic competency (CLC) in all services (Your state’s capacity to provide services, supports, and programs that meet the cultural and linguistic needs of all populations and subpopulations)	□	□	□	□	□
Create a continuum of cross-sector system care – prenatal to 18 (Your state’s capacity to create a continuum of care across systems that include, but are not exclusive to, schools, mental health, law enforcement, juvenile justice, courts, child welfare, public health, counties, states)	□	□	□	□	□
Train workforce (Your state’s capacity to train your state’s workforce in practices, policies, and research that improve the lives of service recipients)	□	□	□	□	□
Demonstrated improvement in knowledge, attitudes, beliefs related to prevention and/or mental health promotion	□	□	□	□	□
Create multidisciplinary and multilevel partnerships (Your state’s capacity to create multilevel partnerships at the local, tribe, county, or state levels)	□	□	□	□	□
Apply evaluation results to adapt evidence-based interventions so they can be effective in diverse communities (Your state’s capacity to use data to adapt evidence-based or best practices so that they meet the specific needs of service recipients in your state and are appropriate to your population’s diversity)	□	□	□	□	□
Develop strategies to reduce health disparities (Your state’s capacity to develop strategies to reduce behavioral health disparities through system-level changes, including funding, policy, practice, training, capacity building, evaluation, resources)	□	□	□	□	□
Enhanced infrastructure to support sustained systems-level collaborations (Enhanced infrastructure and capacity in your state to support sustained systems-level collaboration, e.g., MOUs and MOAs)	□	□	□	□	□
Enhanced infrastructure to support sustained promotion of mental health and youth violence prevention (Enhanced infrastructure and capacity in your state to promote a public health approach to mental health promotion and youth violence prevention through targeted strategic communications, sometimes referred to as public relations, and social marketing or health communication campaigns)	□	□	□	□	□
Enhanced infrastructure to support expanded adoption and diffusion of effective program strategies (Enhanced infrastructure and capacity of your state to expand adoption and diffusion of evidence-based programs so that they meet the needs of service recipients and are appropriate to your state’s cultural context)	□	□	□	□	□
Enhanced infrastructure to support reduction/improvements that align with the Strategic Prevention Framework (Enhanced infrastructure and capacity in your state to follow the 5 steps of the Strategic Prevention Framework (assess needs, build capacity, plan, implement, and evaluate) with a focus on sustainability and cultural competence)	□	□	□	□	□
Enhanced infrastructure to support alignment with SAMHSA’s strategic initiatives (Enhanced infrastructure and capacity in your state to align with SAMHSA's 6 strategic initiatives [LIST])	□	□	□	□	□

[NEXT PAGE]

Please help us understand how the NRC made contributions to your capacity in the following areas in the last 12 months.

46. Please share an example of how the NRC helped your project use and implement EBPs. [DISPLAY IF Q39 = 4-5, OPEN-ENDED]

46. Please share an example of how the NRC helped your project to implement with quality and fidelity to the grant model. [DISPLAY IF Q40 = 4-5, OPEN-ENDED]

46. Please share an example of how the NRC helped your project to be data driven. [DISPLAY IF Q41 = 4-5, OPEN-ENDED]

46. Please share an example of how the NRC helped your project to practice cultural and linguistic competency in all services. [DISPLAY IF Q42 = 4-5, OPEN-ENDED]

46. Please share an example of how the NRC helped your project to create a continuum of cross-sector system care. [DISPLAY IF Q43 = 4-5, OPEN-ENDED]

46. Please share an example of how the NRC helped your project to train your state’s workforce. [DISPLAY IF Q44 = 4-5, OPEN-ENDED]

[NEXT PAGE]

46. In the past 12 months, did you receive T/TA services on any of the following capacity areas [QUESTIONS 39-54: LIST] from sources other than the NRC? [MULTIPLE CHOICE, FORCED RESPONSE]

• No

• Yes

46. Please describe T/TA services received from sources other than the NRC. [OPEN-ENDED, FORCED RESPONSE, DISPLAY IF Q60 = YES]

[NEXT PAGE]

The following questions ask you to report on your overall assessment of the services provided to you by the NRC.

46. Overall, how satisfied are you with the services provided by the NRC? [MULTIPLE CHOICE, FORCED RESPONSE]

• Not at all satisfied

• A little bit satisfied

• Somewhat satisfied

• Very satisfied

• Extremely satisfied

46. Overall, how useful have the services provided by the NRC been to you? [MULTIPLE CHOICE, FORCED RESPONSE]

• Not at all useful

• A little bit useful

• Somewhat useful

• Very useful

• Extremely useful

46. How often have you applied the information learned from the NRC? [MULTIPLE CHOICE, FORCED RESPONSE]

• Never

• Rarely

• Sometimes

• Often

• Very often

46. To what extent has the NRC increased your knowledge in the areas of prevention and/or mental health promotion? [MULTIPLE CHOICE, FORCED RESPONSE]

• Not at all

• A little bit

• Somewhat

• Very much

• Extremely

46. Please indicate any additional comments or suggestions you have about how the NRC can further improve its T/TA services. [OPEN-ENDED]

[NEXT PAGE]

Thank you for your participation!

This is the END of the survey.

You can still go back to review or amend your answers in this section by clicking the “Previous Page” button. Note: Please DO NOT use the back arrow in your browser.

You can also leave and reenter the survey by closing the browser and exiting the survey here without proceeding to the next page. You can return to this page by using the survey link provided to you.

However, please be aware that ONCE YOU CLICK THE “NEXT PAGE” BUTTON BELOW, YOU WILL HAVE SUBMITTED YOUR RESPONSES AND YOU WILL NOT BE ABLE TO GO BACK TO CHANGE ANY OF YOUR ANSWERS. If you do need to regain access to your survey after submitting it, please email TBD.

Also, if you have any questions regarding this survey, please contact survey manager TBD.

Thanks again!

Attachment 3 3