Program Evaluation for Prevention Contract Evaluation Activities
Supporting Statement
Using a census approach, the targeted universe for the PEPC SPF-Rx DMS is all SPF-Rx grantee project directors (N = 25) and grantee project directors from all future cohorts as well as each of the SPF-Rx grantees’ subrecipients (N=123). As grantees agreed to participate in cross-site evaluation data collection activities as a condition of funding, all 25 grantee project directors or other designated grantee staff are expected to complete the SPF-Rx cross-site instruments. The actual response rate for Year 1 of the evaluation was N=24, with one grantee not completing data collection. The response rate for Year 2 is expected to be N=25. The Year 1 total number of subrecipients is not representative of the total number for Years 2-5 of the evaluation because the subrecipients were in the process of being funded. In Year 2 of the evaluation, the evaluation team is working to obtain the Annual Implementation Instrument (AII) and Community-Level Outcomes data from each active subrecipient and expect the response rate to be N=123.
To meet its annual reporting requirements for GPRA and for performance measures, SAMHSA strives to obtain data from all grantees. The evaluation team works with SAMHSA to promote timely and accurate responses from grantees and subrecipients. Data sources vary for indicators reported on by SPF-Rx grantees. For example, some grantees are subject to laws and policies that restrict access to, and use of, PDMP data. With a larger N, the evaluation team is able to answer the evaluation questions (EQs) and assess the impact of the program, even when grantees and subrecipients encounter data challenges.
Pending OMB approval, the instruments will be collected according to the schedule outlined in Section A.2.
Online Data Collection
The SPF-Rx AII, Grantee- and Community-Level Outcomes Modules, are self-administered, web-based data collection tools completed through the DMS. The Grantee-Level Interview is a phone interview conducted by an evaluation team interviewer.
Grantee project directors or their designated staff complete the AII, Grantee- and Community-Level Outcomes. The AII is completed by subrecipient community project directors or their designees, although grantee staff have the option of completing for subrecipients. Before data collection begins, each respondent is provided a unique login to enter the DMS, which requires the creation of a password.
For SPF-Rx, the cross-site evaluation team developed two user manuals to assist grantees and subrecipients in completing data collection in the online DMS; one AII manual and one manual which covers the Grantee-and Community-Level Outcomes Modules. The team has also developed a Frequently Asked Questions (FAQ) document with common questions and challenges respondents encounter when completing the instruments. Each year, the evaluation team provides training webinars to grantees and subrecipients to walk through the online data collection system, review each instrument in detail, and go over data collection procedures.
The DMS is easy to use. The SPF-Rx evaluation team has implemented user-friendly features across all functional areas, taking into account the needs of both SAMHSA and grantees.
Respondents can easily request assistance by calling the toll-free number or sending an email request to the PEPC Help Desk. The toll-free line is routed to an email system that is checked regularly by members of the training and technical assistance team, so voicemail messages are transmitted via email even when PEPC staff are not actively monitoring the phone line. Staff responding to requests are trained in the data collection instruments and use of the DMS. Support staff have ready access to each of the resources available (e.g., manuals, FAQ), and have a system for referring difficult data or technology issues to the appropriate PEPC team subject matter experts for further assistance.
Grantee-Level Interviews
The PEPC team will interview grantee project directors or their designated staff in the final year of the grant (Year 4 of the evaluation), following the protocol outlined in Attachment 5. The PEPC team will contact grantee project directors via email to set up interviews during regularly scheduled business hours. Before conducting the interview, the PEPC team will review the baseline Grantee-Level Interviews and other SPARS reporting data already provided by the grantee (e.g., progress reports and strategic plans). To reduce respondent’s burden, relevant information will be abstracted to familiarize the interviewer with the grantee and to avoid asking redundant questions. Once the interview is scheduled, the PEPC team will provide the grantee project director or designated respondent with an electronic version of the consent form and a Grantee-Level Interview protocol for review. This ensures the participant is prepared to answer the questions or is able to include the correct team member to respond to the questions.
Before beginning each interview, the interviewer will request consent to record the interview as a method of confirming, if needed, the accuracy of noted responses. A senior evaluator from the PEPC team will lead the respondent through the interview while a junior evaluator records responses and takes notes. Notes from the interview will be maintained on a password-protected, secure server, accessible only to members of the PEPC team who need access and have completed the required security and privacy training. After the interview, the interviewer will send an email thanking the respondent for their participation. The PEPC team will transcribe the interview, and then review the transcript for accuracy. The PEPC team will then send the transcript to the grantee for review. This is to ensure that the transcript is accurate and that the grantee does not wish to edit or to provide different or additional context for any responses. Once the transcripts are considered final, the recording will be deleted. An electronic version of the transcript will be maintained on a password-protected, secure server accessible only to the PEPC team who require access and have completed the necessary security and privacy training.
The terms and conditions of the grant awards require grantees to participate in all data collection activities. The PEPC team will employ a number of strategies to help ensure that grantees participate in all required data collections for the SPF-Rx evaluation.
The SPF-Rx program requires participation in data collection as a condition of the grant. The evaluation team received 24 out of 25 grantee responses to the AII and Grantee- and Community-Level Outcomes Modules in Year 1, and expects all 25 grantees to report these data in Year 2 of the evaluation. All 25 of the SPF-Rx grantees participated in the Grantee-Level Interview in Years 1 and 2 of the evaluation. The evaluation team expects the same response rate going forward and will employ the below strategies to maximize the response rate.
AII, Grantee- and Community-Level Outcomes
As described above, the PEPC team developed user manuals for accessing and navigating the DMS online data collection system and an FAQ document to help respondents accurately complete the online instruments. These documents are disseminated regularly to the SPF-Rx grantees and their subrecipients, and are available by contacting the PEPC help desk. Grantees are provided training webinars to walk through the DMS online data collection system and to review data collection procedures. The PEPC team has a help desk email and phone number, with team members available to answer data and technology-related questions. During the data reporting period, the PEPC team sends regular (at least weekly) reminder emails with resources to encourage on-time reporting.
The PEPC team will work with SAMHSA project officers to identify grantees’ data availability concerns early and identify ways to help grantees obtain and report required data. Approximately 1 week after a data submission deadline, the PEPC team provides project officers a list of past-due instruments. SAMHSA project officers follow up with their grantees to ensure submission. Additionally, during the data cleaning process, the PEPC team provides Interim Data Feedback Forms to document missing data elements. The PEPC team follows up regularly with the grantees to remind them to finalize data entry, requiring Interim Data Feedback Forms to be completed and returned within 2 weeks. SAMHSA project officers are copied on these correspondences and alerted to grantees who have not completed data collection. SAMHSA project officers are asked to do additional outreach to their grantees to remind them of the grant’s data reporting requirements and urge them to enter their data. This system is already successfully established and runs smoothly for the current SPF-Rx data collection.
Grantee-Level Interviews
All grantee project directors are required to participate in the Grantee-Level Interview. The PEPC team will ensure participation by outreaching to project directors by email to schedule interviews. If there is no reply within 1 week, a second reminder email will be sent. If this does not elicit a response, the PEPC team will follow up with phone calls to the project director. SAMHSA project officers will be copied on all correspondence and will be alerted to grantees who have not replied. If needed, the PEPC team will ask the project officers to outreach to their grantees and urge them to schedule the interview with the PEPC team. The evaluation team successfully completed baseline Grantee-Level Interviews with all SPF-Rx grantees using these methods.
Each of the SPF-Rx data collection instruments were implemented with current SPF-Rx grantees. Based on Year 1 and Year 2 of data collection for the AII, Grantee- and Community-Level Outcomes Module, Substitute Data Source Request form, and Grantee-Level Interview, revisions were made to the instruments to eliminate duplication with SPARS and unnecessary data collection elements. The changes to the instruments and the related time burdens are outlined in section A.
The PEPC team comprises the following team members who will serve as statistical consultants for the SPF-Rx evaluation. Exhibit 11 provides details of these team members and advisors.
Exhibit 11. Statistical Consultants for the SPF-Rx Cross-Site Evaluation through the Program Evaluation for Prevention Contract
Name & Role |
Title & Address |
Contact Information |
PEPC Evaluation Staff |
||
Dana Hunt, PhD PEPC Project Director |
Principal Associate Abt Associates 10 Fawcett Street, Cambridge, MA 02138 |
Telephone: (617) 349–2733 Email: Dana_Hunt@abtassoc.com |
Jennifer Carter, MPH Project Manager |
Associate 6130 Executive Blvd., Rockville, MD 20852 |
Telephone: (301) 347-5640 Email: Jennifer_Carter@abtassoc.com |
Morris Hamilton, PhD |
Senior Analyst Abt Associates 5001 S. Miami Blvd, Ste. 200 Durham, NC 27713 |
Telephone: (919) 294-7742 Email: Morris_Hamilton@abtassoc.com |
Rayan Joneydi, PhD |
Senior Analyst Abt Associates 5001 S. Miami Blvd, Ste. 200 Durham, NC 27713 |
Telephone: (919) 294-7742 Email: Rayan_Joneydi@abtassoc.com |
Mary Juergens, MPH |
Analyst Abt Associates 10 Fawcett Street, Cambridge, MA 02138 |
Telephone: (617) 520-3094 Email: Mary_Juergens@abtassoc.com |
Sarah Steverman, PhD, MSW |
Associate Abt Associates 10 Fawcett Street, Cambridge, MA 02138 |
Telephone: (301) 347-5045 Email: Sarah_Steverman@abtassoc.com |
Lisanne Brown, PhD, MOH |
Principal Associate Abt Associates 10 Fawcett Street, Cambridge, MA 02138 |
Telephone: (303) 381-8266 Email: Lisanne_Brown@abtassoc.com |
Sarah Hamad |
Research Assistant 6130 Executive Blvd., Rockville, MD 20852 |
Telephone: (301) 347-5336 Email: Sarah_Hamad@abtassoc.com |
Government Project Officers |
||
Stephanie Ziomek
|
Social Science Analyst Substance Abuse and Mental Health Services Administration (SAMHSA) 5600 Fishers Lane, #15E29D Rockville, MD 20857 |
Telephone: (240) 276-1623 Email: Stephanie.Ziomek@samhsa.hhs.gov |
Jon Dunbar-Cooper |
Public Health Analyst Division of Systems Development Substance Abuse and Mental Health Services Administration (SAMHSA) 5600 Fishers Lane, #16E85D Rockville, MD 20879 |
Telephone: 240-276-2573 Email: Jon.Dunbar@samhsa.gov |
Centers for Disease Control and Prevention. 2012. CDC grand rounds: Prescription drug overdoses - a U.S. epidemic. Morbidity and Mortality Weekly Report, 61(1), 10–13.
Centers for Disease Control and Prevention. 2015. Increases in fentanyl drug confiscations and fentanyl-related overdose fatalities. HAN Health Advisory. Retrieved from http://emergency.cdc.gov/han/han00384.asp. Accessed on March 27, 2019.
Hedegaard, H., Minino, A. M., &Warner, M. 2018. Drug overdose deaths in the United States, 1999-2017. NCHS Data Brief No. 329. Atlanta, GA: Centers for Disease Control and Prevention.
Rudd, R. A., Aleshire, N., Zibbell, J. E., & Gladden, M. 2016. Increases in drug and opioid overdose deaths – United States, 2000-2014. Morbidity and Mortality Weekly Report 64(50), 1378-1382.
Substance Abuse and Mental Health Services Administration. (2019). Key substance use and mental health indicators in the United States: Results from the 2018 National Survey on Drug Use and Health (HHS Publication No. PEP19 5068, NSDUH Series H 54). Rockville, MD: Center for Behavioral Health Statistics and Quality, Substance Abuse and Mental Health Services Administration. Retrieved from https://www.samhsa.gov/data/
Annual Implementation Instrument (Attachment 1)
Grantee-Level Outcomes Module (Attachment 2)
Community-Level Outcomes Module (Attachment 3)
Grantee-Level Interview (Attachment 4)
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | King, Summer (SAMHSA/OPPI) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |