Described below are the respondents and sampling methods for each data collection tool included in this request to OMB for the PATH Evaluation.
Web Surveys
The target universe for the State PATH Contact (SPC) Web Survey is the SPCs or comparable staff for all PATH grantees (i.e., the 50 States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands). The total expected number of respondents is 56.
The target universe for the PATH Intermediary Web Survey is representatives from intermediary organizations (i.e., County agencies or regional behavioral health authorities) through which PATH grantees provides PATH funds through to PATH providers. The total expected number of responses is 28.
The target universe for the PATH Provider Web Survey is the PATH Project Director/Administrator or comparable staff for all local, public or non-profit organizations that receive PATH funds from the PATH grantees. The total expected number of responses is not expected to be more than 500.
Site Visits
The sample for the site visits will be purposive. The approach for selecting PATH grantees for the site visits includes geographic spread, including grantees with a good mix of providers in rural and urban counties and a spread of funding levels, and visiting one to two minimum-allocation states (i.e., receiving $300,000 PATH funds annually). As possible, grantees that were visited for the previous PATH evaluation will not be included. For each of the grantees selected, one to two PATH providers will be selected for site visits. The contractor staff will work with the SPC to identify the most appropriate providers. In selecting providers, the team will attempt to identify providers that vary in terms of populations served and service strategies used.
Below we describe the respondent targets for each of the site visit instruments.
Opening Session with State Discussion Guide: The target universe is up to five relevant grantee staff (as determined by the SPC) from each of the 5 PATH grantees that will receive site visits. The staff may include directors and supervisors of the PATH program or the SPC. The total expected number of attendees is 25.
SPC Discussion Guide: The target universe is the SPCs from the 5 PATH grantees that will receive site visits. The total expected number of respondents is 5.
State and Provider Stakeholder Discussion Guide: The target universe is up to five state staff either from the grantee agency or other agencies and with stakeholders of the PATH program. The staff may be involved with intermediary organizations, housing, HMIS or CoCs. At the provider level, the target universe is up to five stakeholders of the PATH provider from up to 2 providers from each of the five PATH grantees that will receive site visits. The stakeholders will be identified by the grantees and PATH providers. The total expected number of respondents is 75 (25 for the state and 50 for the provider).
Opening Session with PATH Provider Leadership Staff Discussion Guide: The target universe is up to five staff from up to 2 providers from each five PATH grantees that will receive site visits. The total expected number of respondents is 50.
PATH Provider Project Director (PD) Discussion Guide: The target universe is one PD or comparable staff from up to 2 providers from each five PATH grantees that will receive site visits. The total expected number of respondents is 10.
PATH Provider Direct Care Staff Discussion Guide: The target universe is up to five outreach workers, case managers or direct care staff from up to 2 providers from each five PATH grantees that will receive site visits. The staff will be identified by the programs. The total expected number of respondents is 50.
PATH Consumer Focus Group Discussion Guide: The target universe is up to 10 current or former consumers from up to 2 providers from each five PATH grantees that will receive site visits. The total expected number of respondents is 100. PATH program staff will ask current consumers who are available at the time of the focus group if they would like to participate in the focus group. The PATH program staff will contact former PATH consumers, whom they are still in contact with, to determine if they would like to participate in the focus group and if they are available during the day and time the focus group is scheduled.
Telephone Interviews
The sample for the telephone interviews will be purposive. First, PATH providers will be selected for the telephone interviews. Once the PATH providers have been identified, then their SPCs and intermediary organizations (if relevant) will be included in the sample for the telephone interviews.
The approach for selecting PATH providers for the telephone interviews will be to identify providers that demonstrate either very high or very low scores on study measures that show high levels of variation. The interviews will be used to obtain more in-depth information on the facilitators that contribute to high scores (e.g. efficient use of information, use of evidence based practices, the existence of well-organized and active CoCs, etc.). Similarly, providers who score low on these measures will be queried about the major barriers they encounter related to specific measures.
The target universe for the PATH Telephone Interview Guide includes three different types of respondents and is as follows:
The target universe for the PATH Provider Telephone Interviews are the PATH Project Director/Administrator or comparable staff from a sample of local, public or non-profit organizations that receive PATH funds from the PATH grantees. The total expected number of responses is 60.
The target universe for the State PATH Contact (SPC) Telephone Interviews are the SPCs or comparable staff for a sample of PATH grantees (i.e., the 50 States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, Guam, American Samoa, and the Commonwealth of the Northern Mariana Islands). Included in the sample, will be the SPC for the selected grantees. The total expected number of respondents is 28.
The target universe for the PATH Intermediary Telephone Interviews are representatives (e.g., directors, managers or other relevant staff) from a sample of from intermediary organizations (i.e., county agencies or regional behavioral health authorities) through which PATH grantees provides PATH funds through to PATH providers. Included in the sample, will be intermediary agencies for the corresponding PATH grantees and providers. The total expected number of responses is 14.
Table 1 shows the number of entities in the universe and total number to be sampled for the proposed data collection activities. Also, shown in Table 1 are the response rates achieved for applicable data collection activities during the last evaluation as well as the response rates expected for the proposed evaluation. It is expected that the response rates for the proposed evaluation will be equal to or greater than those obtained in the last evaluation since data collection will only occur for the PATH program and more follow-up will be conducted than was conducted in the previous evaluation. During the last evaluation, the same survey was conducted with providers that operated other programs in addition to PATH and may have thought that one response was sufficient. See Sections 2 and 3 for methods that will be utilized to maximize response rates.
Table 1. Universe and Proposed Sample Sizes for Proposed Data Collection Activities and Response Rates for Previous and Proposed PATH Evaluation
Data Collection Instrument |
Total Number in Universe |
Total Number to be sampled |
Response Rate for Previous PATH Evaluation |
Response Rates for Proposed PATH Evaluation |
Web Surveys |
|
|||
SPC Web Survey |
56 |
56 |
77% |
77%-80% |
PATH Intermediary Web Survey |
Unknown |
28 |
Not conducted |
75%-80% |
PATH Provider Web Survey |
Close to 500 |
Close to 500 |
61%* |
75%-80%* |
Site Visits |
|
|||
Opening Session with State Staff Discussion Guide |
Unknown |
25 |
Not Applicable |
Not Applicable |
SPC Discussion Guide |
5 |
5 |
Not Applicable |
Not Applicable |
State and Providers Stakeholder Discussion Guide |
Unknown |
75 |
Not Applicable |
Not Applicable |
Opening Session with PATH Provider Staff Discussion Guide |
Unknown |
50 |
Not Applicable |
Not Applicable |
PATH Provider PD Discussion Guide |
Unknown |
10 |
Not Applicable |
Not Applicable |
PATH Provider Outreach Worker/Case Manager/Direct Care Staff Discussion Guide: |
Unknown |
50 |
Not Applicable |
Not Applicable |
PATH Consumer Focus Group Discussion Guide |
Unknown |
Up to 100 |
Not Applicable |
Not Applicable |
Telephone Interviews |
|
|||
PATH SPC Telephone Interviews |
56 |
28 |
80% |
80% |
PATH Intermediary Telephone Interviews |
Unknown |
14 |
Not conducted |
75%-80% |
PATH Provider Telephone Interviews
|
Close to 500 |
60 |
Not conducted |
75%-80% |
*It is suspected that the response rate for the PATH provider web survey was low, because collection for non-PATH programs in the previous multi-program evaluation took place with the same survey and providers that had more than one SAMHSA Homeless program may have thought that one response was sufficient. The survey also included questions which were not as relevant to PATH programs. More follow-up will be conducted with PATH providers during the next round of data collection.
Web Surveys
The contractor will obtain contact information for all of the SPCs and the PATH providers from SAMHSA or from the following link, where it is publicly available: http://www.samhsa.gov/homelessness-programs-resources/grants-programs-services/path-program/state-provider-contacts. Contact information for the intermediary organizations will be obtained from the SPCs. The same procedures noted above will be used for all three target populations. Emails will be sent to the designated respondents to inform them that data collection has started. A link to enter the Web-based survey will be included in the email, as well as information on the background, purpose, types of questions, and length of the survey (see Attachment 6). When the survey is first accessed, the consent form will appear and the respondent will not be able proceed until the consent form is completed. If the respondent declines, a log-off message will appear; if consent is provided, the survey will begin. When respondents complete the survey a “thank you” email will be sent automatically (See Attachment 6). Non-responders will be sent predefined, automated weekly emails as needed, to remind them to complete their survey (see Attachment 6). If the survey is not completed within 3 weeks, the respondent will be contacted via telephone by the contractor staff.
Site Visits
As noted above, the sample for the site visits will be purposive. Once the PATH grantees have been selected for a site visit, the contractor staff will send an email to the SPC to inform them of the selection for the site visits and to provide information on the purpose, types of sessions, and length of the site visit (see Attachment 6). Once the SPC agrees to a site visit, a pre-site visit call will be arranged which may include grantee and provider staff. The contractor staff will work with the SPC and provider staff to finalize the meeting agenda and schedule and to identify the participants for each session.
Informed consent will be obtained from all participants of the site visit discussions and the consumer focus groups (see Attachment 4). Signed consents forms will be collected before the session or focus group formally begins and the tape recorder is turned on (if all interviewees consent). Notes will be taken during each session and the consumer focus groups and summarized. The recordings will be reviewed as needed to clarify the summary notes. Data collected from the site visits will be stored in files that are password protected and access will be limited to individuals who have a need to work on them. All hard copies of forms and notes will be kept in secure, locked cabinets and scanned versions will be stored in files that are password protected. Notes from the site visit discussions and consumer focus groups will be inductively analyzed with the assistance of software such as NVivo or Dedoose. A contractor staff that conducted site visits for the last PATH evaluation, will be conducting the site visits in partnership with another contractor staff.
Focus groups with consumers (project participants) will be conducted during each site visit. The purpose of these focus groups is to understand services from a consumer perspective, including what services they received and satisfaction with services. The focus group participants will be identified by local PATH program staff. The participants will be provided with a $20 gift card for a restaurant or store selected by program staff. The focus group participants will be either current or former PATH consumers. The focus groups will be open discussions structured by an open-ended discussion guide. As noted above, the notes from the focus groups will be inductively analyzed with the assistance of software such as NVivo or Dedoose. This will allow us to identify major themes.
Telephone Interviews
As noted above, the sample for the telephone interviews will be purposive. Once the PATH providers and their corresponding SPCs and intermediary organizations have been selected, the contractor staff will send an email informing them of their selection for the telephone interviews and to provide information on the purpose, types of questions, and length of the telephone interviews (see Attachment 6). The contractor staff will schedule the telephone interview at a time that is most convenient for the respondent. Informed consent will be obtained from all participants of the telephone interviews (see Attachments 5). The informed consents will be read prior to the interview. The telephone interviews will be recorded, if consented by the interviewees. Notes will also be taken during each interview. The recordings will be reviewed as needed to clarify the notes. Data collected from the telephone interviews will be stored in files that are password protected and access will be limited to individuals who have a need to work on them. All hardcopies of forms and notes will be kept in secure, locked cabinets and scanned versions will be stored in files that are password protected. Notes from the telephone interviews will also be inductively analyzed with the assistance of software such as NVivo or Dedoose. A contractor staff that conducted telephone interviews for the last PATH evaluation will be conducting the telephone interviews in collaboration with other contractor staff who were trained and conducted telephone interviews for the cross-program National Evaluation of SAMHSA Homeless Programs which included the PATH program.
Prior to beginning the PATH evaluation, all PATH Grantees’ SPCs will be sent an email from SAMHSA noting the importance of the PATH evaluation and encouraging their participation and that of their intermediaries and providers.
All web surveys and the telephone interviews will include multiple requests for participation to maximize response rates. As noted above, the recruitment process for the web surveys will include sending all potential respondents a recruitment email and weekly follow-up emails that include a link to access the survey. Respondents that start but do not complete the survey will receive a ‘partial responder' email (see Attachment 6). The recruitment process for the telephone interviews will also involve sending all potential respondents an introductory email to describe the purpose of the interview and to request information on potential dates for conducting the interview. Individuals who do not respond to the introductory email will receive up to two follow-up emails and one phone call (see Attachment 6).
Utilized will be web survey, telephone interview and site visit protocols that were developed for the cross-program National Evaluation of SAMHSA Homeless Programs and that were utilized for the last PATH evaluation. Those instruments and protocols have been revised to ensure that only information that is needed for the PATH evaluation is collected and to remove questions or sections that were more relevant to the other SAMHSA Homeless Programs.
The contractor for the evaluation has significant experience with qualitative and quantitative data collection and analysis. SAMHSA staff and the Technical Panel (described in A.8. Consultation Outside the Agency) have been an important part of the evaluation process providing feedback concerning the evaluation plan. The evaluation personnel from the contractor and SAMHSA are noted in Table 2.
Table 2. Statistical Consultants for the PATH Evaluation
Expert |
Affiliation/Contact Information |
PATH Evaluation Staff |
|
Virginia Mulkern, PhD Project Director |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2315 Email: vmulkern@hsri.org |
Teresita
Camacho-Gonsalves, PhD |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2504 Email: tcamacho@hsri.org |
David Hughes, PhD Senior Research Specialist |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2527 Email: dhughes@hsri.org |
Nilufer Isvan, PhD Senior Research Fellow- Quantitative Research Expert |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2505 Email: nisvan@hsri.org |
Ben Cichocki, ScD Research Associate - Qualitative Research Expert |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2502 Email: bcichocki@hsri.org |
Rachael Gerber, MPH Research Associate |
Human Services Research Institute 2336 Massachusetts Ave., Cambridge, MA 02140 Phone: 617-844-2332 Email: rgerber@hsri.org |
SAMHSA Advisors |
|
Kirstin Painter, Ph.D. Contractor Officer Representative |
Public Health Advisor CMHS, SAMHSA 5600 Fishers Lane, Room 14E89D, Rockville, MD 20857 Phone: 240-276-1932 |
Sarah Ndiangui, MPH Alternate Contracting Officer Representative |
Public Health Advisor CSAT, SAMHSA 5600 Fishers Lane, Room 13E85B, Rockville, MD 20857 Phone: 240-276-2918 |
Caroline B. Fernandez Director,
PATH Program |
Public Health Advisor CMHS, SAMHSA 5600 Fishers Lane, Room 14N34F, Rockville, MD 20857 Phone: 240-276-1625 |
References
Charmaz, K. (2006). Constructing grounded theory. London: Sage.
U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration. (2016). FY 2016 PATH Funding Opportunity Announcement (FOA)). No. SM·16·F2 Catalogue of Federal Domestic Assistance (CFDA) No: 93.150
Attachment 1: State PATH Contact Web Survey
Attachment 2: PATH Intermediary Web Survey
Attachment 3: PATH Provider Web Survey
Attachment 4: PATH Site Visit Discussion Guide
Attachment 5: PATH Telephone Interview Guide
Attachment 6: Sample Correspondence
Attachment 7: Sample Tables
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Teresita Camacho-Gonsalves |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |