Supporting Statement to Office of Management and Budget
This evaluation is based on a mixed methodology. The quantitative data include both secondary source information collected annually as part of the PATH program and related data from other Federal data collections and primary survey data collection efforts. The qualitative evaluation component will use data extracted from site applications and data collected from site visit assessments.
The PATH program has a universe of 56 funding jurisdictions: 50 States, the District of Columbia, Puerto Rico, and 4 U.S. Territories. Each funding jurisdiction or site has local PATH-funded providers that provide direct services. The number of local PATH-funded providers varies based on the state/territory and is based on the poverty level of the population in that region. The number of locally funded PATH providers remains relatively invariant over time since the resources for PATH formula grants have also remained relatively constant over time. PATH program characteristics and outcome data will be extracted from applications and reports from all States and Territories and providers. No sampling is required; all members of the potential respondent universe will participate in this part of the assessment. Samples drawn from the potential universe of respondents will be required for primary data collections using 2 online surveys and 10 site visits to SPCs and 10 site visits to 10 PATH providers in those States selected for site visits.
Online Surveys
An online survey will be conducted with the entire universe of 56 SPCs. Additionally, a sample of locally funded providers will be surveyed. To generate a fully representative, self-weighting sample of providers, a respondent providing direct services will be identified from among the cadre of staff supported by PATH funds and providing direct, face-to-face services. Each SPC will receive instructions and telephone guidance on how the local provider sample is to be selected. We will ask the SPC to support the survey and encourage provider response. SPCs will receive copies of all correspondence with providers. Each Provider agency will receive instructions from the contractor detailing how to array eligible staff and make a random draw of the identified respondent. The contractor will provide unique identifiers that will be assigned by the Provider to each respondent. Instructions will specify that the survey will take approximately 45 minutes and should be completed within a 14 day calendar period subsequent to the respondent receiving their unique identification number and URL for the online survey site. We will track unique identifiers and contact providers if respondents do not complete the survey in a timely manner.
Site Visit Interview Guides
The universe for potential site visits does not include the four U.S. Territories or Puerto Rico because of both cost and limited generalizability from findings drawn from these sites to the continental United States where most funds (98.1%) are expended. Puerto Rico and the other Territories also have unique homelessness issues as a result of their geography, culture, systems of care and other differences that would provide limited information about the overall PATH program. The cost of site visits relative to potential return on investment would be prohibitive. However, Puerto Rico and the four U.S. Territories will be included in both data extraction efforts and online surveys.
Pilot-testing of all instruments took place during site visits to four states at the beginning of Year 2. Four SPCs and eight local providers within the four States were selected to review all questions and provide feedback on the types of questions asked as well as the comprehensiveness and clarity in all survey instruments. Level of effort and burden were assessed, and instruments were revised to incorporate user feedback.
During the third year of the contract, 10 site visits will be conducted. These visits will focus on identifying PATH implementation models and challenges and best practices at the State and local provider level. Focus groups with consumers and interviews with SPCs and providers will identify successes, barriers to success, and perceptions of PATH operations, services, and effectiveness. Ideally, site visits will provide clear and tangible data concerning what works, what is needed, and how to improve PATH operations.
Ten sites will be selected from the remaining 47 funded sites—the 50 States and the District of Columbia excluding the 4 first-year site visit locations. The 10 sites will be selected from a stratified random sample of the 47 eligible PATH initiative State sites. The stratification will place the sites in a matrix using region, urbanicity, poverty level, funding level, and special populations served as potential selection criteria. The 10 sites will be chosen randomly from the completed sampling matrix to provide good representation of the breadth of PATH initiatives. Final site selection will be approved by the SAMHSA Task Order Officer and the PATH Technical Advisory Panel.
The 10 site visits will include an interview with the SPC and between 2 and 6 in-person interviews with locally funded PATH providers. PATH-funded efforts range from having 1 provider (e.g., Alaska) to 44 (e.g., California) locally funded providers in each State. In instances where the number of providers is large (>10), and when near the location of the site visit, providers may be interviewed in person using a group format. When providers are spread over a large geographic area, interviews may need to be held by audio/video conference calling. There will be 1–2 focus groups including a total of up to 12 PATH clients. These groups will be held at the provider site during the in-person site visit. Clients will be selected from those receiving services at the provider location on the final 2 days of the 3-day assessment. A convenience sample of clients will be used. Service providers will be asked to refer consumers to one of the site visitors who will solicit their participation. Participation will be incentivized. The incentive will have less than a U.S. $20 value and will be approved by the provider prior to soliciting consumer participation.
Online Surveys
A complete census of SPCs will be required. Each SPC will be asked to respond to the online SPC survey. The universe of SPCs is limited since there is only one representative for each State or Territory. If the response rate is 80%, the sample would be limited to 45 respondents. However, every effort will be made to secure a complete canvass.
Like SPCs, we will attempt to canvass provider organizations. Each SPC will receive instructions and telephone guidance on how local provider sample is to be selected. We will ask the SPC to support the survey and encourage provider response. SPCs will receive copies of all correspondence with providers. Each provider will be given instructions on how to array all staff, both funded at least partially with PATH dollars and performing direct face-to-face services (e.g., outreach, case management, assessment) with clients. The provider will also receive instructions on how to select a potential respondent randomly from the list of those eligible. For example, each provider site will enumerate a list of eligible respondents and organize them alphabetically by first name. There should be no more than one or two eligible respondents for each provider in the universe. Providers will contact the identified respondent with the URL and password for the survey and urge him/her to participate (introductory letter template appears in Attachment G).
State PATH Contact and PATH Provider Interviews
A total of 25% of PATH sites (14 of 56 sites) will receive a site visit; this includes the 4 pilot site visits that have already been completed and 10 site visits during Year 3 that have not yet been conducted. With an examination of one-quarter of the sites, the site visits will provide a wide range of first-hand qualitative data collected through interviews and focus groups.
During Year 2, site selection (10 sites) for the formal assessment will be made from a stratified random sample. The stratification will place the sites in a matrix based on the State/region, level of urbanicity, poverty level, funding, and special populations served. The 10 sites will be chosen randomly within the different elements. Each site visit will include interviews with SPCs and key staff implementing PATH at the State level, local PATH-funded providers, and PATH-funded service recipients/advocates.
As part of the pretesting effort during Year 2, each site and the corresponding SPC was purposely selected from a stratified sampling frame. The stratification was based on the sites’ region, level of urbanicity, and past success as viewed by the PATH Technical Panel. The PATH Technical Panel, SAMHSA Task Order Officer, and contractor staff selected four sites that have solid experience in PATH implementation that were willing to provide feedback on the initiative process and functioning and on data collection protocols and instrumentation. Stratification ensured that site visit assessments reflected a broad a range of sites based on program context and focus.
During Years 2 through 3, site selection (10 sites) for the formal assessment will be made from a stratified random sample. The stratification will place the sites in a matrix based on the State/region, level of urbanicity, poverty level, funding, and special populations served. The 10 sites will be chosen randomly within the different elements. Each site visit will include interviews with SPCs and key staff implementing PATH at the State level, local PATH-funded providers, and PATH-funded service recipients. These participants will be identified for participation using a snowball sample to ensure they are the most qualified of the possible participants. The qualifications include experience as either a service provider or a service recipient and experience with advocacy for homelessness issues. The SPC will recommend the local PATH-funded providers. These providers will then recommend PATH-funded service recipients. There are no known deficiencies with the sampling frame, and there are no issues with timeliness or completeness. The focus of these in-person assessments is to provide qualitative information regarding the process, experience, and challenges of program implementation, service provision, and receipt of services. The large sample of sites from a stratified random sampling matrix should assure the selection of sample that is representative of the population on factors that have been found to differentiate implementation process (e.g., rural versus urban). An introductory letter template appears in Attachment G.
Focus Groups with PATH Clients
Each site visit will include 1–2 focus groups with up to a total of 12 PATH-funded clients/consumers. These participants will be identified for participation using a purposive sample to ensure they are the most qualified of the possible participants. The qualifications will include experience as a service recipient and diagnosis of a mental illness. Providers will identify clients to participate from among those present on site and receiving services.
Up to 12 consumers will be interviewed at each site assessment in 1 or 2 groups, depending on timing, availability of clients, and logistics. Each focus group is time-limited (90 minutes).
Online Surveys
The response rates are anticipated to be high, especially for the SPCs who work directly with SAMHSA. At least an 80% response rate will be acceptable for both SPCs and local providers. However we anticipate a 100% response rate from SPCs and will replace Provider survey sample to achieve a canvass
Several methods of maximizing the response rates on the survey for the SPCs and locally funded providers have been planned. First, the importance of an accurate and detailed evaluation will be emphasized. Federal Project Officers will convey the importance of completing the survey to the SPCs who will in turn convey the importance and urgency of responding to the providers.
The survey format, which will be available online, will allow participants to complete the survey on a flexible time schedule. The survey will provide flexibility to be completed over multiple logins using the same password.
A formative pretest of the interview and survey protocols and instruments was conducted with the PATH Expert Technical Panel. The panel’s feedback was incorporated in revised instruments. A formal pilot-test of protocols and instruments was conducted using four SPCs and four PATH provider sites. The pilot-test ensured questions were appropriate and concise. Response burden was estimated as part of the pilot-testing procedures.
If SPCs or locally funded providers fail to respond, the data collection team will contact the identified respondents in writing and by telephone. If response is below the necessary 80% and reminders have not improved response rates, the Government Project Officers will contact the SPCs asking them to respond and/or solicit their help in securing responses from locally funded PATH providers.
Site Visit Interview Guides
Site visit participants include SPCs, local PATH-funded providers, and PATH clients. To make the site visits less burdensome and to increase response rates, the scheduling and location of the interviews will accommodate the participants’ requirements. The scheduling of the site visits and related interviews will be based on the SPCs’ and providers’ needs. The location of the interviews and focus groups will be based on the site’s geography and the needs of the providers. Telecommunications, electronic mail, and audio/video discussions via the Internet will be used to reduce the travel burdens when necessary.
The participation of PATH clients in the focus groups will likely require some incentive. The PATH client focus groups will be held on site during the in-person assessment. Service providers will be asked to refer consumers to participate. The participants will receive an incentive that has a value of less than U.S. $20 and will be approved by the provider prior to soliciting consumer participation.
Both SAMHSA/CMHS PATH Government Task Order Officers and the PATH Technical Panel reviewed all materials used in the initial four pilot-tests. Their comments and concerns were addressed in the revisions made to the data collection protocols and instrumentation used in more formal pilot-testing. Pilot-tests of the instruments occurred during the Year 2 site visits. The four initial site visits resulted in feedback from a total of four SPCs and eight providers. The four initially selected sites represented a purposeful sample. Selections were made to ensure that sites represented key program discriminators (e.g., region, urbanicity, perceived need, innovation, effectiveness in implementation) and to realize the best potential for receiving feedback on protocols, instrumentation, and relevant PATH program information. The SAMHSA PATH Government Task Order Officers and the PATH Technical Panel worked to identify the best sites for pilot-testing data collection protocols and instrumentation.
Online Surveys
Copies of the online surveys were sent to the site prior to the pilot site visit. While on site, the surveys were administered, and each item was discussed with the respondents to identify areas of potential confusion, informational areas and/or sources that may have been omitted, and suggestions for improvement. Both SPCs and providers were questioned about the data collection method, survey timing, and ideal respondents at the local level. The finalized survey instruments attached to this submission have been adjusted based on feedback from the SPCs and local providers.
Site Visit Interview Guides
Pilot sites were chosen randomly using criteria such as geographic distribution, urbanicity, poverty levels, and levels of PATH funding. Prior to each site visit, all respondents were sent copies of the online survey to complete as well as the interview questions to be asked during the site visit.
At each of the year two pilot site visits the SPC and two local PATH-funded providers were interviewed and asked to review and comment on the focus group discussion guides, as appropriate. These respondents were asked not just topics for inclusion but also appropriate language and potential sensitivities of consumer focus group respondents. Feedback from these interviews has been incorporated in the finalized instruments attached to this submission.
The contractor for the evaluation has had ample experience with qualitative and quantitative data collection and analysis. SAMHSA staff and the Technical Panel have been an important part of the evaluation process providing feedback concerning the evaluation plan, determination of sites for pilot tests, and data collection procedures. See Table 3.
Table 3. Evaluation Personnel
Name |
Title |
Organization |
Contact |
Contract Staff |
|||
Lisa Kleppel, MPH |
Project Director |
MANILA Consulting Group, Inc |
571-633-9797 x 209 lkleppel@manilaconsuting.net |
Paul Brounstein, PhD |
Technical Director |
MANILA Consulting Group, Inc |
571-633-9797 x 234 pbrounstein@manilaconsulting.net |
Julie Gloudemans, MS |
Senior Program Analyst |
MANILA Consulting Group, Inc |
571-633-9797 c/o x 234 jgloudemans@manilaconsulting.net |
Frances Wallace Bailey, MPH |
Senior Program Analyst |
MANILA Consulting Group, Inc |
571-633-9797 c/o x 234 fwallace@manilaconsulting.net |
SAMHSA Staff |
|||
Pamela Fischer, PhD |
Government Project Officer |
SAMHSA, CMHS |
240-276-1901 |
Deborah Stone, PhD |
Social Science Analyst |
SAMHSA, CMHS |
240-276-2411 |
List of Attachments
Attachment A—State PATH Contact Online Survey
Attachment B—PATH Provider Online Survey
Attachment C—PATH Site Visit Interview Guide: State PATH Contact
Attachment D—PATH Provider Site Visit Interview Guide: Supervisor/Administrator
Attachment E—PATH Provider Site Visit Interview Guide: Outreach Worker/Case Manager
Attachment F—PATH Consumer Focus Group Discussion Guide
Attachment G—Letters of Introduction
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Monica Capizzi |
| Last Modified By | DHHS |
| File Modified | 2011-08-16 |
| File Created | 2011-08-16 |