tribal HPOG_Supporting Statement Part B_revised 10.11.11

OMB Clearance Application

Evaluation of Tribal Health

Profession Opportunity Grants (HPOG)

Supporting Statement Part B

June 2011

2. Collection of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

• Site visits: Site visits will take place at each of the 13 sites (5 prime implementation sites; 8 secondary implementation sites) where the Tribal HPOG programs are being implemented; however, it is likely that data collection activities for secondary implementation sites outside of the prime grantee implementation site may be conducted via phone or teleconference during the middle data collection year (follow-up 1 site visits).

  • Interviews with Grantee and Partner Administrative Staff: Interviews will be conducted with the grant director and up to 6 additional staff members across primary and secondary implementation sites for each of the 5 grantee programs, for up to a total of 35 interviews annually. While respondents will represent a variety of positions across grantee programs, they will most likely include the project director and manager, staff from the workforce development and TANF partner organizations, public and private university-based partners, and not-for profit organizations. An advance letter (see Attachment A7) that describes the project, its goals, and the benefits of participation, will be sent to the grant director, and a recruitment letter will be used to recruit additional grantee and partner administrative staff, if necessary (see Attachment A8).

  • Interviews with Program Implementation Staff: Interviews will be conducted with staff responsible for implementing the Tribal HPOG programs in primary and secondary implementation sites. While respondents will represent a variety of positions due to the unique nature of each grantee program, they will most likely include instructors, recruitment and orientation staff, and providers of supportive services. Interviews will be conducted with up to 9 implementation staff at each primary and secondary implementation site, for a total of up to 117 interviews annually. Phone interviews may be conducted in lieu of in-person interviews with implementation staff at secondary implementation sites in middle data collection year (follow-up 1). The universe of implementation staff will be identified by grantee administrative staff prior to initial site visits. While recruitment methods may vary by site, when possible we will send a recruitment letter signed by the grantee program (Attachment A8). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling a time for the interview if appropriate.

  • Interviews with Employers: Interviews will be conducted with local or regional employers at public or private companies or organizations that are partnering with the Tribal HPOG program or have employed program participants. Interviews will be conducted with up to 4 employers at each primary and secondary implementation site, for up to a total of 52 interviews annually. Phone interviews may be conducted in lieu of in-person interviews with employers at secondary implementation sites in the middle data collection year (follow-up 1). This total may vary based on the number of employers located and identified by grantees at each distinct site. While recruitment methods may vary by site, when possible we will send a recruitment letter signed by the grantee program (Attachment A8). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling a time for the interview if appropriate.

  • Focus Groups or Interviews with Program Participants: Focus groups or interviews will be conducted with current student program participants across all primary and secondary implementation sites. Based on the expected number of program enrollees reported in the grantees’ 120 work plans, we believe the total program participant respondent universe is approximately 1,500¹ across all the Tribal HPOG grantees (average of 300 expected participants across grantees per year). Up to 9 individuals will participate in focus groups or interviews (conducted with individuals in secondary implementation sites in the middle data collection year) in each prime and secondary implementation site, for a total of up to 117 program participant respondents, annually. Grantees will provide a list of all current participants by implementation site prior to each round of site visits. The tribal evaluation team will work closely with grantees to identify the appropriate recruiting mechanism for a targeted group of respondents. While recruitment methods may vary by site, when possible we will first send potential focus groups and interview participants a recruitment letter signed by the grantee (see Attachment A9 for focus group recruitment letter; see Attachment A10 for interview recruitment letter). Depending on the number of current participants in each implementation site, we may send recruitment letters to the entire pool of current participants, or randomize the lists of participants and recruit in waves. The use of random sampling has been employed as a sampling strategy in similar qualitative research efforts². We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and communicating additional information, such as meeting date, time, and place, as appropriate. In other cases, we may work with the grantee site to distribute a flyer to program participants (Attachment A11).

• Telephone Interviews with Program Completers: Telephone interviews will be conducted with student participants across all primary and secondary implementation sites who completed the Tribal HPOG programs. Based on the expected number of program completers reported in the grantees’ 120 work plans, we believe the total program completer respondent universe is approximately 1,125 across all the Tribal HPOG grantees (average of 225 program completers across grantees per year). Up to 20 program completers in each grantee program will participate in interviews, for a total of up to 100 interviews annually after initial and follow-up 1 site visits, respectively. At least 4 interviews will be conducted in each implementation site. Grantees will provide a list of all program completers by implementation site after initial and follow-up 1 site visits. The tribal evaluation team will work closely with grantees to identify the appropriate recruiting mechanism for a targeted group of respondents. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). Depending on the number of program completers in each implementation site, we may send recruitment letters to the entire pool of completers, or randomize the lists of completers and recruit in waves. As noted above, the use of random sampling has been employed as a sampling strategy in similar qualitative research efforts³.We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate.

• Telephone Interviews with Program Non-Completers: Telephone interviews will be conducted with participants that did not complete the Tribal HPOG program. This number may vary across sites. Best estimates derived from the grantees indicate that the total respondent universe of non-completers is 375 across all the Tribal HPOG grantees (average of 75 program non-completers across grantees per year). The grantees will provide a list of student non-completers across implementation sites every other month starting after the initial site visits. Up to 4 interviews will be conducted with non-completers in each grantee program for a total of up to 20 interviews annually. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). Depending on the number of program completers in each implementation site, we may send recruitment letters to the entire pool of completers, or randomize the lists of completers and recruit in waves. As noted above, the use of random sampling has been employed as a sampling strategy in similar qualitative research efforts⁴. We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate. Alternatively, upon notice to the grantee of a participant’s intent to withdraw or de-matriculate from the program (which is a standard procedure in institutions of higher or vocational education), the program coordinator (or designate) will provide the recruitment letter to all program non-completers.

2. Procedures for Collection of Information

No statistical methods will be used for stratification and sample selection.

The table below provides an estimated timeline of data collection activities. Funding for Tribal HPOG grantees began on September 30, 2010 and ends on September 29, 2015. Evaluation data collection activities conducted in the second, third and fourth years of the Tribal HPOG program.

Timeline of Data Collection Activities

Activity	Expected Date
Conduct Initial Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person interviews with Program Implementation Staff In-person interviews with Employers Focus groups with Program Participants	1-8 months following OMB approval
Conduct Follow-up 1 Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person or telephone interviews with Program Implementation Staff In-person or telephone interviews with Employers Focus groups or interviews with Program Participants	15-22 months following OMB approval
Conduct Follow-up 2 Site Visits In-person interviews with Grantee and Partner Administrative Staff In-person interviews with Program Implementation Staff In-person interviews with Employers Focus groups with Program Participants	27-34 months following OMB approval
Conduct Interviews with Program Completers	8-15 months following OMB approval; 22-27 months following OMB approval
Conduct Interviews with Program Non-Completers	8 to 34 months following OMB approval

• Site Visits

Information for the Grantee and Partner Administrative Staff, Program Implementation Staff, Employers interviews and Program Participant focus groups will be collected by members of the tribal evaluation team during annual site visits (or by telephone with respondents located in secondary implementation sites in the middle data collection year). The tribal evaluation team will prepare an advance letter that describes the project, its goals, and the benefits of participation, and send it to the grantee directors (see Attachment A7). Site visits will be conducted at three points in time - 1-8 months following OMB approval (at the beginning of second year of the grantees’ funding), 15-22 months following OMB approval, and 27-34 months following OMB approval. The site visits will allow the evaluation team and ACF to assess how the programs have progressed and changed over time.

Interviews with Grantee and Partner Administrative Staff will be conducted to gain insight on high-level program strategies, program development and lessons learned (see instrument #1). Interviews with Program Implementation Staff will shed light on program processes including recruitment; orientation and program implementation (see instrument #2). Interviews with Employers will provide information on employers’ general impressions of program graduates, their degree of awareness of the program as a whole, and their views on the extent to which programs are making an impact on the creation of a workforce that is equipped to meet the demands of the current health care needs of tribal communities (see instrument #3). All interviews guides are constructed in a modular fashion so that different sets of questions can be combined according to the role of the respondent and topics of interest.

Focus groups (or by telephone with respondents located in secondary implementation sites in the middle data collection year) with Program Participants will be conducted to gain program participants’ perceptions around program design and curriculum; supportive services; recruitment and orientation; incorporation of Family Education Model; quality of instruction; educational attainment; and satisfaction with the Tribal HPOG program. Collecting this data on an annual basis will demonstrate whether there is change over time with regard to participants’ experiences with the HPOG program (see instrument #4 to be used for focus groups and interviews).

Tribal HPOG grantees will provide the tribal evaluation team with a comprehensive list of grantee and partner administrative staff, staff responsible for implementing the program, local employers, and current participants and their contact information. In consultation with each Tribal HPOG grantee, the tribal evaluation team will recruit using recruitment letters or flyers and if necessary, follow-up phone calls (see Attachments A8, A9 and A10 for recruitment letters and Attachment A11 for flyer). Should the Tribal HPOG grantee prefer to have potential participants contacted by program staff (e.g., the Program Coordinator) we will provide the recruitment materials as described to facilitate recruitment. All interviews will be conducted by appointment.

Each site visit will be three to five days in length. Interviews will be conducted by at least two team members, one senior staff member to conduct the interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Interviews will be conducted using detailed protocols and note-taking guides that are designed to offer maximum flexibility in gathering input from the respondents. Assuming that meeting logistics are well coordinated with attention to what will be the most convenient time for focus group participants is crucial to support adequate participation and, importantly minimize burden on participants. In all cases, we will work with grantees to identify the best time and venue for conducting focus groups. Following each site visit, the site visit team will meet with the rest of the project team to conduct a formal debriefing of the findings.

• Program Completers

Information concerning Program Completers will be collected through telephone interviews with participants that have completed the Tribal HPOG program. These interviews will be used to obtain information on key program outcomes related to educational attainment and employment (see instrument #5). These interviews will occur at two points in time – 8-15 months after OMB approval (after initial site visits) and 22-27 months after OMB approval (after follow-up 1 site visits). We anticipate that some participants in the annual focus groups will also participate in the interviews; others may have had no previous contact with the Evaluation Team.

Grantees will provide a list of all program completers by implementation site after initial and follow-up 1 site visits. While recruitment methods may vary by site, when possible we will first send potential interview participants a recruitment letter signed by the grantee (Attachment A10). We will follow up with phone calls, respectfully inquiring about the individual’s willingness to participate and scheduling an interview, as appropriate. Interviews will be conducted by at least two team members, one senior staff member to conduct he interview and a junior staff member to take comprehensive notes and identify areas to be covered as the interview progresses. Following the conclusion of each interview, the tribal evaluation team will produce a clear, electronic copy of interview notes that summarize key findings.

• Program Non-Completers

Information concerning Program Non-Completers will be obtained through telephone interviews with participants that did not complete the Tribal HPOG program. The purpose of these interviews will be to learn about reasons for leaving the program and challenges experienced, elements of the program that were effective or non-effective and identification of any outcomes resulting from program participation (see instrument #6).

While recruitment methods may vary by site, whenever possible upon notice to the grantee of a participant’s intent to withdraw or de-matriculate from the program (which is a standard procedure in institutions of higher or vocational education), the program coordinator (or designate) will provide a recruitment letter supplied by the tribal evaluation team which states the purpose of the interview, addresses respondent privacy, provides contact information about the evaluation team and human subject protections, and details the procedures for receiving an incentive for participation (see Attachment A10). In addition, grantees will provide the tribal evaluation team with a list of non-completers, including contact information, every other month starting after initial site visits through the completion of follow-up 2 site visits. Non-completers will likely have very limited contact information, limited telephone access or may have moved from the area. A member of the tribal evaluation team will then attempt to contact each non-completer by phone to ask if he or she is willing to participate in the exit interview. As with the participant interviews, the evaluation lead will seek verbal Informed Consent.

All members of the evaluation team will be trained to conduct the site visits, administer the interviews, and facilitate focus groups in a culturally competent manner. Interviewer will also be trained on their responsibilities to ensure privacy. The content of the interview protocols and focus group guides were developed by tribal evaluation team partners in consultation with key stakeholders, including the Tribal HPOG grantees, the internal Advisory Committee, ACF officials, and shared with the HPOG evaluators.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Gaining cooperation and buy-in from grantees are key objectives of this data collection effort. For this data collection effort, we believe that the specialized nature of the respondent groups – grantees and partners, and in some cases implementation staff and employers - that received funding from ACF and who are interested in supporting HHS efforts to design and implement health professions training programs will increase their propensity to respond, resulting in a response rate of at least 80 percent. It is important to note that the tribal HPOG grantees are aware of this assessment and have agreed to participate in the assessment as part of their cooperative agreement with ACF.

In the case of the participant focus groups and interviews, the tribal evaluation team will work with the tribal HPOG grantees to recruit these individuals. Given that the participants are likely involved with the Tribal HPOG program or are receiving its services, it will be effective to collaborate with the Tribal HPOG grantees to secure the participation of these individuals.

In addition, the tribal evaluation team will use a number of proven methods to maximize participation and cooperation in the study:

• Advance and Recruitment Letters: Advance and/or recruitment letters will be sent to all potential respondents. A description of the evaluation will be included in this mailing. Follow-up introductory calls will be made with prospective respondents to answer questions and address concerns.

• Pre-interview preparation: Grantee and Partner Administrative Staff, Program Implementation Staff, and Employers will be provided topics that will be covered in the interview prior to the interviews to gain cooperation and minimize burden during the actual interview.

• Administration: Interviews conducted during the site visits will be scheduled at the respondents’ convenience. Focus groups will be conducted during the evening hours and light refreshments and transportation vouchers will be provided. Telephone interviews with program participants (completers and non-completers) will be scheduled at the respondents’ convenience.

• Assurances of privacy: Respondents will be assured that reported data are aggregated and not attributable to individuals or organizational entities.

• Use of Incentives: A non-cash honorarium valued at $50 will be provided to the Program Participants (focus groups)). Program Participants (at secondary implementation sites during follow-up 2 site visits) and Program Completers that participate in the telephone interviews in lieu of the focus groups will receive a non-cash honorarium valued at $25. Non-completers that participate in telephone interviews will receive a non-cash honorarium valued at $10. Each grantee will be consulted to determine the most appropriate method of payment (e.g., gift certificate to a local grocery store).

While we expect a few hard refusals to the interviews and focus groups, we have found that these techniques are highly effective ways to increase response rates, particularly in cases where the overall sample size is small enough to provide a “personalized” or high-touch level of follow-up.

4. Tests of Procedures or Methods to be Undertaken

The final instruments have been reviewed by (1) Administration for Children and Families staff, (2) A group of consultants comprised of workforce and tribal researchers, (3) staff at the National Indian Health Board staff, a tribal membership organization and tribal employer, and (4) staff from Red Star Innovations, a tribally owned small business. Their comments were incorporated into the final versions.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and /or Analyzing Data

The information for this study is being collected by NORC at the University of Chicago, a non-profit social science research organization, and its subcontractors, the National Indian Health Board and Red Star Innovations, on behalf of the US Department of Health and Human Services (HHS). With HHS oversight NORC and its subcontractors are responsible for the collection and analysis of qualitative data and producing site visit reports, practice briefs, an interim report, and a final report. Key input into the design was received from the following individuals:

NORC at the University of Chicago team

Michael Meit, Project Director, NORC at the University of Chicago

Alana Knudson, Senior Advisor, NORC at the University of Chicago

Carol Hafford, Co-Task leader, NORC at the University of Chicago

Aleena Hernandez, Co-Task Leader, Red Star Innovations

Jessica Bushar, Project Manager, NORC at the University of Chicago

Hilary Scherer, Research Analyst, NORC at the University of Chicago

Heather Langerman, Research Assistant, NORC at the University of Chicago

Evangeline Dotomain, National Indian Health Board

Consultants

Matthew L. Boulton, MD, MPH

Mark Doescher, MD, MPH

Kristie Gebbie, DrPH, RN

Jacque Gray, PhD

Felicia Schanche Hodge, DrPH

Hugh Tilson, MD, DrPH

Department of Health and Human Services, Administration for Children and Families

Michael Dubinksy, Project Officer, Office of Planning Research, and Evaluation

Hilary Forster, Project Officer, Office of Planning Research, and Evaluation

Molly Irwin, HPO Evaluation Team Lead, Office of Planning Research, and Evaluation

Tribal Health Professions Opportunities Grants grantees

Terry Tatsey, Blackfeet Community College

Carol Wren, Cook Inlet Tribal Council

Kimberly O’Leary, College of the Menominee Nation

Larretta Hall, Turtle Mountain College

Cynthia Lindquist-Mala, Cankdeska Community College

The HHS Project Officers, Michael Dubinsky and Hilary Forster, have overseen the design and can be contacted at:

Michael Dubinsky

Office of Planning, Research, and Evaluation

Administration for Children and Families

US Department of Health and Human Services

370 L’Enfant Promenade, SW

Washington DC 20477

(202) 401-3442

Hilary Forster

Office of Planning, Research, and Evaluation

Administration for Children and Families

US Department of Health and Human Services

370 L’Enfant Promenade, SW

Washington DC 20477

(202) 619-1790

Data collection and analysis will be conducted by Michael Meit, Alana Knudson, Carol Hafford, Jessica Bushar, Hilary Scherer, and Heather Langerman, all of NORC at the University of Chicago, Aleena Hernandez of Red Star Innovations, and Evangeline Dotomain of National Indian Health Board.

1 The total number of program enrollees (1,500) was calculated based on the total number of expected enrollees provided in grantee 120 day work plans. It is important to note that when calculating this total, for College of Menominee, only the number of students they expected to enroll in their CNA program (300) were included in this total. Menominee program participants who will receive support services, but not enroll in the new CNA program, were excluded from this total.

2 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

3 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

4 Huston, A.C., Miller, C, Richburg-Hayes, L., Duncan, G.J., Eldred, C.A., Weisner, T.S., et al. (June 2003). New Hope for Families and Children: Five-Year Results of a Program to Reduce Poverty and Reform Welfare. NY: MDRC.

1