HPOG-NIE Sampling Questionnaire for the HPOG surveys

Supporting Statement for OMB Clearance Request

Appendix B: HPOG-NIE Sampling Questionnaire for the HPOG Surveys

National Implementation Evaluation of the Health Profession Opportunity Grants (HPOG) to Serve TANF Recipients and Other Low-Income Individuals and HPOG Impact Study

0970-0394

April 24, 2013

Submitted by:

Office of Planning,
Research & Evaluation

Administration for Children & Families

U.S. Department of Health
and Human Services

Federal Project Officers:

Molly Irwin and Mary Mueggenborg

Appendix B: HPOG-NIE Sampling Questionnaire for the HPOG Surveys

Advance email to HPOG study liaison

Dear [name of HPOG study liaison]:

As you may know, [name of local HPOG program] is participating in a national evaluation of the Health Profession Opportunity Grants (HPOG), sponsored by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services (HHS). The evaluation is being conducted by Abt Associates and the Urban Institute. It is studying all HPOG-funded education and training programs across the country, and examining how they help low-income individuals, including Temporary Assistance for Needy Families (TANF) recipients, secure well-paying healthcare jobs.

Surveys will be a critical source of information for the evaluation. You have been identified by the director of [name of local HPOG program] as our point of contact to identify appropriate survey respondents for the HPOG Grantee survey, Management and Staff survey, Stakeholder/Network survey, and Employer survey. You will be asked to participate in a training webinar with other HPOG study liaisons to introduce you to the content of these surveys, and to an online sampling questionnaire that you will be asked to complete, with the names of organizations and their internal contact person(s). The questionnaire should take you approximately two hours to complete. You will subsequently ask these individuals to complete one or more of these surveys on their organization’s or program’s behalf. You will have phone and online access to a contact person at Abt Associates or the Urban Institute to provide you with additional support to help you with this process of identifying people to complete the surveys, and then following up with them to do so. Your participation in this questionnaire is completely voluntary, but it is important that we have as much input as possible to ensure accurate evaluation of these programs.

Thank you in advance for your assistance in completing this questionnaire and providing important information for the study. With your help, we will have better information about the practices of participating HPOG programs across the nation.

Sincerely,

Abt Associates HPOG Project Director

The Paperwork Reduction Act Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is xxxx-xxxx, and it expires xx/xx/xxxx. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (xxxx-xxxx).

Note to reviewers:

The following survey sampling questionnaire and forms are to be accessed online, for completion by the Study Liaisons online. The Study Liaisons, appointed by the HPOG grantees, will be asked to complete them after participating in a training webinar (or webinars) to introduce the Study Liaisons to the content of the four surveys (Grantee, Management and Staff, Stakeholder/Network, and Employer surveys). The webinar(s) will prepare Study Liaisons to complete this online sampling questionnaire, including identifying organizations and internal contacts to respond to each of the surveys, and inputting their information into the online survey sampling forms.

Note that Abt Associates and the Urban Institute will be making follow-up phone calls only to the organizations and internal contacts suggested by the Study Liaison for the Stakeholder/Network survey, in order to seek additional stakeholders and internal contacts for that survey.

• Programming instructions are in blue font

• Prefilled text from previous responses is denoted in green font

• Rollover definitions or examples are shaded in aqua

Sampling Questionnaire for the HPOG Surveys

As you may know, [name of local HPOG program] is participating in a national evaluation of the Health Profession Opportunity Grants (HPOG), sponsored by the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services (HHS). The evaluation is being conducted by Abt Associates and the Urban Institute. It is assessing a range of promising HPOG-funded education and training programs around the nation for helping low-income individuals, including Temporary Assistance for Needy Families (TANF) recipients, to secure well-paying healthcare jobs.

Surveys will be a critical source of information for the evaluation. You have been identified by the director of [name of local HPOG program] as our point of contact to identify appropriate survey respondents for the HPOG Stakeholder/Network survey, Employer survey, Management and Staff survey, and Grantee survey. The survey sampling questionnaire should take you approximately two hours to complete. Your participation in this questionnaire is completely voluntary, but it is important that we have as much input as possible to ensure accurate evaluation of these programs.

You are asked to complete online sampling forms for the Stakeholder/Network survey and for the Employer survey, in which you indicate organizations to include in these respective surveys, and the names and contact information of individuals within these organizations who can complete these surveys on their organization’s behalf. You are also asked to complete an online sampling form for the Management and Staff survey, indicating the names and contact information of [name of local HPOG program] management and staff who interact directly and regularly with program participants in a service capacity. Finally, you are asked to complete an online sampling form with candidates to complete the Grantee survey. This online questionnaire provides detailed instructions to help you along as you complete these forms.

If you have any questions or concerns while completing these forms, the following phone number and email address will connect you to support persons at Abt Associates or The Urban Institute.

[INSERT CONTACT INFO OF SAMPLING QUESTIONNAIRE SUPPORT PERSON AT ABT ASSOCIATES AND THE URBAN INSTITUTE HERE].

Thank you in advance for your assistance in helping us to administer the HPOG surveys. With your help, we will have better information about the practices of participating HPOG programs across the nation.

The Paperwork Reduction Act Burden Statement: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this collection is xxxx-xxxx, and it expires xx/xx/xxxx. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to [Contact Name]; [Contact Address]; Attn: OMB-PRA (xxxx-xxxx

1. Identifying Respondents for the Stakeholder/Network survey

[Name of local HPOG program] is part of a broad network of HPOG stakeholders or organizations with varying levels of interest in and involvement with [name of local HPOG program] and its efforts to help low-income individuals to secure healthcare training and employment. The Stakeholder/Network survey seeks the following information about these organizations:

• Information about the roles and responsibilities of organizations that have assisted [name of grantee institution] in the design and implementation of [name of local HPOG program] – examples will follow.

• Perceptions that these and other organizations have about the effectiveness of [name of local HPOG program] (e.g., “To what extent do you agree with each of the following statements about the effectiveness of [name of local HPOG program] in accomplishing the following goals? a. Meeting area healthcare needs, b. Developing career ladders for HPOG participants, etc.”)

• Perceptions of the effectiveness and sustainability of the collaboration on behalf of HPOG beyond the grant period (e.g., “Which of the following represent challenges to the sustainability/future of HPOG-related activities after the end of the HPOG grant? a. Unfavorable economic conditions, b. Excess of labor supply (e.g., too many new low- to mid-skilled healthcare graduates), etc.”)

For the Stakeholder/Network survey, we want to gather a list of organizations or agencies that have been involved with [name of local HPOG program]. For this, please think about the following categories of organizations or agencies with which [name of grantee institution] or [name of local HPOG program] may interact in operating your HPOG grant. We ask you to list organizations or agencies that fall into each category. We know that over the years since implementation, the organizations involved with the program may have changed. Please think about organizations that have been involved at any point over the entire program, starting with the design or early implementation phases.

You are then asked to recommend at least one individual from each of the organizations you identify to complete the survey on behalf of their organization. Ideally, these individuals should be well informed about their organization’s involvement with [name of local HPOG program], and who can describe [name of local HPOG program]’s progress in improving access of low-income individuals to healthcare training and education in the area.

Sampling Form for Stakeholder/Network survey

Organization	Contact Person	Title & Key Responsibilities	Phone number	Email	Contract/MOU with [name of local HPOG program]?
					Yes	No

First, think about those organizations that have a contract or formal agreement (e.g., memorandum of agreement) with [name of the grantee institution] to assist with the design and implementation of [name of local HPOG program] or the hiring of program graduates. Examples of formal roles or responsibilities these organizations have for the program include the following:

[THE FOLLOWING EXAMPLES IN AQUA WILL BE LINKED DEFINITIONS.]

• Planning and design of HPOG grant activities (e.g., grant writing, letter of commitment, member of advisory/steering committee)

• Referral of applicants for services provided by [name of grantee institution] (e.g., formal referral arrangement, initial screening of applicants, referral of current employees)

• Marketing and outreach (e.g., printed materials available on-site, information available on partner’s website, mentions during presentations to stakeholders, mentions during orientation for organization’s services, mentions during assessment and counseling session)

• Curriculum development (e.g., offered examples of relevant curricula, provided feedback on draft curricula, wrote modules for curriculum)

• Vocational or occupational training (e.g., operation of training program, provision of faculty/instructors, provision of training space, provision of equipment, provision of learning technologies, provision of work-based learning opportunities—e.g., internships, clinicals)

• Pre-training activities (e.g., prior to training, provision of workshops on healthcare occupations and educational requirements, reading or math refresher courses, computer skills; and/or provision of pre-training faculty/instructors, training space, equipment, and/or learning technologies)

• Basic academic skills education (e.g., education for foundational math, reading, and writing skills, such as General Equivalency Degree (GED) classes, pre-GED Classes, English as a Second Language (ESL) instruction, adult basic education)

• Counseling and support services (e.g., academic supports and counseling, personal supports and counseling, financial supports, other social supports)

• Job development activities (e.g., job readiness workshops, job search skills training, individual job search assistance, job coach navigator, group job search support, post-placement and retention support)

• Job placement activities (e.g., obtained and screened job listings for HPOG participants, screened HPOG participants for suitability for a position, scheduled interviews for a job candidate, provided interview space)

• Begin inputting organizations that meet these criteria, and appropriate internal contacts and their information into the online form.

Next please list organizations that you did not include already that have been involved with [name of grantee institution] and/or [name of local HPOG program] in one or more of the ways listed previously, but on an informal basis, without any contract or formal agreement with [name of the grantee institution] to do so. Examples of these are a faith-based food pantry, a home heating assistance company, and the Public Housing Authority, to which the [name of local HPOG program] routinely refers participants for emergency assistance. Other examples are local education or training schools, some of which may have even been utilized by program participants in the past, but are not under any contract or formal agreement to be the preferred education or training vendor for [name of local HPOG program].

• Continue inputting organizations that meet these criteria and internal contacts and their information into the online form.

• Note that we have prefilled the form with organizations that agreed to be involved in developing and implementing [name of local HPOG program] in the original proposal submitted by [name of grantee institution] for HPOG. Please confirm the subsequent involvement of these organizations (or delete if an organization was never involved), and specify a current internal contact person for the survey.

• We also ask that you check the involvement of organizations that the original ACF HPOG Funding Opportunity Announcement (FOA) suggested that the [name of grantee institution] engage in undertaking the HPOG grant:

• State TANF agency

• Local workforce investment board

• State workforce investment board

• State registered apprenticeship agency

Finally are organizations that could be considered HPOG stakeholders that you have not yet listed. These are organizations that have not been directly involved in program design or operation, but are aware of HPOG, share the same constituencies or objectives and may benefit from [name of local HPOG program]’s success. Examples of such organizations might include community advocacy groups, labor unions, other community colleges or training providers, policymakers or their offices, local economic development agencies, or local healthcare industry professional groups.

• Input organizations that meet these criteria and internal contacts and their information into the online form.

2. Identifying Respondents for the Employer survey

The HPOG evaluation will also be fielding an Employer survey. It seeks information from area employers that have hired [name of local HPOG program] participants or have been contacted by [name of local HPOG program] about hiring participants.

The survey will gather information about these employers’ hiring needs and practices for filling entry-level healthcare positions. Additionally, we will ask about their perceptions of [name of local HPOG program] and of its graduates as job candidates and employees. Examples of survey questions are as follows:

• How much does each of the following factors weigh into your hiring decisions for [most common healthcare occupation] positions? a. Job-related specific skill training, b. High school completion, c. Education beyond high school d. English fluency, etc.

• If a worker in [most common healthcare occupation] completes job-related training or education, are there opportunities at this organization for advancement into a position with more skilled responsibilities and pay? YES, NO, DOES NOT KNOW.

• How would you rate the applicants referred to your organization by [name of HPOG grantee institution] compared to all other applicants for the same or similar positions? a. Their interpersonal skills, b. Their desire to work hard, c. Their willingness to work odd or flexible hours, etc.

Please list employers that have hired participants/graduates of [name of local HPOG program] or have been contacted by the program about hiring program graduates. Include employers who have hired the largest number of program participants, if possible. For each employer, we ask you to propose at least one internal contact who can be contacted to complete the survey. These individuals should be familiar with their employer’s connection with [name of grantee institution] or [name of local HPOG program] They also should be knowledgeable of the organization’s hiring needs for entry-level healthcare positions, and its hiring policies and practices and personal development opportunities for employees (e.g., human resources managers, business proprietors). Your colleagues who are responsible for assisting [name of local HPOG program] participants with job search and placement, and/or reaching out to employers to explore hiring opportunities, should be able to help you identify employers and internal contacts. The number of employers you propose should not exceed 10 candidates.

If you listed employers previously in the sample for the Stakeholder/Network survey who also meet these criteria for the Employer survey, please list them again here. For these employers, the Employer survey will be an additional module of the Stakeholder/Network survey.

Sampling Table for Employer survey

Employer Organization	Contact Person(s)	Title & Key Responsibilities	Phone Number	Email	Hired [name of local HPOG program] graduates?
					Yes	No

3. Identifying Respondents for the Management and Staff survey

The Management and Staff survey examines the experiences of staff that provide direct support and services to [name of local HPOG program] participants (e.g. case managers, academic or career advisors). The survey also seeks the perspectives of those in management roles, but only those who interact directly and regularly with [name of local HPOG program] participants. Examples of subject areas and related survey questions are as follows:

• Types of interactions staff and managers have with HPOG participants: “Using a scale of 1 to 5, where 1 = Strongly Disagree and 5 = Strongly Agree, please respond to the following statements: ‘Staff in this program make an effort to learn about participants’ personal and family situations.’ ‘Staff in this program closely monitor the academic progress of its participants.’”

• Amount of time required to provide services: “Using a scale of 1 to 7, where 1 = None of My Time and 7 = Most of My Time, please indicate how much time you spend on each of the following activities: ‘Arranging instructional support such as tutoring or study groups for participants.’ ‘Referring or connecting participants to support services (childcare, TANF, SNAP, transportation, housing, etc.)’”

• Educational or service approaches that staff utilize: “In your opinion, which do you feel the more important goal of the program should be - To help participants move along the career pathway by finding employment in their desired field as quickly as possible? OR To help participants move along the career pathway by continuing their education with the aim of achieving further credentialing to support higher-skilled employment? EMPLOYMENT, EDUCATION, BOTH EQUALLY.”

• Professional development opportunities: “Are professional development opportunities available to you as part of your job? YES, NO.” “If YES, please select all that apply: workshops/training, professional conferences, etc.”

• Perceptions of work conditions: “Using a scale of 1 to 5, where 1 = Strongly Disagree and 5 = Strongly Agree, please rate how strongly do you agree or disagree with each of the following statements about [name of local HPOG program] and your experiences in your position: ‘Your program has supervisors who are capable and qualified.” “Some staff members seem confused about the main goals for your program.’ ‘Your program has enough staff to meet current participant needs.’”

Please complete the following sampling form with the names and information of individual(s) in management and staff positions who would be most appropriate for completing the Management and Staff survey. These would be only managers and staff who interact regularly and directly with participants of [name of local HPOG program] in the context of service delivery, such as case managers, academic or career counselors, or program managers who meet regularly with participants.

This survey is not intended for individuals in staff or management positions who do not have regular and direct contact with participants. In addition this survey does not target academic or training instructors unless utilized by [name of local HPOG program] unless their role combines a formal well, define case management and/or counseling role as well.

Sampling Form for Management and Staff Survey

Contact Person	Organization	Title & Key Responsibilities	Phone Number	Email	Management or Staff?
					Management	Staff

4. Identifying Respondents for the Grantee survey

The Grantee survey is the largest, most complicated of the four surveys. It seeks to gather comprehensive information from all HPOG grantees about the administration, locations, service partners, and recruitment and intake activities of their HPOG programs, and the education, training, support services, and employment assistance offered to HPOG program participants. As the survey orientation webinar emphasized, the knowledge needed to complete this survey will likely reside with a number of different people in [name of grantee institution] and/or [name of local HPOG program].

The following are issue areas and example questions from the Grantee survey that highlight the range of information and detailed knowledge sought by the survey.

• Grantee Background (including organizational type, target population, healthcare training experience, and [name of local HPOG program] context and locations: “Does [name of local HPOG program] have physical locations (distinct from on-line or by phone) for the following activities? Select YES or NO – a. Obtaining program applications or information, b. Submitting completed applications, c. Meeting with a program representative during enrollment, d. Completing required assessments, etc.”
  
  

• Grantee perspectives on HPOG mission and healthcare training: “Thinking about the accessibility and quality of healthcare training opportunities for low-income individuals in your community, please rate how strongly you agree or disagree with the following statements about the result of receiving your HPOG award, using a scale of 1 to 5, where 1=Strongly Disagree and 5= Strongly Agree. As a result of receiving the HPOG award, a. Employers are more likely to hire low-income individuals, b. Low- income individuals in my community have more access to organizations that provide healthcare training, etc.”

• Grantee relationships with other organizations: “On a scale of 1 to 5, where 1=Not a challenge and 5=A serious challenge, please rate the extent to which the following factors could make it challenging for [Name of grantee institution] to sustain relationships with other organizations involved in [name of local HPOG program], after the HPOG grant period ends. Here, we are asking about your overall perceptions of the factors that could make it challenging to sustain relationships with these organizations. a. Leadership changes in partner organizations, b. Lack of shared goals, c. Unfavorable economic conditions, d. Lack of resources in partner organizations (e.g., budget, staff, equipment, space), etc.”

• Marketing, Outreach, Intake and Enrollment: “During the application/intake period (from initial orientation to formal acceptance into the program), about how many separate in-person or phone meetings (orientations, interviews, reviews, etc.) are [name of local HPOG program] applicants required to take part in, on average? ____________ # required meetings (including in-person and phone meetings.”

• Education and Training:

Q. “For each vocational training activity [name of local HPOG program] offers (e.g. C N A, Pharmacy Aide, Registered Nurse), please identify the two most commonly used instructional methods: a. Large Group Instruction (8 or more students at one time), b. Small Group Instruction (fewer than 8 students at one time), c. Individualized (One-on-One) Instruction, d. Labs or Other “Hands-on” Exercises, e. Self-Paced Instruction, f. Online Courses/Tutorials.”

Q. “Thinking about your training providers as a group, please check the three most common ways in which HPOG participants receive academic support while engaged in occupational training beyond that which is provided during regular classroom hours. a. Spend extra one-on-one time with the instructor, b. Attend group study or “help” sessions, c. Assigned a tutor by our organization, d. Assigned a tutor by the training institution, e. Referred by instructor to an academic counselor or case manager to determine the best next steps, f. Referred by instructor to an academic “help” center at the training institution, g. Provided additional ‘self-study’ resources.”

• Support Services:

Q. “Using a five-point scale, where 1 = Strongly Disagree and 5 = Strongly Agree, please respond to the following statements about [name of local HPOG program]’s capacity to meet participants’ needs in the following areas: a. Book costs; b. Licensing and certification fees; c. Exam/exam preparation fees; d. Work/training uniforms, supplies, and tools; e. Computer/technology equipment, etc.”

Q. “Are these job development, placement, and retention services provided by dedicated staff (whose primary or only responsibility is providing that service) or staff with other primary responsibilities? a. Job-readiness workshops, b. Job search skills workshops, c. Identifying job openings for program graduates, d. Meeting with employers to make connections for the program, e. One-on-one job search assistance, etc.”

As the webinar underscored, no one individual is likely to be able to answer all the Grantee survey questions. Some questions require the perspective and experience of a frontline staff member who provides services to participants. Other questions require input from administrators, for instance, who worked on the original HPOG proposal with local partner organizations, and/or reports on the grant to HPOG’s federal funders. Keeping this in mind, please identify individuals in the [name of grantee institution] or [name of local HPOG program] who are most appropriate to help complete the Grantee survey, and input them into the sampling form for the Grantee survey.

Sampling Form for Grantee survey

Contact Person	Organization	Title & Key Responsibilities	Phone Number	Email

On behalf of ACF, thank you for your assistance in fielding the HPOG surveys.

Screen Shots of Sampling Questionnaire for the HPOG Surveys

Note to reviewer: the “@@” items will be pre-filled with respondent-specific data.