Supporting Statement for OMB Clearance Request: First Visit Interview Guide to Collect Information from Program Staff
Innovative Strategies for Increasing Self-Sufficiency (ISIS)
Contract # HHSP23320072913YC
April 11, 2011
Prepared for
Brendan Kelly
Office of Planning, Research &
Evaluation
Administration for Children & Families
U.S. Department of Health and Human
Services
Prepared by
Abt Associates Inc.
ISIS Site Teams will be conducting two rounds of implementation research (IR) site visits. The first, or baseline, will occur during the pilot stage and will document the planned intervention and the context in which each program operates, including the pilot counterfactual condition. The information collected at this point will inform the decision to move to the full implementation stage. The second set of visits will occur during the second year of the full implementation stage and will document the “steady state” program operations and any changes that occurred in the sites since the pilot visit. Information gathered during these site visits will be used to create three key deliverables (per site): the program logic model, the participant flow chart, and the site profile report.
This document serves as a discussion guide for meetings with possible informants during the first round of visits, including: program directors, managers, administrators, and other site staff; agency officials; college administrators; site partners; community organizations; employers; and other local informants. Questions for the second IR site visit are outlined in a separate document. Prior to meeting with each of the informants, consider which questions are most relevant. Possible relevant informants are listed along with each set of questions. When asking these questions, request supporting documentation, including annual reports, MOUs/partnership agreements, and other program documents.
Information will be collected from secondary data sources to inform the IR Study, including BLS and Census data. However, questions about informants’ perceptions of local labor and economic trends are also included in this discussion guide.
Introductory statement for sites: We are members of the ISIS Implementation Research Team. We are visiting all of the ISIS sites during the planning phase to document the planned intervention. Today we’d like to ask about your program’s design and the local context in which it operates, including any services control group members will be able to access during their participation in the study. We would also like to discuss your plans for recruitment, intake, and random assignment and other activities you will conduct to support the study. Neither your name nor the names of your colleagues we speak to will appear in any public document. Do you have any questions before we get started?
Introductory statement for other organizations: The U.S. Department of Health and Human Services is funding a large-scale random assignment evaluation of employment and self-sufficiency programs. Abt Associates is leading this study, which is called Innovative Strategies for Increasing Self-Sufficiency, or ISIS. We are members of the ISIS Team. [Local site or program name] will be participating in this evaluation. We are speaking with service providers in the area to document the other types of employment and training services available in the community. Today we’d like to spend some time learning about your organization and the services you provide. Neither your name nor the names of your colleagues we speak to will appear in any public document. Do you have any questions before we get started? (Note: Additional information about ISIS should be provided to organizations as necessary, especially those who are site partners. We suggest utilizing the recruitment PowerPoint presentation and/or ISIS two-pager.)
I. Program Design
Program Model – Program Director & Managers
What are the goals of the program?
Have these goals changed over time? If so, how and why?
What is the target population (socio-economic and demographic characteristics, including age, educational attainment, work history, marital status, children, US citizen or immigrant)?
Who is eligible?
What are the hypothesized impacts of the program on the target population?
Immediate (during the program period) (e.g., program enrollment, hours of training, career readiness certificate)
Subsequent (e.g., credential obtainment, employment, hours worked, wage progression)
Ultimate (e.g. self-sufficiency, career progression)
What is/are the long-term condition(s) that the program is expected to improve?
What problems will the program help to address (e.g., skills mismatch, lack of training opportunities in the community, need for support services for students)?
What is the source of the program model (i.e., is it based on existing program that has been successful elsewhere)?
What industries does the program target? Why?
Is the design of the program linked to defined employer or local workforce needs? If so, how?
What is the underlying theory of why the program will work?
Why does the target population need this program?
Where would you hope to see your program graduates in 5 years?
Do you have a logic model of the program?
How long have you been operating this specific program model?
Did anything change during the ISIS planning period? If so, what and why?
Program Components – Program Director & Managers
What are the major services provided?
Instruction/training (including hard and soft skills)
Bridge program/remediation program
Support services (financial and non-financial)
Funding for tuition, fees, supplies, uniforms, tools, certifications, other
Counseling/case management
Transportation
Childcare
Financial incentives for program completion
Emergency funding (for car repairs, medical bills, personal items, etc)
Other
Employment services
Other
Describe these services.
Why are they necessary? Why did you choose these particular program components?
Are there additional program components you would like to offer?
Are efforts underway to offer these additional components?
Service Delivery – Program Director, Managers, & Staff
For each service listed above:
Who provides these services?
How long is the program (e.g., total hours; length in weeks/months)?
How long is each of the program components (e.g., total hours; hours per day; days per week)?
How are these services delivered?
Where does each of the program components take place (e.g., classrooms, labs, program offices)?
Who provides the facilities for the program?
What performance measures are in place to track the quality of the services delivered?
Do you have data on program performance you can share with us?
Can you describe the recent trends in the performance of the program and of the service providers?
Counterfactual Condition – Program Director & Managers, Other local informants (such as WIB or TANF agency representatives)
Please describe what services (if any) will be available to the control group from:
Your organization
Your partners
Other organizations in the community, including government programs
What services do you plan to refer control group members to?
Who provides these services?
Is there an application process/waiting period that you are aware of?
Where are these services located?
Do you have a contact with whom we can speak at this organization?
Is it feasible that control group members will utilize these services?
What barriers might they have to accessing these services?
(Note: The site team should guide the site in determining the appropriate embargo period for the control group.)
(See additional questions on “Program Context” to obtain information on other services available in the community.)
Staffing – Program Directors, Managers, & Staff
How many FTEs does the program have?
Please describe the staffing/organizational structure (e.g., program departments, how the program is situated/managed within the larger organization or institution, who has authority over the program).
Can you please provide the following information about your staff positions:
Titles
Required experience and qualifications
Primary responsibilities
Where are these staff members located (if the program has multiple sites)?
Do your staff members ever work off-site (e.g., at a TANF office, one-stop, or on campus)?
If so, how often and for how many hours?
How do you believe this benefits program participants (or potential participants)?
Would you say your program is fully staffed or short-staffed?
If short-staffed, how many additional FTEs do you think your program needs?
In what areas are these FTEs needed?
How do you believe having additional staff would improve your program?
Does other staff at your organization (outside of your program) contribute to the program (e.g., a counselor who provides services to all students at a college, not just those in the program)?
If so, what services do they provide?
How accessible are these staff/services (if known)?
Have you considered providing this service through the program (if it’s not already part of the program)? Why/why not?
Would you say there are a sufficient number of staff to provide this service, or are more needed?
What kinds of training or staff development activities have been offered by the program or institution? Please describe.
Partnerships & Linkages – Program Directors & Managers, Partners
For Program Directors & Managers
What other organizations does your program partner with (including community organizations, non-profits, government agencies, employers, other)?
Can you please provide us with the following information on these entities:
Name of the organization
Type of organization
Names and phone numbers of your contacts at the organization
Length of the partnership
Nature of partnership (Formal/informal, e.g., is there an MOU or contract? If so, please provide a copy.)
Purpose of partnership
Services provided by the organization to participants of your program (including referrals)
Other information you may know about the organization
Have you encountered any difficulties in these partnerships? If so, please describe.
For Partners
Please tell us about your organization.
Name
Type of organization
Purpose of organization
Number of staff
History (e.g., how long has the organization been in the community and served the target population)
Service area
Number of clients served during the last program year
Services provided
Can you please describe the nature of the partnership and why it exists?
What services does your organization provide to participants of the program?
Do you feel this partnership has been successful? Why/why not?
Resources & Capacity – Program Directors & Managers
What resources are required to operate the program?
Funding
Curriculum/materials
Web- or computer-based training
Facilities (classrooms, labs, program offices)
Technology (laptops, internet cards)
Other
Who provides these resources for the program?
(If an outside organization) Do you have a partnership or affiliation with the institution(s) providing these resources? Please describe.
What is the program’s annual budget?
How are these resources allocated?
Does the program have sufficient resources (financial and non-financial) to provide program services to all of the participants who need/request them?
If not, which services are limited?
How many participants receive each service?
What is the process for determining who will receive each service? (See also questions under “Application”.)
How many individuals who apply for each service are denied?
Data Collection – Program Director & Managers; Agency Administrator or College President/Dean
What data are collected by/about the program and its participants?
How are these data collected, when (timing and frequency), and from whom?
Please describe how you use or plan to use these data (i.e. what is the purpose of the data collection)?
(If desired) Can you share these data with us?
What programmatic trends have you noticed recently/over time based on these data?
Do you collect information from participants about their experiences in the program? If so, please describe.
How is this information used?
II. Participant Flow – Program Director & Managers, Staff
Outreach/Recruitment
How is outreach and recruitment conducted for the program (e.g., website, word-of-mouth, fliers)? (Note: Site teams should guide programs in making plans to scale-up recruitment efforts if necessary to reach the desired sample size.)
Have you found these activities effective? Why/why not?
Have you tried other recruitment approaches in the past?
If so, why did you change your approach?
If not, what difficulties has the program encountered?
Have you adjusted/do you plan to adjust your recruitment approach as a result?
Assessment & Screening
What assessment criteria are used to determine whether applicants qualify for services?
How is the assessment conducted?
What information is collected from the applicant? How?
What are the applicant screening criteria? Why does the program use these criteria?
What assessment tools does the program use?
Have you found these tools effective?
What grade level must applicants obtain to be admitted to the program?
What happens if applicants do not obtain the required score/grade level?
Do they have an opportunity to try again?
Does the program assist them in reaching the required grade level (e.g., by providing training)?
When and where is the assessment conducted?
Who conducts and scores the assessment?
How and when are participants notified of the results?
Who makes admission decisions based on the assessment?
Does staff have some discretion in making these determinations, or must they strictly follow the criteria?
Application
How do individuals apply to the program? (e.g., online, on-site)
How long is the application process, on average?
Is there a waiting list? If so, how long is the list (what is the proportion of applicants admitted vs. placed on the waiting list)?
What are the steps of the application process (e.g., application form, interview, assessment)?
When does each of these components take place (in what order)?
Who conducts each of these components?
Who reviews the applications and other information and makes admissions decisions?
Can you provide us with a copy of the program application?
Do applicants apply for other program support services separately once they are admitted to the program?
If so, please describe this process.
If an applicant is not admitted to the program, may s/he reapply?
Is there a waiting period to reapply?
Intake
Please describe the intake process.
What information is collected and when?
When do you plan to administer the informed consent, the BIF, and the SAQ? (Note: Site teams should guide sites in making plans for these processes.)
Who will input information into the MIS? How frequently?
Who will conduct this process and where?
Random Assignment
Please describe the program’s planned method for randomly assigning participants to the treatment and control groups. (Note: Site teams should guide sites in making these plans during the planning stage. Although site staff will not have completed RA training when these questions are asked, site leadership should understand the RA process and have plans in place as to how it will be implemented.)
At what point during the intake process will random assignment be conducted?
Who will conduct RA?
How and when will participants be notified of the results of RA?
Do you anticipate any challenges in maintaining the integrity of random assignment?
If so, please describe, and discuss plans to overcome these challenges.
Enrollment/Orientation
Do participants complete any orientation or information sessions?
Are these sessions required?
Please describe the content of these sessions.
What information is provided to participants after they enroll?
What activities (if any) are participants required to complete prior to enrolling in the program (e.g., training, assessment, orientation)? Please describe.
(If applicable) How do you plan to administer incentives to the participants?
Who will conduct this process, and when?
Referrals
Does the program provide referrals to participants for other services?
If so, which organizations do you provide referrals to?
What services do these organizations provide?
Do you consider them a program partner? (See also questions under “Partnerships and Linkages”.)
Have participants been successful in obtaining services from these providers (as far as you are aware)? If not, why (e.g. high demand)?
Do you receive referrals to the program from other organizations?
If so, who provides these referrals?
Does this organization provide any other services to the individuals they refer?
Could you describe the nature of your program’s interaction with the organization that provides the referrals?
Do you consider them a program partner? (See also questions under “Partnerships and Linkages”.)
Service Receipt
How do participants access program services? Where?
Do you track how often participants utilize program services?
If so, how?
Can you provide us with this information?
Have you noticed any trends in the utilization of program services? If so, please describe.
Are there any requirements placed on program participants to utilize services (e.g., meeting with a counselor every month, completing program training sessions)?
Under what circumstances (if any) do participants stop receiving program services, or become ineligible to participate in the program?
Is an individual service strategy or training plan developed for each participant?
If so, please describe.
Can you provide an example of this plan?
How is participants’ progress monitored?
How would you identify if a student is having problems? What action would be taken?
Program Completion
What services/supports do participants receive upon program completion?
Alternative Pathways
What options are available to participants who aren’t able to complete the program?
What other pathways are they likely to pursue?
Do you provide referrals for other services in the community to participants who are unable to complete the program or otherwise meet program requirements? If so, please describe
Follow-up Services
What services do you provide to participants who have left the program (e.g., employment services)?
How long are these services generally provided to individuals after they leave the program?
What data are collected about/from these individuals?
III. Program Context
Program and Site Characteristics -- Program Director and Managers
How long has the program existed (e.g., is this a new or established program; does it build on an existing program)?
How many participants does the program serve (currently, historically)?
How many participants in the history of the program have successfully completed training or graduated? (What proportion?)
Why do participants typically drop out of the program?
Are there ways in which the program has fallen short of meeting its goals? Please describe.
How many participants will be in the treatment/control groups in each cohort? [Note: Site teams should guide sites in determining the size and ratio of treatment and control groups during the planning stage, and in making plans to scale-up the program, if necessary.]
How many cohorts will participate in the study period?
When will each cohort begin the program (i.e. when does enrollment occur/when do classes begin)?
How many offices does the program have?
Where are these offices located?
What services are offered at each office (if they vary)?
Is the program co-located with another service provider (e.g., a one-stop)?
If so, what provider is co-located with the program?
What services does the provider offer to program participants, others?
Area Served -- Program Director and Managers
What is the service area of the program (or organization)?
How large is this area?
What are the particular characteristics of this area (e.g., is the service area geographically large, is there public transportation, what are population trends)?
How is the program tailored to meet participant needs as a result of these characteristics (if applicable)?
Area Demographics -- Program Director and Managers
What are the demographics of the service area?
Have the demographics been consistent over time, or have they changed recently? Please describe.
Program challenges/successes -- Program Director and Managers
What do you believe have been the program’s greatest successes?
What are the main factors that contributed to this success?
What challenges did the program encountering during start-up? How were these challenges overcome?
What challenges has the program encountered recently? How were these challenges overcome?
What are the most important areas for program improvement that you or your organization/institution has identified?
How do you plan to address these?
Local economic conditions -- Program Director and Managers
Can you describe the local economic conditions?
Have any employers recently left/come to the area? Please describe.
Has there been a recent increase or decrease in the number of jobs available?
In what industry/what employer?
What are wage rates in the area (particularly for the occupations in which the training is being offered)?
Unemployment (pull from BLS prior to visit and confirm recent trends)
Any other local economic conditions that may have affected the program’s ability to recruit/retrain participants or participants’ ability to find employment.
Institutional – Organization or Agency Administrator/College President or Dean
Please describe the goals of your organization/institution.
How does the program we are examining fit into these goals?
What oversight activities does the organization/institution conduct for the program?
What performance measures are used to track or assess program success?
What do you consider to be the program’s main challenges and successes?
Program Director and Managers
To what extent is the management at your organization/institution involved in the program?
How often do you meet with management to provide information about the program?
What information do you provide?
What performance measures does the organization use to determine if the program is successful? Please describe.
To what extent are the funders of the program (if outside of the organization) involved in the program?
(Same questions as above.)
Community & Government Supports
Program Director and Managers
What community and government supports do program participants primarily rely on? (These can include support from your organization that is not part of this program; support from the government, such as welfare; or support from organizations active in your community, such as scholarships.)
What services do they provide?
Does the program coordinate with this provider in any way (e.g., have a formal or informal partnership, refer participants to this provider, receive referrals from this provider, exchange information, etc.?) (If so, see questions under “Partnerships and Linkages”.)
Do you collect information on program participants’ enrollment in these programs?
If so, what information do you collect? Can you provide this information to us?
What happens if participants lose their benefits? Is the program able to assist them?
Are you aware of any other organizations in the area that may provide services to students, parents, or low-income individuals? If so, please describe.
Other Service Providers
Please tell us about your organization.
Name
Type of organization
Purpose of organization
Number of staff
Target population
History (e.g., how long has the organization been in the community and served the target population)
Service area
Number of clients served during the last program year
Services provided
What information do you collect about the individuals you serve (e.g., their enrollment in other programs)?
Are you aware if you serve individuals who participate in (name of training program)?
| File Type | application/msword |
| File Title | Abt Single-Sided Body Template |
| Author | NicholsonJ |
| Last Modified By | DSHOME |
| File Modified | 2011-07-14 |
| File Created | 2011-07-14 |