ATTACHMENT
b
GOALS Site Visit Master Interview Guide
An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB # for this collection is 0970-XXXX and it expires XX/XX/XXXX.
Introduction for the Respondent (Discussions with Program Staff and Representatives of Community Partner Organizations)
As you may know, Mathematica Policy Research, Inc. is conducting the Goal-Oriented Adult Learning in Self-Sufficiency (GOALS) Study on behalf of the Administration for Children and Families (ACF) within the U.S. Department of Health and Human Services. This study will document how programs like [Name of the program] use frameworks informed by psychological concepts to provide employment and training services to low-income families.
As part of the GOALS study, we are interviewing program directors/administrators, staff, and others to better understand the extent to which programs provide services designed to strengthen goal-oriented skills. For the purposes of this study, we broadly define goal-oriented skills as a set of skills that involve managing oneself and one’s resources in order to identify, work towards, and achieve a given goal. We are interviewing staff at [name of program] to understand the types of services provided and your experience with providing services. The interview will take approximately 75 minutes. Your participation in this study is important and will help us understand more about the services provided to low-income families.
Your responses will be kept private and used only for research purposes. They will be combined with responses from staff and no individual names will be reported. Information you provide will not be shared with other program staff, including your supervisor. Only the study team will have access to the information you provide during the interview.
Participation in the study is voluntary. You can choose not to answer a question and may stop the interview at any time.
Do you have any questions before we begin?
Note to the researcher: This master interview guide will be used to collect the information gathered during GOALS project. The benefit of a standard template is that we will gather information on similar topics across interviewers. The blue bolded second level headers are considered qualitative codes used to label the data. The questions will be used to gather the information during the interviews. These questions help guide the interview and remind you of the content we are trying to collect under each main topic area. It’s also important to note that service systems available to clients vary across programs. As a result, some sections may include more detailed information than other sections.
What are the demographic characteristics of the community (for example, racial/ethnic composition, percent of individual completing a high school diploma or GED)?
How does this influence the number and types of clients served by [name of the program] structure and/or implementation?
What is the state/local unemployment rate? How would you describe the local economy (for example, economically depressed or a healthy economy)?
What is the availability of jobs for program participants? In which industries? To what extent are there good jobs, that is, full-time, well-paying positions?
What are the starting wages for low-income families in your area?
What are the opportunities for career advancement?
To what extent do low-income individuals or families have access to public assistance programs (for example, what percentages of poor families receive TANF assistance, what is the SNAP caseload)?
How would you describe the local service environment? Are there adequate services to meet the needs of low-income individuals and families? To what extent are there resources available at local community partners?
What types of services are in high demand?
What are some of the primary service gaps?
What state and local policies are relevant to the population you serve?
How might these policies help or hinder the services you provide?
What do you hope to achieve with your program (e.g., organizational and client goals)?
What would you expect to be different in the lives of clients 6 months after they begin the program? What would you expect to be different after a year?
What organizational investments (for example, staff time, funding, leveraging resources from other programs) and program services are used to achieve those goals?
What is the main message to staff and participants regarding the program goals?
What is your program’s theory of change? Do you have a Theory of Change document? How is this document used in your program?
What was the impetus for creating the program? Why and how did the program come about?
What was the initial design process? Who was involved? How long did it last?
How were decisions made about the philosophy and goals for the program? How were decisions made about the amount and types of services provided? What about decisions about outcome measures? Who was involved with these decisions? What factors influenced these decisions?
How much of the program design was driven by the research evidence? What evidence was most influential?
For purposes of this study, we are defining ‘goal-oriented skills’ as a set of skills that involve managing oneself and one’s resources in order to identify, work towards, and achieve a given goal. What does ‘goal-oriented skills’ mean to you? What skills are encompassed in ‘goal-oriented skills’? What are some examples?
How does a focus on the development of goal-oriented skills fit into your program model? What are the ‘goal-oriented skills’ your program seeks to develop among participants or clients?
Are there other ways your program would foster the development of skills that help participants attain their [short- and long-term] goals?
What were you hoping to achieve as a result of the program? What did you want to be different about how your staff interacts with clients? What did you want to be different about clients?
To what extent did you use research to inform the program design? If so, please describe.
How does the focus on goal-oriented skill development fit into your program model?
How would you describe the initial implementation of the program? To what extent did the hopes for the program match what actually happened?
What worked well during the initial implementation? What were the main challenges? How were they addressed?
How has the program evolved over time? What are some of the main lessons?
What major changes have been made over the past years? What was the impetus for these changes? What has been the effect of these changes?
How are the core values of the program reflected in program staffing and operations?
How are the program goals and approach modeled by administrators and staff?
How do administrators and supervisors reinforce the program goals and approach?
What is the administrative staffing structure? What are their roles and responsibilities?
How would you describe the management and leadership styles of program administrators?
How would you describe the communication within and outside the organization?
How would you describe the leadership skills of administrators (e.g., technical and adaptive)? (Note: When interviewing administrators, ask them how they would describe their own leadership skills and style and, if relevant, the skills of their colleagues.)
How would you describe the types and level of involvement of core program leadership with direct service workers?
How would you characterize the relationship between program leaders and organizational staff? What works well? What would you like to change?
Who are the primary program champions?
What was their role during the initial design and implementation?
How do they influence program growth and operations?
What is the program staffing structure?
What are the roles and responsibilities of different positions?
What are the caseload sizes for direct service workers?
What are the supervisory caseloads?
How are staff recruited? What strategies have worked best?
What is the process for selecting staff? What selection criteria are used?
What are the skills, education, and experience of supervisors and staff?
What skills are most critical for each of the positions?
What is the process for training staff (e.g., how, when, what)? What are the ongoing efforts to train staff?
What explicit efforts are implemented to train staff on improving the goal-oriented skills you identified earlier in this interview among clients?
How are training needs identified? How are they addressed?
What additional training might benefit staff?
What supervisory support is provided to staff?
What other supports are available to staff?
What are the lessons learned with staffing your program?
How would you describe the organizational climate?
To what extent have staff bought into the service model? How have they bought in?
What is the level of staff turnover? For what reasons do staff stay? Leave?
Who is the target population?
What are the eligibility criteria?
How many clients do you serve at any point in time? During a typical month?
How are individuals recruited to the program?
What strategies have been most effective with finding clients who are a right match for the program?
What are the characteristics of program participants?
What are the past and current stressors that may impede participant success?
In general, what is the level or readiness for change? How does the program assess a client’s level of readiness for change? Which clients appear to be most motivated? Least motivated?
To what extent are services individualized based on the participant’s needs, abilities, and interests?
What aspects of the program are more uniform?
What are the core program components? What purpose do they serve in general and in building the goal-oriented skills you identified earlier in this interview?
Assessment (e.g., types of tools, process for assessing clients, how the information is used) (Probe: How does the program assess an individual’s abilities (skills) to set and achieve their goals?)
Case planning
Case management
Coaching
Peer mentoring
Use of incentives
Group activities
Mental health counseling
Vocational education
Job retention services
Other
How long does each activity last?
What is the intensity of the services provided? If it the same level of intensity for all participants or does it vary? If it varies, how?
What is the process for deciding the services provided to a participant?
What are the most common services provided to participants?
How is goal-oriented skill development incorporated into or part of the service delivery process? What tools or resources are used to support the development of goal-oriented skills?
What additional services are needed?
What is the order in which services are provided (case flow)? How do clients progress through the service delivery process? Who provides each of the different types of services?
What is the length of time between activities?
How successful is the program with encouraging program participation and progress?
What strategies are used to engage them? What has been most useful?
How does the program encourage and monitor progress? What strategies have been successful?
What is the role of motivation in serving clients?
To what extent and how does the program use incentives?
What types of work and personal supports are provided to program participants?
What services do participants use most often?
What additional services are needed?
How do program administrators ensure that services are being implemented as intended? What measures are used? How often? Who collects this information?
How is the information gathered on program implementation used?
What community partners are involved with the program? How are they involved? What is their level of involvement?
What services, resources, and supports are available through community partners?
What formal or informal agreements are in place?
What additional partners and services might be useful?
How are data collected and stored? How often? By whom?
What information is collected?
How is the quality of the data entered? How is this monitored?
What are the strengths and limitations of the: (1) MIS system and (2) process for gathering the information?
What reports are generated from the MIS data? How often are they created? Who receives this information?
At what level are data available (e.g., administrators, supervisors, direct service staff)?
How is the information used?
What is the process for measuring program success?
What short- and long-term outcome measures are used? How are they defined? Who was involved in the process for selecting the outcome measures and defining them?
What measures focus on progress on improving goal-oriented skills? (For example, are the program measures directly related to the skills?)
To what extent are the program measures related to behavioral outcomes as a result of achieving the goals?
How are these or other measures used to document achievements in:
Goal-oriented skills
Employment outcomes
Family and economic self-sufficiency/independence
Improved participant well-being
Improved family well-being
Other outcomes, such as educational progress
What measures are used to measure client progress?
What reports are generated specifically to measure program performance? How often are they generated? How is this information used?
What evidence has been used to inform the measurement of program objectives and outcomes? How else has the research been used to inform the program?
What evaluation efforts have been completed? If completed, how was the program evaluated? Who conducted the evaluation? What were the findings? How were evaluation findings used?
If an evaluation has not been completed, then is an evaluation in the works or being planned? If so, what are the evaluation plans?
What additional information would you like to learn from a formal evaluation?
What aspects of the program work well for you? What would you see as the primary successes? Why?
What has been the long-term progress in goal-directed behaviors as a result of the program?
What have been the primary challenges? How were these challenges addressed?
What are the lessons from operating the program?
What are the amount and types of funding to support the program?
How has the program funding changed over time? For what reasons?
What additional leveraged (in-kind) resources are available?
What is the overall cost of operating the program?
What accounts for the greatest proportion of the budget?
What is the cost per participant?
What is the sustainability of the program?
What factors might influence the sustainability?
How easily might the program be replicated? Scaled up?
What is required to implement the program?
GOALS site visit topics by topics or data source
|
Program directors/ administrators |
Program staff |
Community partners |
Review of existing program documents |
|
|
|
|
|
|
|
|
|||
|
Supervisors |
Case managers |
Coaches |
Other specialized staff |
|
|
|
|
|
|
|
|
|||
Community context |
|
|
|
|
|
|
|
|
|||||||
Demographic characteristics |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Economic indicators |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Service environment |
X |
X |
X |
X |
|
X |
X |
|
|
|
|
|
|
|
|
State and local policies |
X |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Program Design and Organizational Integration |
|
|
|
|
|
|
|
|
|||||||
Program philosophy, purpose, and goals |
X |
X |
X |
X |
|
|
X |
|
|
|
|
|
|
|
|
Program impetus, initial implementation, and evolution |
X |
X |
|
|
|
|
X |
|
|
|
|
|
|
|
|
Institutional values and integration |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
|
|
Administrative Infrastructure and Characteristics |
|
|
|
|
|
|
|
|
|||||||
Administrative infrastructure |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Leadership skills and involvement |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
|
|
Program champions |
X |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Program Staffing |
|
|
|
|
|
|
|
|
|||||||
Staffing infrastructure |
X |
X |
X |
X |
|
|
X |
|
|
|
|
|
|
|
|
Staff recruitment, selection, and backgrounds |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
|
Staff training and support |
X |
X |
X |
X |
X |
|
X |
|
|
|
|
|
|
|
|
Organizational climate/staff morale |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
|
Target population and Core Program Components |
|
|
|
|
|
|
|
|
|||||||
Target population |
X |
|
X |
X |
|
|
X |
|
|
|
|
|
|
|
|
Recruitment and outreach |
|
X |
X |
X |
|
X |
X |
|
|
|
|
|
|
|
|
Program individualization process |
X |
|
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
Core program components |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
Service delivery pathway |
X |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
Participation and progress |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
Supportive services |
|
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
Implementation fidelity |
X |
X |
|
|
|
|
|
|
|
|
|
|
|
|
|
Coordination and linkages with other services within the community |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
Outcomes and Impacts |
|
|
|
|
|
|
|
|
|||||||
Data systems |
X |
X |
X |
X |
|
|
X |
|
|
|
|
|
|
|
|
Use of data |
X |
X |
|
|
|
|
X |
|
|
|
|
|
|
|
|
Outcome measurement |
X |
X |
|
|
|
|
X |
|
|
|
|
|
|
|
|
Efforts to evaluate program implementation and impacts |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Program successes, challenges, and lessons |
X |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
Funding and Program Costs |
|
|
|
|
|
|
|
|
|||||||
Funding |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Program costs |
X |
|
|
|
|
|
X |
|
|
|
|
|
|
|
|
Replicability and Sustainability |
|
|
|
|
|
|
|
|
|||||||
Sustainability |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
Replicability |
X |
X |
X |
X |
X |
X |
|
|
|
|
|
|
|
|
|
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Mathematica Staff |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |