VISTA Assessment of Project Sustainability

/home/ec2-user/sec/disk/omb/icr/200809-3045-001/doc/8534001

Protocol for Pre-Test Site Visit

Objective: The purpose of the site visit interviews is to gain insights about VISTA project development and implementation that otherwise may not be obtainable from written project documentation. The interviews will focus on obtaining the following:

• Background information about the sponsoring organization, including how long it has been in business, its overall mission, and the depth and breadth of the organization’s community networks.

• Information about the political economy of the target service area/population, and the ways in which the organization tries to fulfill area/population needs and interests;

• Details about the genesis of the VISTA project. At the time of its development, in what ways was the project intended to support organizational capacity? And in what ways did the VISTA project resonate with/emerge from target population/community needs? Finally, why VISTA?

• For those project goals that were achieved:

  • An assessment of the most important factors contributing to project success (e.g., VISTA characteristics, community involvement, support from network organizations).

  • A description of challenges that were faced along the way, and successful strategies project leadership implemented in response to these challenges (e.g., staffing changes, modifications to goals/objectives, requests for additional training or support).

  • A discussion of how those achievements continue to positively effect the organization. Was the goal something tangible (e.g., a database) that continues to be used? Was the goal a step in a larger process, a foundational achievement on which the organization continues to build?

• For those project goals that were not achieved:

  • An assessment of those factors that contributed to the lack of success (e.g., VISTA characteristics, lack of community buy-in, need for more training/support).

  • A description of specific challenges the project encountered and ways in which project leadership tried to respond to those challenges.

  • A brief discussion of “lessons learned” or “how things might be done differently” if the organization had an opportunity to revisit the unachieved goals.

• A frank assessment of the VISTAs who worked on the project (e.g., work ethic, attitude, goodness of fit with the local community). The number of VISTAs and their turnover rate can be extracted from the written record; but this set of questions will aim to understand the qualities of VISTAs – both those that contribute to positive outcomes, as well as those that are less than desirable.

We anticipate that the narrative obtained in response to these and related questions will enhance the analysis of the quantitative variables, possibly providing insights into project dynamics – or relationships between the variables – that would not be obvious from a statistical analysis by itself.

INTERVIEWER PREPARATION: Prior to going on-site, the interviewer should obtain as much background information as possible about the organization and its VISTA project. Materials to be reviewed will include (but are not limited to):

• Project abstraction form and its derivative, the project profile – this will offer the site visitor background information on the development of, changes to, and progress of the VISTA project.

• Copy of the completed telephone interview – the interview summary offers a first-person assessment of the project’s relative success, and historical information about training and support for the project.

• Sponsoring organization’s website – websites can provide the interviewer with a wealth of information about the organization, its relationship to the local community, background information on key staff members, and, sometimes, details about the VISTA project that will be the focus of the interview. A thorough review of this information source will greatly facilitate the on-site interview since such details provide the foundation for understanding how the VISTA project fit in with the organization’s mission, history, and accomplishments. With this foundation of knowledge, the interviewer will be well-positioned to understand not only the historical details of the VISTA project, but also the dynamics of how the project was implemented.

PROCESS: Interviews will take place with a small group of respondents, which might include the organization’s Director, current VISTA, staff member/s working with the legacy of the identified project, as well as former staff (organization, VISTA) who may still be available for consultation. The interview should take approximately two hours; the team should allow at least an additional hour, as needed, to observe or review ways in which the identified project continues to support the program’s efforts.

INTRODUCTION

Hi, my name is _______________________ and I work for Westat, a private research company in Rockville, Maryland. We were contracted by the Corporation for National and Community Service – the Federal agency that runs the VISTA program – to learn more about how VISTA projects work: how sponsoring organizations develop the ideas for projects, how community members are involved in project implementation, the kinds of training/support that you and your VISTAs receive along the way, and the qualities of VISTAs that contribute to a successful project.

There are no right or wrong answers to any of the questions that I’ll be asking. Your participation in the discussion is entirely voluntary, so not answering a question or stopping partway through the discussion is fine. Your job won’t be affected by your decision, nor will there be any impact on your organization’s funding related to VISTA. We are not focused on who said what, so much as what was said – so no names will be used in our reports, nor will there be any identifying information included in those write-ups.

Based on our pilot visit, I expect that this discussion will take about two hours. We’ll take a short break midway through, but you should feel free to step out at any point if you need to.

I will probably take a few notes as you all talk, but in order for me to better focus on the discussion and ensure that I have an accurate record for later analysis, I would like to audio-record the group. These are digital files that will be stored on a secure network location at Westat, and only staff members who are working on this evaluation will have access to the files. Before we start, do you have any questions? And do I have your permission to record this interview? [Once recorder in on, ask for verbal permission again.]

1. Organizational and Community Background

1. I’d like to start off by making sure I have a good understanding of your organization. Please tell me about [organization]:

   1. What was the impetus for the organization’s development?

   2. What is the organization’s mission?

   3. How long has it been in existence? How long has it been in this community?

   4. Approximately how many employees do you have? What are the key organizational roles?

   5. Who are some of its key community partners? And in what ways do they contribute to the organization’s mission?

2. [If not already discussed above] Let’s talk a little about the needs of the target population/community that your organization serves.

   1. What are some of the critical issues that the population/community faces?

   2. (If not obvious) In what ways does your organization try to meet those needs?

   3. In what endeavors do you find you get a lot of support [community strengths]?

   4. And where do you continue to face challenges? Why do you think those challenges persist?

2. History of the VISTA Project

   1. When/How did the idea for the project emerge?

   2. What was the impetus at the organizational level?

1. How did the project fit into your organization’s mission?

2. In what ways was the project seen as increasing your organization’s capacity to meet its mission?

1. What needs did the project aim to meet at the community level?

1. What was the anti-poverty vision of the project?

2. What long-term benefits were envisioned for the community?

3. What was the process by which your organization worked with the community to identify those needs?

4. How did the project come to be one that involved VISTA?

   1. What value did you feel the VISTA would add to your organization?

   2. What other options did you consider prior to applying for a VISTA?

3. Project Goals and Accomplishments – In reviewing our background information, I have that the VISTA project had n goals when the project was initiated: ENUMERATE THE GOALS FOR RESPONDENT; THERE MAY BE MORE THAN 2 GOALS -

   1. I’d like to start with Goal 1, which the documentation from the telephone interview says was successfully achieved,

      1. What does “successful” look like with respect to this goal? That is, by what measure/s do you feel your work was successful?

      2. What would you say were the most important factors contributing to your success in meeting GOAL 1? [e.g., VISTA characteristics, community support, training, etc.]

      3. What challenges to GOAL 1 emerged along the way? What strategies did project implement to successfully overcome those challenges? How were those strategies devised (trial and error, consensus meetings, etc.)?

      4. In reading through these reports, it seems that one outcome of GOAL 1 was to [DESCRIBE – e.g., develop a database, create a website, write up training materials]. Is [the outcome] still in use in your organization today?

         1. IF YES – Please describe how it continues to function within the organization. What, if any, changes have been made over the years so that it can successfully adapt to changes in your organization?

         2. IF NO – Why do you think the [database, website, training outline] is no longer being used by your organization?

         3. Did these n goals generate any new ideas or projects for your organization to pursue? If so, please describe.

   2. Now I’d like to look at GOAL 2, which you indicated was not met.

      1. First, what factors came into play such that the project did not meet GOAL 2 (VISTA charac., training, lack of community support)?

      2. Is GOAL 2 something that you think is still important to the organization? Why/not?

      3. Looking back, what if anything do you think could have been done differently that would have helped GOAL 2 come to fruition?

      4. What would be the measure/s by which you would be able to say that Goal 2 was successful?

4. VISTA Characteristics

Finally, I’d like to learn more about the VISTAs that you had over the course of this project, and how each contributed (or did not) to the project meeting its goals. In reviewing these background materials, it looks like you had [X] different VISTAs during the life of the project.

1. Thinking about the VISTAs that you had on this project:

   1. What characteristics do you think were shared by those VISTAs who were optimally suited to this project? (e.g., Attitude? Age? Training? Local community member?) Example of a staff member who epitomized this combination of traits?

   2. What, if any, characteristics do you think were shared by those VISTAs who were not as good a fit for this particular project? In what ways do you think that/those characteristic/s created challenges? What other projects do you know of – if any – where those traits might have offered a better fit?

   3. What kind of training did your VISTAs have prior to coming to this organization? How well do you think the training prepared them to work in your organization? What other kinds of training do you think would be helpful for VISTAs to have?

   4. What, if any, training did you/your organization have to prepare for having VISTAs here?

      1. IF R HAD TRAINING: How well did it prepare you for supervising your VISTAs?

      2. IF R DID NOT HAVE TRAINING: Is there training that you think would have been helpful in preparing you to supervise your VISTAs?

   5. What project-related activities did the VISTA perform?

   6. What other activities did the VISTA perform, in addition to project activities?

   7. How did having a new VISTA (or VISTAs) each year affect the project? For example, was there overlap in the VISTAs at all?

   8. What suggestions – if any – do you have for how to improve how VISTAs can contribute to your organization and its objectives?

5. In hindsight, what, if anything, would you have done differently with this particular project? [Possible probe areas: created different goals/objectives; linked better with the community? Asked for more/different training? Selected different VISTAs?]

6. Is there anything else about this project that we haven’t asked about, but that you think would help us to better understand the project’s development and contribution to the community?

7