Corporation for National and Community Service (AmeriCorps)
AmeriCorps Program Life Cycle Evaluation – Puerto Rico Bundled Evaluation
OMB Control Number 3045 – NEW
Justification – Part A Supporting Statement
This is a request for approval of a New Collection.
This information collection is authorized by the National and Community Service Act of 1990, as amended, at 42 U.S.C. 12639, which requires AmeriCorps to evaluate programs that receive assistance under the national service laws. This new information collection, for the Puerto Rico Bundled Evaluation, will evaluate the:
Context, implementation, and outcomes of 13 AmeriCorps-supported organizations in Puerto Rico with AmeriCorps State and National formula grants (funded through the Puerto Rico service commission, Comisión de Voluntariado y Servicio Comunitario), as well as those that have both AmeriCorps State and National grants and AmeriCorps VISTA projects.
Effectiveness of evaluation capacity building workshops to be provided to the bundle participants.
AmeriCorps is a federal agency that promotes volunteering and service through a wide range of programs and grants to improve lives, strengthen communities, and foster civic engagement. AmeriCorps has had an important presence in the Commonwealth of Puerto Rico and has supported organizations in tackling some of Puerto Rico’s greatest challenges related to disasters, the economy, education, environment, and health. For example, in 2022, AmeriCorps invested more than $11 million in federal funding and 1,046 members and volunteers served in Puerto Rico.
The Puerto Rico Bundled Evaluation is part of a multi-project contract entitled “Life Cycle Evaluations” (LCE). This project is being conducted by ICF through a contract with AmeriCorps. The LCE contract will conduct four sequenced program evaluations (descriptive and/or outcome based) and provide tailored evaluation capacity building workshops to grantee and sponsor organizations “bundled” together in programmatic or priority focus areas.
The bundling approach seeks to support grantee and sponsor organizations that may not yet have the required capacity to conduct evidence building activities on their own and/or where no evidence base exists for their program area of work. By bundling, AmeriCorps can combine groups of programs across different funding streams with similar models and intended outcomes into a single evaluation while at the same time, providing a peer-learning and networking opportunity through evaluation capacity building workshops. Furthermore, this approach supports the establishment of communities of practice to identify common best practices and lessons learned that can be used by the broader public.
Conducted by AmeriCorps contractor ICF, this study will be the first time AmeriCorps will synthetize information about its portfolio of projects in Puerto Rico across funding streams.
To inform the evaluation plan, ICF reviewed relevant literature about the major challenges faced in Puerto Rico related to the disasters, the economy, education, environment, and health. Our document reviews included 19 applications from fiscal year (FY) 2023 AmeriCorps State and National (ASN) grants and Volunteers In Service To America (VISTA) sponsors. We included applications from all 13 organizations that were funded through the Comisión de Voluntariado y Servicio Comunitario with AmeriCorps State and National formula grants. We also included the AmeriCorps VISTA project applications from those same organizations (6 of the organizations had both types of projects). The review of applications illustrated the diversity across programs and funded projects, with variation in activities, outcomes, community-level partner organizations, and target populations. The literature review revealed that evidence was limited regarding promising strategies to address the major issues faced in the commonwealth, as well as the effects of participating in AmeriCorps service on Puerto Ricans.
Spanning 15 months, this bundled evaluation will include 13 organizations with AmeriCorps State and National formula grants as well as those with both AmeriCorps State and National grants and VISTA projects to examine program context, implementation, and outcomes using surveys, interviews, focus groups, and site visits. The evaluation respondents will include a wide range of stakeholders:
Comisión de Voluntariado y Servicio Comunitario staff
Grantee and sponsor program directors and staff
Partner organization staff
National service members
National service member alumni
Community members
Staff from non-AmeriCorps-supported organizations in Puerto Rico
The instruments submitted for approval include the following surveys and interview/focus group protocols that examine the program context, implementation, and outcomes for national service in Puerto Rico:
Alumni focus group protocol
Alumni survey
Member focus group protocol
Member survey
Commission staff interview
Community member focus group
VISTA Alumni focus group
Partner organization interview/focus group
Grantee/organization director/staff interview
Non-supported organization focus group
Pre-Post Evaluation Technical Assistance Program Assessment
Evaluation Capacity-Building Post-Session Survey
Evaluation Capacity Observation Form
Data collection will be overseen by ICF. Surveys will be administered online using Qualtrics platform. Interviews and focus groups will be conducted in-person during the site visits with virtual collection as backup.
Public’s Beginning-to-End Experience. The experience consists of the following steps:
ICF will first work with grantees and sponsors who agree to participate in the evaluation to identify appropriate participants for each data collection.
Grantee and sponsor directors will send a letter (drafted by ICF) to participants to introduce ICF, endorse the evaluation, and encourage individuals to cooperate with data collection requests.
ICF will send recruitment materials and invitations to the participants in advance of the data collection.
For surveys, individuals will receive a link to the consent form and questionnaires; for interviews and focus groups, ICF will coordinate with grantee and sponsor directors on scheduling.
All participants will receive a thank-you note for their contribution.
The surveys will be administered electronically and interviews/focus groups will be conducted in-person or virtually. The time to read instructions and fill out the questionnaires was accounted for in the estimates. Given the nature of the questions, we do not anticipate additional time required for learning about program rules, tracking down records and documents, or consulting with any third parties to help with the response. Since the in-person/virtual interviews will be scheduled as part of the ongoing program activities, we do not expect extra travel time for the respondents. However, we have accounted for time and effort for the grantee and sponsor project director or their designees for assistance in identifying participants and coordinating with data collection (e.g., communicating with ICF, identifying participants, coordinating site visit activities).
Psychological costs. Given the nature of the questions, we do not anticipate any psychological costs including cognitive load, discomfort, stress, anxiety, increased institutional distrust, or loss of sense of autonomy that respondents may experience as a result of participating in data collection. The evaluation approaches, instruments, consent forms, and recruitment materials have been reviewed and approved by ICF’s Institutional Review Board (IRB) for human subject protection.
Learning costs. We fully account for the learning costs in the time estimates. Every attempt was made to minimize the response burden and provide the public with transparency about the burden involved in each of the data collection approaches in terms of response ease and duplication. Multiple approaches and considerations have been applied to minimize burden, such as
Being transparent about the burden. Respondents will be first sent to a short recruitment email prepared by ICF that describes the purposes of the data collection and estimated time, along with endorsement from the grantee and sponsor organizations. Each of the data collections will include a consent form and questionnaire or protocol.
Simplifying submission requirements by using the online Qualtrics survey platform for surveys and scheduling interviews/focus groups at times convenient to the respondents.
Enhancing outreach, navigation, and communication to reduce learning costs. Links to the online surveys will be emailed to potential respondents, allowing them to complete the survey at their convenience without retrieving and/or returning paper forms. Virtual interview and focus groups will be conducted using ICF-secured and managed Microsoft Teams. Surveys will be administered and interviews/focus groups will be conducted in Spanish, with options for English if preferred.
Improving submission processes. We will offer paper survey options for respondents who do not wish to use an online survey as well as virtual interview/focus group options in addition to in-person.
Employing leading design practices. All survey questionnaires and interview/focus group protocols were pilot-tested for readability, clarity, and flow.
The purpose of this evaluation is to provide insights into the unique context of operating AmeriCorps-supported projects in Puerto Rico, including language barriers, operating outside of the continental United States, and recent major disasters. The evaluation will also explore implementation of AmeriCorps supported projects to tackle issues related to disasters, the economy, education, environment, and health. Implementation will also consider how national service members are recruited, onboarded, and retained. Finally, the evaluation will also explore outcomes of the projects on communities, organizations, and national service members. The research questions for this evaluation are:
How are national service and AmeriCorps perceived by grantees, partner organizations, national service members, and the community in Puerto Rico?
What are the grantees’ experiences related to accessing and maintaining AmeriCorps State and National and VISTA projects in Puerto Rico?
What are specific facilitators and barriers to expanding national service opportunities in Puerto Rico for grantee organizations (including those that serve as intermediaries) and non-supported organizations? Is the match in funding requirement a barrier to getting more organizations to apply for AmeriCorps? If so, why?
What are the major community/environmental contextual factors for their AmeriCorps projects—including both assets and challenges?
What have been the impacts of disasters on programs? Has AmeriCorps been able to properly support grantees and national service members?
What is the overall profile of program infrastructure for each organization—including staffing, relationships with partner organizations, governance structure? How many staff are AmeriCorps alumni? Which grantee organizations serve as intermediaries for grant funds?
What were the processes and motivations for each organization to apply for an AmeriCorps State and National grant and, if applicable, to become a VISTA sponsor? What external supports, if any, did organizations receive? In which order did organizations with both types of projects apply for their projects?
How and when are data and other materials translated from English to Spanish? What are the overall challenges when dealing with translation?
What types of programs/services do organizations provide to the communities through their AmeriCorps projects? Are there specific program models they implement? What activities do organizations engage in? What communities are served? What are commonalities across projects in how services are delivered?
To what degree are programs implementing their activities as intended? What are facilitators and barriers to implementation? What is the level and scope of organizational infrastructure supports for implementation?
How is the VISTA sponsorship supporting capacity building efforts in the organization and/or supporting other AmeriCorps programming? What types of organizational capacity building services are delivered through the sponsorship?
What types of data, if any, are grantees collecting? What systems, if any, are grantees using for collecting and storing data? How are grantees using data? In what ways, if any, do grantees use data to assess their work? To what degree are the AmeriCorps performance measures useful to the organizations?
How do grantee organizations recruit, retain, and manage national service members in Puerto Rico? What are effective approaches for national service member recruitment and retention in Puerto Rico? What materials/resources developed by AmeriCorps headquarters (in English or Spanish) do organizations use and how do they use them? How can those materials/resources be improved?
What types of workforce training are national service members receiving and what certifications are they obtaining through their service? What workforce skills are they building? Who is provided the training?
What are the perceptions of national service members related to their member benefits, including the living allowance and education award? What are the reflections of national service members in terms of the importance/ranking of the range of benefits provided to them (e.g. stipend, health insurance, education award)? What other supports do they need?
What are the perceived outcomes of AmeriCorps-supported projects in Puerto Rico on their respective communities as they relate to the following:
Disaster Services – preparing for, responding to, and recovering from natural and man-made disasters
Environmental Stewardship – conserving natural habitats, improving energy efficiency, protecting clean air and water, and helping to mitigate the effects of climate change
Economic Opportunity –building economic opportunity for national service members and populations served
Healthy Futures – improving physical and mental well-being
Education – supporting students in schools and out of school time programs
What are the perceived outcomes of the organization capacity building activities on organizations?
Are AmeriCorps grantees and partners in Puerto Rico able to retain skilled national service members in their communities? What factors contribute to retention of skilled national service members in grantees’ communities?
What are the job pathways for AmeriCorps members and VISTAs in Puerto Rico? How, if at all, does the service experience improve skillsets and employability?
Thematic analyses will be conducted from the data collected through interviews and focus groups. For data collected through surveys, analysis will focus on responses from national service members and national service member alumni. In addition to participating in the evaluation, the grantees and sponsors will participate in evaluation capacity building workshops with 4 virtual sessions, 1 in-person session, and up to 4 individualized technical assistance consultations facilitated by ICF in one year using experiential learning. To that end, ICF will assess the effect of evaluation capacity building using pre–post outcome surveys and post-presentation surveys.
Information gained from this evaluation will be used by AmeriCorps and the Comisión de Voluntariado y Servicio Comunitario to inform future investment and policy decisions, and participating and organizations and partners to adopt evidence-based practices, engage in continuous program improvement, and inform national service member recruitment and retention practices. We will disseminate (1) a final technical report with detailed information on research, evaluation, and capacity building, and (2) practitioner and research briefs on effective strategies or practices for AmeriCorps-supported projects in Puerto Rico. It will be AmeriCorps’ decision regarding whether these materials will be disseminated or used by other Federal agencies. In any event, these materials will be prepared in a way that will not allow identification of any individual.
Several efforts have been made to minimize the burden on respondents. In designing the surveys and focus groups, each question was carefully considered and aligned with the research questions to avoid duplication of data available from other sources.
The surveys will be administered online using the Federal Risk and Authorization Management Program-approved Qualtrics survey platform for secure data collection. Links to the online surveys will be emailed to potential respondents, allowing respondents to complete the survey at their convenience without retrieving and/or returning paper forms. Interviews and focus groups may be carried out in person or virtually. In-person interviews and focus groups will be conducted at a convenient time and location for the participants. Virtual interview and focus groups will be conducted using ICF-secured and managed Microsoft Teams.
The instruments developed for this evaluation go beyond any reporting that AmeriCorps grantees and sponsors currently submit. Development of data collection instruments was informed by extensive review of literature, grantee and sponsor applications, and consultation with the Comisión de Voluntariado y Servicio Comunitario and AmeriCorps supported organizations in Puerto Rico. Instruments seek information regarding grantees and sponsors’ implementation of their projects and the outcomes of this work on individuals and organizations, which is not otherwise being collected. This information is needed to understand program context and implementation, and to determine the outcomes, challenges, and promises of AmeriCorps investments in Puerto Rico for future AmeriCorps grantees and sponsors.
Every effort was made to minimize the length of time for each data collection effort (e.g., surveys take 15–20 minutes).
For interviews and focus groups, the researchers will schedule the data collection at times that are convenient to the participants in an effort to minimize disruption or interference with the normal activities of the respondents. Surveys will be administered online so respondents can respond when it is most convenient for them.
If this information collection were conducted less frequently, AmeriCorps would not be able to evaluate the program. Currently, AmeriCorps has limited information on the context, implementation, and outcomes of grantees and sponsors’ approaches to addressing projects on disasters, the economy, education, environment, and health in Puerto Rico. To properly assess the usefulness of these projects and compile promising practices and challenges, there is an urgent need from the agency and the field to learn about grantees and sponsors’ approaches and the outcomes of these approaches on program implementers, program beneficiaries, and national service members.
As a key component of decision-making about what approaches are effective in addressing these issues in Puerto Rico, these empirical data are important to aid those considering the options and the modifications that may be needed. Without these data, uninformed and inadequate decisions might be made. This input also provides information and insights to help policymakers and researchers determine the research agenda in subsequent years.
There are no special circumstances that would require the collection of information in a manner described in this section.
As required by 5 CFR 1320.8(d), a notice was published in the Federal Register on March 26, 2024, at 89 Fed. Reg. 20952, opening a public comment period of 60 days. No comments were received in response to this notice.
A notice was also published in the Federal Register on July 8, 2024, at 89 Fed. Reg. 55932 with a public comment period for 30 days.
AmeriCorps and its contractor, ICF, conducted the following efforts to consult with individuals and groups both internal and external to the Federal Government to ensure a diverse set of perspectives and experiences related to this information collection effort. Consultations were used to ensure information in the evaluation is not collected elsewhere, disruption or interference with normal activities of the respondents is minimized, and usefulness of the data for program efficacy and best practices. Consultations also contributed in determining burden estimates as indicated in Section A12. Consultations included:
A technical working group made up of six external experts in national service and evaluation methodologies was invited to comment on the evaluation and data collection in February 2024.
A field working group made up of AmeriCorps program impact and operation leaders in ASN and VISTA offered input regarding the evaluation in February 2024. It offered input regarding the evaluation plan and data collection. Additionally, the CEO and the Chief Program Operations offices were consulted to ensure that the project aligned with the agency’s strategic goals, including goals and objectives to better serve communities in Puerto Rico.
In January 2024, the Office of Research and Evaluation presented the evaluation project goals and research questions to the Puerto Rico Board of Commissioners and received positive feedback.
Outreach consultations with individuals representing potential state commission staff, grantee and sponsor directors, and organization staff were held between December 2023 and February 2024. A presentation about the project goals was delivered in December 2023. A follow-up presentation was delivered about the research plan in February 2024. During both presentations, participants had the opportunity to ask questions and share feedback on data collection processes, their roles in the project, and the burden on their organization staff.
Community members and national service member alumni who participate in focus groups will be offered a research incentive consisting of a $25 gift card as a small token of appreciation for their time. National service member alumni who respond to the survey and opt to receive a gift card will be eligible to receive one of 15 gift cards. In total, 15 gift cards, worth $25 each, will be provided to the first 15 respondents to the survey.
The incentives will be used to improve data quality in focus group participation and survey responses. In addition, for the community focus group members, the incentive will help to minimize burden on the participants for taking the time to participate in the hour-long data collection. Past efforts to collect data from community members for the Life Cycle Evaluation project have demonstrated poor response rates. The incentives for the current study will be critical for counteracting this trend. (This rationale aligns with guidance from the following OMB guidance document: Questions and Answers When Designing Surveys for Information Collections (whitehouse.gov).
No other gifts or payment will be offered to respondents.
ICF will comply with AmeriCorps privacy policies and procedures and adhere to AmeriCorps standards. In addition, all of the data collection instruments, consent forms, and recruitment materials will be reviewed by ICF’s IRB, to ensure confidentiality. In the reporting, there will be no identifying information supplied with findings that would permit a comment or finding to be associated with an individual or a position.
Since the evaluation is of grantees and sponsors’ approaches to addressing disasters, the economy, education, environment, and health in Puerto Rico, there are no questions of a sensitive nature or of matters that are commonly considered private.
Exhibit A-1. Estimated hour burden
Comisión de Voluntariado y Servicio Comunitario staff
Grantee and sponsor program directors and staff
Partner organization staff
National service members
National service member alumni
Community members
Staff from non-AmeriCorps-supported organizations in Puerto Rico
Respondent Category |
Number |
Participants |
Time (minutes) |
Total Hours |
Comisión de Voluntariado y Servicio Comunitario staff interview |
1 |
3 |
60 |
3 |
Grantee/sponsor director interview #1 (initial interview) |
1 |
38 |
45 |
28.5 |
Grantee/sponsor director interview #2 (follow-up interview) |
1 |
38 |
60 |
38 |
Partner organization interviews |
1 |
38 |
60 |
38 |
National service member surveys |
1 |
97 (assuming 50% response rate from a pool of 193 national service members) |
20 |
32.3 |
National service member focus group |
1 |
104
|
60 |
104 |
National service member alumni surveys |
1 |
150 (assuming a 10% response rate from a pool of 1,500 alumni from the last 10 years) |
20 |
50 |
National service member alumni focus group |
1 |
65 |
60 |
65 |
Community member focus group |
1 |
104 |
60 |
104 |
Staff from non-AmeriCorps-supported organizations in Puerto Rico |
1 |
8 |
45 |
6 |
Evaluation Technical Assistance Pre–Post Surveys |
2 |
26 |
10 |
8.7 |
Evaluation Technical Assistance Post-Workshop/TA Session Surveys |
9 |
26 |
10 |
39 |
Grantee/sponsor director assistance in data collection |
NA |
NA |
780 |
13 |
|
Total response |
697 |
Total hour |
529.5 |
|
Annual total responses (rounded) |
697 |
Annual total hour (rounded) |
530 |
Overall, the total estimated nonrecurring costs is $7,825.27, which is broken out as follows:
Organization staff (commission staff, grantee/sponsor staff, and partner organization staff): The estimated total cost is $2,924.82, which is based on an estimated hourly wage rate of $16.79 and total burden hours of 174.2. Wage estimates are based on mean hourly wage data for community and social service occupations in Puerto Rico in May 2022, which was $15.52 according to the Bureau of Labor Statistics, accessed at https://www.bls.gov/oes/current/oes_pr.htm#21-0000. Adjusting for 4% annual wage inflation, the rate for 2024 is estimated to be $16.79.
National service member alumni and community members: The estimated total cost is $3,679.20, which is based on an estimated hourly wage rate of $16.80 and total burden hours of 219. Given that alumni and community members may be working in any profession, wage estimates are based on mean hourly wage data for all occupations in Puerto Rico in May 2022, which was $15.54 according to the Bureau of Labor Statistics, accessed at https://www.bls.gov/oes/current/oes_pr.htm#21-0000. Adjusting for 4% annual wage inflation, the rate for 2024 is estimated to be $16.80.
National service members: The estimated total cost for national service members is $1,221.25 for 136.30 hours. Assuming a 7-hour work day, the average hourly stipend yields $8.96 based on a daily living allowance of $62.69 in 2023, accessed from https://americorps.gov/sites/default/files/document/FY23-County-Living-Allowance-Rates-VISTA.pdf.
For designing and implementing the evaluation, capacity building, and dissemination of the Puerto Rico bundled evaluation, the estimated annual cost to the federal government is $486,000. This number is based on the portion of the contract with ICF for FY 2020–2025 that is devoted to this evaluation.
No change in burden is requested. This submission to the Office of Management and Budget (OMB) is for an initial request for approval.
A report of findings, tools, and a brief will be prepared for AmeriCorps. Publication and dissemination will be decided by AmeriCorps after the completion of the evaluation in 2025.
AmeriCorps does not seek this exemption. The OMB expiration date will be displayed on the introductory page of the data collection instrument.
AmeriCorps is not requesting an exception to the certification requirements.
See Part B Supporting Statement document.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |