20240312 life cycle evaluation PR SSB

Corporation for National and Community Service

Puerto Rico Bundled Evaluation

Justification – Part B Supporting Statement

For this evaluation, we will conduct online surveys of national service members and national service member alumni to understand the program implementation and outcomes, with a special focus on the outcomes on the members themselves.

B.1. Respondent Universe

The universe of projects to be included in this data collection is shown in Exhibit B-1. Since the full universe of projects will be included, no statistical sampling methods will be implemented.

Exhibit B-1. Estimated size of universe to be sampled in surveys

Population	Universe:	Sample:
Organizations with AmeriCorps projects	13	13
National service members supported by grantee and sponsor organizations	193	193
National service member alumni	1,500	1,500

B.2. Statistical Methodology

B.2.1. Statistical Methodology for Stratification and Sample Selection

For these surveys, we will include all national service members from all 19 fiscal year (FY) 2023 AmeriCorps State and National formula grantee organizations and AmeriCorps VISTA sponsors in Puerto Rico. National service member alumni from the past 10 years will be further identified by the grantees and sponsors.

B.2.2. Estimation Procedure

We will generate descriptive statistics for each item in the surveys. Descriptive analyses including means, frequencies, and standard deviations will help to answer questions about program context, implementation, and outcomes.

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Jurisdiction

All statistics produced will be descriptive. No inferential statistics will be used.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No unusual problems are anticipated.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

The surveys will be administered in fall 2024.

B.3. Methods to Maximize Response Rates and Deal With Issues of Non-response

For national service members and national service member alumni, a list of individuals will be provided to ICF by participating grantees and sponsors. The list will include names and emails.

Before each data collection, participating grantees and sponsors will email a notification and encourage all perspective respondents to participate in the survey. Within one week of the prenotification, ICF will follow up with a recruitment email providing a link to the
20-minute survey URL delivered via Qualtrics. Before responding to the survey, participants will be asked to read the consent form describing the purpose of the survey. The form assures that participation is voluntary and that responses are confidential and provides contact information for the study lead and Institutional Review Board (IRB). After providing consent, participants will proceed to respond to the survey. Respondents will be given 1 week to respond to the survey. The research team will continue to monitor the completion. We will send two reminders on a weekly basis to all survey participants with a note to disregard the email if they completed the survey. Respondents will be sent a “thank you” email for their participation.

We estimate the response rates for national service member survey to be 50 percent, and 10 percent for alumni. The response rate for national service members is based on a rate from a similar bundle evaluation on climate change funded by the life cycle evaluation project as well as factoring in the enthusiasm so far for the evaluation by the grantees and sponsors in Puerto Rico. A non-response bias analysis will be conducted using auxiliary data, including demographics, from the sampling frame. Any strong predictors of non-response may be candidates for building a response propensity model, which could then be incorporated into the non-response adjustment as part of the survey weights for this group.

B.4. Tests of Procedures and Methods

The survey design, data collection procedure, and analysis approaches have been reviewed and approved by the AmeriCorps Office of Research and Evaluation. In addition, as described in Section A.8., the research plan has been reviewed by major evaluation stakeholders—including the Comisión de Voluntariado y Servicio Comunitario and the 13 grantee/sponsor organizations participating in the evaluation. Further, an external technical working group consisting of six nationally recognized experts in evaluation of national services were invited to comment on the research plan. In addition, a field working group consisting of 6 program leaders in AmeriCorps weighed in on the plan. Finally, the survey instruments, consent forms, and recruitment letters will be reviewed and approved by the ICF IRB.

B.5. Names and Telephone Numbers of Individuals Consulted

Agency Unit

• Amy Borgstrom, Associate Director of Policy, COO Immediate Office, AmeriCorps, 202-606-6930 

• Jehyra M. Asencio-Yace, Research Analyst Team Coordinator, Office of Research and Evaluation, AmeriCorps, 202-956-9736

Contractor

• Samantha Spinney, Project Director and Principal Investigator, ICF, 703-272-6681

2