RSVP Survey SS Part B 5.29.14

Part B: Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling

During Fiscal Year 2013 there were 210 RSVP grantees with over 150,000 volunteers that submitted work plans for future service plans. Approximately 150 of the grantees submitted Periodic Progress Report (PPR)data in April 2013. The PPR data will be usedas the sampling frame for this evaluation. The other 60 grantees will submit their PPR data in August 2013, which makes it too late to include them in the evaluation. The division of when grantees report PPR data is purely administrative. There is no known systematic difference between the two groups. The advantage of using PPR data is that it provides actual performance measure data, as opposed to projected data reported in the work plans. The limitation of using the PPR data is that the results will not be generalizable to all Fiscal Year 2013 grantees. RSVPgrantees support senior volunteers in six focus areas: Disaster Services, Education, Environmental Stewardship, Economic Opportunity, Healthy Futures, and Veterans and Military Families. Volunteers also address Community Priorities, and Capacity Building. Service activities are as varied as providing companionship to other seniors, building homes, supporting students in educational settings and recruiting other volunteers.

Among Fiscal Year 2013 grantees, the range in the number of unduplicated volunteersper grantee is 81 to 6,000 volunteers. The universe from which the sample will be drawn has a hierarchical structure such that RSVPvolunteersexist within RSVPgrantees or projects. An additional level of structure isthat volunteersare nested within sites or stations. Moreover, volunteers may be engaged in multiple work plans within grantees or stations. Therefore, the unduplicated count of volunteers may include volunteers who are counted in multiple work plans.

JBS will draw a sample of 1,500RSVP volunteers, yielding a final sample size of 1,200 assuming a response rate of 80 percent. It is assumed that all volunteers devote hours on at least one workplan and it is anticipated that, based on the available 2013 data, approximately 76 percent of volunteers devote hours on either a Primary Focus area or Other Focus Area/Capacity Building.

This study will samplevolunteers who are nested within grantees and stations. The sampling unit is the volunteer and the sampling universe is the universe of unduplicated RSVP volunteers from the Fiscal Year 2013grantees who will report their PPR data in April 2013. JBS will select the sample of volunteers using a three-stage stratified probability proportionate to size(PPS) sample that is expected to result in an equal probability sample. The size measure will be the number of unduplicated RSVP volunteers. At the first stage, a sample of grantees will be selected from the cohort of grantees reporting PPR data in April 2013 using PPS. In order to assure there is adequate representation of grantees of varying capacity, the grantees will be stratified into three groups based on the estimated number of unduplicated volunteers.At the second stage, the sampled grantees will be asked for a roster of stations and the number of volunteers at each station. A sample of stations will be selected from this list using PPS, where the size is defined by the number of unduplicated volunteers. At the third stage, stations will be contacted and full volunteer lists will be requested. Asimple random sample of volunteers will be drawn from these lists.

In the first stage of sampling JBS will divide the grantees into three strata based on the projected unduplicated volunteer counts from the most recently available 2013 grantee data: Large Grantees (those with more than 900 unduplicated volunteers), Medium Grantees (501-900 volunteers) and Small Grantees (1-500 volunteers). A total of 33 grantees will be sampled for inclusionin the study. Table 3 shows the number of grantees in each stratum, total and average number of estimated unduplicated volunteers for each stratum, and the proposed number of sampled grantees within each stratum. Based on previous experience conducting studies with Senior Corps grantees, we anticipate that the actual number of unduplicated volunteers will be lower than the estimated number that is recorded in CNCS’s database. The variance between the actual and the projected number is expected to be greater among the small and medium size grantees. In order to account for this variance, and to assure there is a sufficient pool of volunteers to meet the allocation within each stratum, JBS will over-sample the number of small and medium size grantees to be included in the study. As illustrated in Table 5, JBS will include 14 grantees in the small stratum and 13 grantees in the medium stratum. In the large stratum, the largest grantee with an estimated number of 6,000 unduplicated volunteers will be selected with certainty; an additional 5 grantees will be selected in the large stratum. Within each stratum, grantees will be selected proportional to the number of unduplicated volunteers.

Table 5.RSVP Sampling

		Estimated Average Number of Unduplicated Volunteers
Stratum	Actual Number of Grantees	Mean	Range	Proposed Sampled Grantees
Small	79	335	1-500	14
Medium	69	690	501-900	13
Large	62	1407	910-6006	6

Note: The table is based on current data for all Fiscal Year 2013 RSVP grantees.

For the second stage of sampling, JBS will select stations within grantees using the following process:

1. JBS will obtain information on stations/sites from sampled grantees. This intermediate step will provide information on the number of stations and expected number of volunteers within each station for each of the sampled grantees. The information on stations will inform whether we will sample stations within grantee using probabilities proportional to size (PPS). If a selected grantee only has one station/site, then the station/site is selected with certainty.

2. Each grantee will provide a list of their stations. The list should include the following information:

1. Each station’s unique identifier  (e.g., name)

2. Number of volunteers at each station

3. City, state, zip code and 4-digit zip for each station

For the third stage of sampling, JBS will select a sample of volunteers within each station using the following steps:

1. JBS would obtain volunteer lists from each sampled station/site.

2. The list of volunteers will inform whether for any station we will draw a random sample of volunteers.The number of interviews allocated to each station will be proportional to the number of volunteers at that location. If volunteers are to be sampled, JBS will use simple random sample.

The list of volunteers will include full contact information such as name, contact phone number, primary language spoken to assist with assigning interviewers, and the volunteer’s primary assigned work plan. This information will be transferred to JBS via a secure upload process. Grantees and stations will receivethe names of the sampled volunteers andwill be asked to contact these volunteers to let them know they have beenselected to participate in the study.

The expected sample size of 1,200 volunteer surveys will allow analysis at95 percent confidence interval of less than +/-5 percentage pointsfor two subgroups defined by Performance Measure Categories.The first subgroup will consist of all volunteers under the Primary Focus Area. The second subgroup will consist of volunteers under the Community Priorities Focus Area or Capacity Building Focus Area. Table 6shows the minimum detectable effect size for analysis of responses within the survey with a power of 0.80 and alpha value of 0.05. The sensitivity analysis shows that an effective sample size of 1,200 responses should allow us to detect effect size of about 0.08 to 0.10 depending on the statistical test performed. Subgroup analyses with a total of 600 responses within the subgroup should allow us to detect effect size of about 0.11 to 0.14. The stratified probability proportional to size sampling design proposed here is intended to be an equal probability sample. Although this equal probability samplingdesign deviates from a simple random sample, the design effect is expected to be close to 1.

Table 6.Effect Size

Effective Sample Size	Point Biserial Correlation	Goodness-of-fit Test
600	0.11	0.15
700	0.10	0.14
800	0.10	0.13
900	0.09	0.12
1000	0.09	0.11
1100	0.08	0.11
1200	0.08	0.10

Note: The point biserial correlation model assumes a two-tails t-test; the goodness-of-fit model is based on a chi-square test with 5 degrees of freedom. Both models assume a power of 0.80 and alpha value of 0.05.

2. Procedures for Collection of Information

JBS staff will obtain a list of the 210RSVP grantees from CNCS.The list of grantees will include the name of the grantee or sponsor organization,a list of workplans and the correspondingperformance measures activities, outputs and outcomes, the number of unduplicated volunteers per workplan, and the state and city where the granteeis operating. Each grantee will provide JBS with a list of stations and the number of volunteers at each stationthat will be needed to draw the total sample of 1,500volunteers. The list will include the grantee/sponsoring organization, name and contact information for the grantee’s project director, the city and state where the volunteer station is operating, the total number of volunteers at each station, contact information for the stations and the sampled volunteers, and each sampled volunteer’s assigned primary work plan. JBS proposes to use probability proportional to size sampling, drawing upon the universe of 210 grantees. That is, JBS will first select 33 grantees from the universe of 210 grantees. CNCS will notify each of the sampled grantees that they have been selected to participate in the study. Following notification from CNCS, JBS will contact the 33 project directors to explain the study, and to obtain information about the number of stations/sites and volunteers for their projects. Once the lists of stations/sites and volunteers are received from each RSVP project, a random sample of volunteerswill be selected proportional to the number of volunteersserved by the grantee at their stations or sites.

Each RSVP projectwill be provided the list of theirvolunteers sampled for the study. CNCS and JBS will engage proactively with theprojects to help them encourage their volunteers to participate to maximize the response rate. To facilitate this JBS will draft an invitation letter for RSVP project directors to give to the sampled volunteers. Theinvitation will introduce JBS and its function in the study, and succinctly inform potential respondents of the study’s risks and benefits, and provide assurance of anonymity. Respondents will also receive information on whom to contact if they have questions. Respondents will be informed of the voluntary nature of the research and that their responses will be anonymous.

Trained interviewers will make three contact attempts for each sampled volunteer. Upon contacting respondents, trained interviewers will explain the study and will provide information on who is conducting the study and why the study is being conducted. Interviewers will describe the background, research activities, risks and benefits, and provide assurance of anonymity. Respondents will also receive information on whom to contact if they have questions. Respondents will be informed of the voluntary nature of the research and their right to end participation at any time. Respondents will be informed that their responses will be anonymous. Once the interviewing process begins, interviewers will enter responses directly into the CATI system. The CATI has built-in protocols to monitor data collection activities such as progress, response and reasons for nonresponse for each contact attempt. In addition, research staff will review the data daily, as they are conducted, for completeness and quality. Research staff will periodically download the data to cross-check queries so that potential missing fields and other discrepancies can be founded and corrected as part of the interviewing process.

• Weights

While the PPS design proposed results in equal probabilities of selection, should there be unforeseen issues that make sampling or non-response weights appropriate, JBS will calculate them. JBS will perform quality checks on the weight calculation to catch programming or logic errors.

3. Methods to Maximize Response Rates

Oversampling of volunteers will be used to ensure satisfactory response rates. We assume an80 percent response rate. CNCS will develop messaging to use with grantees that will emphasize the positive value of data collected and their participation.

4. Tests of Procedures or Methods

The primary objective of the proposed research is to examine the distribution of volunteers across types of work plansand provide information on RSVP volunteer characteristics, volunteer service activities and their psychosocial health. The proposed evaluation is also designed to inform two future studies: 1) allow for a possible quasi-experimental study, and 2) provide information on the relationships between the volunteer service activities in a work plan and performance measurement data submitted by grantees in the future.

The strength of this plan is that it uses an equal probability sample which makes the analysis more straightforward and decreases complications in matching the volunteer data to another data for a quasi-experimental study. The drawback of a design that samples unduplicated volunteers is that the number of responses per work plan will vary and though it is likely that the number of work plans with sampled volunteers would increase, it also means that the ability to report on individual work plans tied to outcomes would be reduced.

5. Contact Information

   1. The following individuals were consulted on statistical aspects of the design:

Susan Gabbard, PhD

Annie Georges, PhD

Nicole Vicinanza, PhD

Peter Lovegrove, PhD

JBS International, Inc.

Aguirre Division

555 Airport Boulevard, Suite 400

Burlingame, CA 94010

T: 650.373.4949 (direct)

T: 650.373.4900 (main)

F: 650.348.0260 (fax)

E:sgabbard@jbsinternational.com

Robert L. Fountain, PhD

University of Oregon

8601 SW 50th Ave
Portland, OR 97219
E: bob@bobfountain.com

Field Working Group Members

Carole Mandino

Director, NAU RSVP Program

E:carole.mandino@nau.edu

Betty Ruth

Executive Director, Alabama RSVP Program

E:bruth@al-rsvp.com

MelodyeKleinman

Executive Director, RSVP of Santa Monica

E:mkleinman@ntvcc.org

GerriZabusky

Director, RSVP of Rockland County

E:gzabusky@sunyrockland.edu

2. Name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency

JBS International, Inc.

Aguirre Division

555 Airport Boulevard, Suite 400

Burlingame, CA 94010

Page 5 of 5