CEY OMB Case Studies Part B

OMB Clearance Submission

Communities Empowering Youth (CEY) Evaluation Study

June 10, 2008

Prepared for

John Tambornino

DHHS/ACF/OPRE

370 L’Enfant Promenade SW, 7th Floor

Washington, DC 20447

Part B: Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

Given limited resources, it is not possible to conduct in-depth longitudinal case studies with all grantees. Our aim is to select 10 grantees that capture a range of factors and dimensions related to capacity growth and partnership development. We propose to use a purposive grantee selection process. This method will provide us with a final set of grantees that reflect different characteristics of the CEY grantees to ensure that the selection allows us to explore the implications of various factors in these partnerships.

Based on our prior experience working with CCF and CEY grantees, we propose a sample that includes a mix of lead organizations that are faith-based and community based, and leads whose primary activity is capacity building as well as those who engage in both direct service and capacity building. In addition, the sample will include partnerships of varying sizes, and a mix of partnerships that serve primarily rural and urban areas. To the extent possible, the sample will include partnerships that utilize different partnership structures and that are focused on specific populations (e.g., Tribal Agencies).

Selection Process

We have reviewed basic information contained in the CEY grant applications, which provides us with initial information related to many of these factors. However, the applications do not always provide sufficient information for classifying partnerships on all of the dimensions of interest. Thus, we propose to use the information collected from grantees through the baseline survey to confirm faith-based characterization, partnership structure, rural vs. urban service area, partnership size, and primary activity of the lead organization. Once we know how many grantees have each of these characteristics, we will determine the number of grantees to be selected from each group.

Selecting Grantees to Ensure Variation on Key Characteristics. We have identified the following four key characteristics or features of the lead grantee and/or the partnership that we will take into account in site selection:

• Faith-based and secular lead organizations

• Lead organizations that do both direct service and capacity building as well as organizations whose primary role is capacity building

• Partnerships of varying sizes

• Partnerships that serve primarily rural and urban areas

These characteristics are of interest as they may be associated with differences in partnership development and improved capacity over time.

Selecting Grantees with Unique Characteristics. We will identify partnerships with unique characteristics that occur with low frequency that we want to select with certainty. Based on the application information, one partnership appears to use the “peer network” or “cascading” partnership model, and one grantee is a tribal agency. These two grantees provide interesting and relevant variation with distinctive features that may affect the partnership and the development of the partnership over time.

Taking into account those selected with certainty, we will determine the remaining number to be selected for each key characteristic. A matrix that categorizes grantees on all key characteristics will aid in organizing the selection process. We will identify 10 grantees that together include these key characteristics and that also contain unique traits of interest. In situations where two or more partnerships would fulfill the desired mix of characteristics, the final selection would take other variables such as geographic region into account.

B.2 Information Collection Procedures

The information collection will be through interviews with key persons during annual on-site visits and telephone calls between site visits. Prior to conducting the site visits, respondents will be notified about the study via an introductory letter, which will include the purpose of the study. Evaluation team members will meet in person with each CEY Project Director, other relevant staff from the lead organization, directors of CEY partner organizations, and key partner staff. During the meetings, evaluation staff will conduct interviews using the Lead and Partner Interview Protocols included in this submission. Two person teams will conduct the interviews; while both team members will participate in the exchange, one will be assigned the role of primary note taker for each interview.

B.3 Methods to Maximize Response Rates

To maximize response rate from the selected partnerships (lead and partner organizations), ACF has stressed to the grantees the importance of the evaluation. ACF explained the purpose of the evaluation at the CEY 2008 Orientation & Leadership Workshop January 17–18, 2008. Site visits will be scheduled in coordination with the CEY grantees and partner organizations.

B.4 Test of Procedures

We will pilot the interview protocols with 2 of the 10 CEY sites selected for the CEY longitudinal case studies. The pilot test will serve as the first site visit for these sites, and the data collected will be analyzed and included in the first interim report. We will pilot the protocols for interviews and observations and the process for handling field notes. After the pilot test, we will modify the instruments as necessary to clarify questions that were not well understood or reorder the sequence of topics. The piloting of the protocols will not involve more than nine individuals.

B.5 Individuals Consulted on Statistical Aspects of Design

These data collection plans were developed by Branch Associates, Inc., and Abt Associates Inc. The research team is led by JoAnn Jastrzab, Project Director. Other members of the evaluation team who worked on the design include: Barbara Fink (Branch Associates), Cynthia Sipe (Branch Associates), Matthew Coll (Branch Associates) and Ryoko Yamaguchi (Abt Associates). Contact information for these individuals is provided below.

JoAnn Jastrzab

Abt Associates Inc.

617-349-2372

Barbara Fink

Branch Associates, Inc.

215-731-9980

Cynthia Sipe

Branch Associates, Inc.

215-731-9980

Matthew Coll

Branch Associates, Inc.

215-731-9980

Ryoko Yamaguchi

Abt Associates, Inc.

301-634-1778

. Part B: Collection of Information Employing Statistical Methods 2