TA OSY Supporting_Statement_B

Request for Clearance of Data Collection Instruments for the

Re-Engagement Center study/Resource Guide

II. SUPPORTING Statement For Paperwork Reduction Act Submission

Part B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sample Selection

The research team (The Millennium Group and Johns Hopkins University Center for Social Organization of Schools) will collect the information during one-time, one-day site visits to five re-engagement centers, and conduct one-time 90-minute phone interviews with lead staff from an additional five re-engagement centers. Because re-engagement centers are a relatively new phenomenon, qualitative interview and observation methods are necessary to produce the detailed descriptive data to inform the development of the Resource Guide (the primary contract goal and deliverable). The data collected through this study may inform the development of a survey in the future, but collecting data via a survey at this time would be premature as key operational components and performance metrics of re-engagement centers are currently un- or under-specified.

Respondents for the site visits will represent the range of participants and stakeholders in the re-engagement center initiatives (key informants). These key informants will include re-engagement center project directors and staff, youth clients, and representatives of organizations and school districts/schools that work in partnership with the centers. Respondents for the phone interviews will be the re-engagement center project directors and another lead staff member identified by the project director.

The five site sample identified for visits is made up of sites selected from a list developed through consultation with the COR and experts in the field serving on the project’s technical working group (TWG). The following priorities guided the choice of the five sites: Geographic Diversity (regional, urban, suburban, rural); Established (sites in existence for a minimum of three years); Evidence (sites that have demonstrated evidence of student and program success, with an eye on specific sub-groups of youth, e.g. special needs, minority, and immigrant youth); Variation (sites representing different organizational models); and Accessibility (sites with streamlined processes for on-site data collection as evidenced by local IRB processes).

The additional five site sample slated for phone interviews also is made up of sites selected from the list developed through consultation with the COR and experts in the field serving on the project’s technical working group (TWG). These sites represent less established re-engagement centers, but will offer additional variation both in program operation and development context to the data informing the Resource Guide.

Table B.1. identifies the number of sites according to the quantifiable selection criteria (Geographic Diversity and Established). A list of all sites identified for visits and calls and the rationale for their selection are summarized in Table B.2. We expect a 100% response rate from the 10 sites. Data has not been collected previously from these sites.

Table B.1: Sample Sites by Primary Selection Criteria

Collection Method	# Sites	Geography				Established
		Northeast	Midwest	West	South	3+ Years
Site Visit	5	2	1	1	1	5
Call	5	1	2	1	1	2
Total	10	3	3	2	2	7

Table B.2: Proposed Sites, Research Method, and Selection Rationale

	Name/Location	Research Method (visit or call)	Selection Rationale
	Boston Re-Engagement Center Boston, MA	Visit	Geography: Northeast, urban Established (in operation since 2009) School District is host agency Evidence of impact (urban, minority youth) Service model: Intake and referral Accessibility: High Unique aspect—started outside the school system as part of workforce development initiative. Private Industry Council remains key partner. Site recommended for study
	Transition Education Employment Management (TEEM) Gateway/YES Center Newark, NJ	Visit	Geography: Northeast, urban Established (in operation since 2008) Higher Ed (Rutgers) is host agency Evidence of impact (minority youth) Service model: recruit, intake, counsel, connect to education & employment Accessibility: High Unique aspect—part of multi-partner service system. Has expanded to Trenton and Camden. Site recommended for study
	Fast Forward Center Dayton, OH	Visit	Geography: Midwest, urban Established (in operation since 2001) Higher Ed (Sinclair Community College) is host agency Evidence of impact Service model: intake, assessment, placement Accessibility: Moderate Unique aspect—longest continuously operating re-engagement center Site recommended for study
	Reconnection Youth Center (RYC) Portland, OR	Visit	Geography: Northwest, urban Established: 2010 School District is host agency Evidence of impact Service model: Intake, referral, direct education service, follow-up Accessibility: High, members of research team based in Portland so minimal budget impact Unique aspect— The centers use consistent data-driven analysis of student strengths and challenges based on academic, career interest and social assessment.  Site recommended for study
	Youth for a Change Denver, CO	Visit	Geography: West, urban Established (in operation since 2005) Non-profit (Colorado Youth for a Change) is host agency Evidence of impact Service model: Intake and referral Accessibility: High Unique aspects—Emerged from citywide mobilization effort; drew from data analysis of student characteristics that predicted dropout; specialists working across six school districts; virtual and online services; explicit system-building approach Site recommended for study
	Philadelphia Re-Engagement Center Philadelphia, PA	Call	Geography: Northeast/Mid-Atlantic, urban Established (in operation since 2008) School District is host agency Evidence of impact Service model: Intake and referral Accessibility: High Unique aspects—operates as one of 55 programs under the district’s multiple pathways program Site recommended for study
	D2 (Directions, Diploma) Center Omaha, NE	Call	Geography: Midwest, urban New (opened in summer 2011) Non-profit (Building Bright Futures) is host agency Service model: Intake, referral, direct service, follow up Accessibility: High Unique aspects—emerged from community mobilization effort; funded through Buffett family foundation Site recommended for study
	Dubuque, IA	Call	Geography: Midwest, rural New (opened in summer 2012) Non-profit (Community Foundation) is host agency Service model: Intake, referral, direct service, follow up Accessibility: High—participated in Denver conference and interested in national network Unique aspects—partners represent community college, school district, and community-based youth services.
	Los Angeles, CA	Call	Geography: West, large urban New (five centers some newly operating, some in planning phase) Non-profits (13 Community Based Organizations) serve as host agencies Service model: tbd Accessibility: High—participated in Denver conference and interested in national network Unique aspects—thoughtful connection between dropout prevention and youth engagement in mega-district. Some in planning phase—will give researchers insight into types of questions new sites have that could be addressed in the Resource Guide
	South TX	Call	Geography: South/Southwest, rural towns Established (in operation since 2007) Pharr-San Juan-Alamo school district is host Service model: door-to-door outreach and connection to appropriate education pathways Accessibility: moderate—has hosted visitors but is not a “place.” Unique aspects-- Community-wide outreach effort to bring back every dropout and reenroll them in appropriate option

2. Data Collection

The sampling issues related to the site visit data collection activities are covered in the previous section and described in the data collection tasks and deliverables described in Supporting Statement A.

3. Methods to Maximize Response Rates

TWG experts have already identified project directors for each site identified for a visit or a call. Project directors will then be asked by the research team to help to identify key informants for site visits. We will work with the project directors to identify the most appropriate way to gain access to “off-site” partners and school system employees and provide scheduling templates when necessary. Multiple attempts will be made to reach identified respondents including phone calls, emails, and follow-up by research team staff. In the rare cases when identified informants cannot be reached or are unavailable during the time period of the site visits we will work with the project directors to identify alternate informants. Informants will be contacted by experienced and well-trained interviewers who will introduce the study by providing relevant background and rationale. In similar studies we have found that interviews such as these provide a venue for respondents to share experiences and contribute to the body of knowledge which motivates many respondents. In addition we have taken the following steps to maximize participation and minimize respondent burden:

• We have worded all data collection instruments as concisely as possible. To the extent possible, we will coordinate data collection activities within the research team to ensure that these activities impose a manageable burden on respondents, while yielding data that collectively answer the research questions of most interest to policy-makers and the field.

• Prior to data collection, we will send letters of introduction to project directors informing them of the study and describing all relevant data collection activities. The letters will include: (1) contact information for research team staff who can answer questions about the study, (2) information about OMB clearance, and (3) contact information for the study team leaders.

4. Protocol Development and Review

The TMG/JHU team developed five protocols designed to help team members systematically collect information from program directors, staff, youth clients, and community and school system partners in the ten re-engagement center sites as outlined in the Data Collection Plan for the project. The research team used two methods for testing draft interview protocols—pilot interviews and protocol review.

Pilot Interviews: Because the Project Director (PD) protocol is the primary tool the research team will use to gather information about the re-engagement centers, the research team tested this protocol via three pilot interviews followed by debrief sessions with individuals who had direct experience leading a re-engagement center. Interviews were between 60-90 minutes in length followed by a 15-20 minute debrief using a set of guiding questions. One pilot interview was conducted in person by TMG staff with a recent director of the re-engagement center initiative in Portland, OR. The other two pilot interviews were conducted via phone by TMG staff with current project directors of re-engagement centers in Chicago and in Washoe County, Texas. These pilots enabled the team to collect feedback from knowledgeable and experienced leaders and to test the PD protocol in the context of initiatives representing important variations in size, longevity, and geography. They also gave the team the opportunity to test how the protocol worked for both in-person and phone interviews.

Protocol Review: In addition to the pilot interviews of the PD protocol, the team retained three other individuals to carefully review all five draft protocols (project director, staff, youth client, school district partner, and community partner) and provide feedback via a structured questionnaire. Each of these reviewers have extensive experience in education research, reform, and program development aimed at improving outcomes for at-risk youth. The reviewing and feedback process took between 90-120 minutes for each reviewer.

In general, the pilot interviews and reviews found that the initial drafts were strong. The assent scripts were clear, the time estimates were realistic, and the questions covered the major substantive areas that needed to be addressed to achieve the goals of the research. The protocols also worked for both in-person and phone interviews.

The testing process did indicate need for refinements of the research plan and some revisions to the protocols. For example, the Chicago phone interview revealed that a site may have more than one project director in large cities/districts, and that turnover in these positions may require contact with a prior project director. We responded by increasing the number of potential project director respondents from 1-2 individuals per site to 1-3 individuals. Other feedback focused on: specifying different types of counseling youth received through the Center (e.g. career/college, academic; health/mental health, or social services); including teen parents as a possible targeted subgroup; and asking about youth who disengage and dropout more than once (i.e. youth who cycle through the re-engagement program multiple times). In total, the research team responded to a dozen points of feedback. These responses were detailed in a memo to the COR and responded to via revisions to the interview protocols accompanying the memo (tracked-changes and final versions).

5. Contact Information

The contact person at the U.S. Department of Education is Lynyetta Johnson. The primary contractor of this study is The Millennium Group, based in Sterling, VA. The Johns Hopkins University Center for Social Organization of Schools/School of Education is the subcontractor. The principal investigators of the study are Dr. Nettie Legters and Dr. Leslie Rennie-Hill and the project director is Michelle Feist. Data collection will be conducted by researcher team members from The Millennium Group and Johns Hopkins under the direction of Dr. Legters. The contact information for these individuals is as follows:

Michelle Feist
Project Director
The Millennium Group International
46169 Westlake Drive – Suite 240
Sterling, VA 20165
410-542-9959
michelle.feist@tmgi.net

Leslie Rennie-Hill
Project Director
The Millennium Group International
46169 Westlake Drive – Suite 240
Sterling, VA 20165
503-381-4164
leslie.rennie-hill@tmgi.net

Nettie Legters, Ph.D.

Research Scientist

Center for Social Organization of Schools

School of Education, Johns Hopkins University

410-516-8800

nlegters@jhu.edu

4