ISIS Followup OMB Part B Justification REV 7 26 13

This document presents Part B of the Supporting Statement for the follow-up data collection activities that are part of the Innovative Strategies for Increasing Self-Sufficiency (ISIS) evaluation sponsored by the Office of Planning, Research and Evaluation in the Administration for Children and Families (ACF) in the U.S. Department of Health and Human Services (HHS).

B.1 Respondent Universe and Sampling Methods

For the follow-up data collection the ISIS evaluation has two respondent universes—ISIS program staff and partners, and ISIS study participants.

B.1.1 ISIS Program Staff and Partners

The ISIS study recruited programs that have innovative career pathway programs in place and are able to implement random assignment tests of these programs. The nine sites included in ISIS are a purposive sample of the most promising career pathways interventions, rather than a nationally representative sample.

Program selection began with conversations between key stakeholders and the ISIS research team. Each program selected into ISIS satisfied criteria in three categories:

Programmatic criteria which fit the career pathways framework and include assessments, basic skills and occupational instruction , support-related services, and employment connections;
Technical criteria that emphasize the statistical requirements of the evaluation design, such as programs with the capacity both to serve a minimum of 500 participants and to recruit a minimum of 1,000 eligible applicants over a two-year enrollment period; and
Research capacity criteria that address the site’s ability to implement an experimental evaluation.

Additionally, ACF required that three of the programs be Health Profession Opportunity Grant (HPOG) recipients.¹

The nine programs selected all promote completion of certificates and degrees in occupations in high demand and, to this end, incorporate multiple steps on the career ladder, with college credit or articulation agreements available for completers of the lower rungs (see Appendix A for a depiction of the career pathways ladder and theory of change). While varying in specific strategies and target populations, they all provide some level of the core career pathways services (assessment, instruction, supports and employment connections) although the emphasized placed on each varies by program. Appendix B provides summaries of the nine ISIS programs.

For the implementation study, the research team will interview relevant site staff, including program managers, instructors, case managers, advisors, and organization leadership. The research team will also interview staff from key partners involved in the program (e.g., referral partners, instruction providers). The number of staff interviewed by the research team will vary according to site structure. Seven of the nine sites are single entities, one has three sub-sites and one has eight sub-sites. At single entity sites, the research team will select up to three leadership/managers/supervisors, up to five instructional staff, up to four case managers/advisors, and up to four partners. For the programs with sub-sites, the research team will interview management, instructors, case managers/advisors and partners at each sub-site. For each staff group, the team will interview two to three individuals.

The research team will supplement the interviews with case managers/advisors, instructors and managers/supervisors with an online survey. For the on-line surveys, the research team will attempt to survey the universe of case managers/advisors, instructors and managers/supervisors affiliated with the nine programs. The team estimates there will be an average of five case managers/advisors surveys per single entity site and five per sub-site in multiple entity sites. The expected universe of instructors is ten per single entity site and six per sub-site in multiple-entity sites. The expected universe of managers/supervisors is three per single entity site and two per sub-site in multiple entity sites.

This OMB package requests clearance for the implementation research guides to conduct these interviews and an online survey of case managers/advisors and instructors.

B.1.2 ISIS Programs and Study Participants

The ISIS program selection process spanned a period of over two years and included detailed assessments of more than 200 potential programs. After winnowing down the list of prospective programs based on a combination of factors, such as the intervention, its goals, the primary program components, program eligibility criteria, and the number of participants that enroll in the program annually, the ISIS team recruited nine promising career pathways programs into the study. Each program selected into ISIS satisfied criteria in three categories:

Programmatic criteria (e.g., assessment of skills, interests, service needs; innovative approaches to basic skills and occupational training; support-related services provided as part of the program and beyond; and employment-related services provided as part of the program and beyond);
Technical criteria that emphasize the statistical requirements of the evaluation design, including program size and the type of services available to the control group; and
Research capacity criteria that address the site’s ability to implement a random assignment study.

The nine programs are located in different parts of the country and thus have varying local labor markets, which has implications for data analysis. In terms of describing local variation in employment conditions and factors affecting these conditions, our analyses will be limited largely to interpretation of site-specific impacts, as we lack a sufficient number of sites to address the question of how economic conditions are related to variation in impacts across sites. The limited syntheses that is under consideration on a very exploratory basis will concern narrow questions of what factors may contribute to program impacts through moderation, mediation and moderated mediation. The ISIS team hopes that by making the analyses highly conditional, the lack of our ability to control on local economic conditions will be less troubling. Still, the ISIS team will collect labor information by site and will use it in the implementation study to describe the program context. A cross-program implementation analysis report will synthesize the collection of career pathways approaches and will, among other things, describe how environments (including the labor market) shaped the programs; however, although this report will describe differences across programs, it won’t be used to interpret impacts.

The universe of potential respondents is low-income adults (age 18 or older) who are interested in occupational skills training and who reside in the geographical areas where ISIS sites are located. The target enrollment for the study is an average of 1,000 individuals in eight of the nine sites, and 2,700 in the remaining site, for a total of 10,700 individuals in the study as a whole.

ISIS study participants will be the subjects for several data collection instruments for which this OMB package requests clearance (revised BIF, contact update form, follow-up survey, in-depth interview guide for a sample of study participants). The respondent universe for the supplemental BIF questions collected at baseline is the individuals who (1) apply to the ISIS program and agree to participate in the study and (2) have children. The maximum size of this universe is approximately 7,950 – that is, the full study sample minus those who have been randomly assigned as of the time the revised BIF is cleared by OMB. The research team estimates that of the maximum sample, 5,645 will have children and thus need to fill out the supplemental questions.² The respondent universe for the contact update form, the follow-up survey, and the in-depth interviews is the universe of study participants in both the treatment and control groups. The universe of potential in-depth interview participants is study participants in each site who are randomly assigned between the time of instrument clearance and the first scheduled interviews in late 2013.

Program staff recruits individuals, determines eligibility, and if the individual is eligible, obtains informed consent from those who volunteer to be in the study. The specific steps are as follows: Program staff informs eligible individuals about the study. Staff then administers the participant agreement form, which describes the study and requires individuals to sign the form if they wish to participate in the evaluation. Those who refuse to sign the consent form are not included in the study and are not eligible for the career pathways program. They will receive information about other services in the community. Appendix C contains two ISIS Participation Agreements: one for ISIS sites that are HPOG grantees and one for ISIS sites with no HPOG funds.

For those who consent, program staff collects baseline data, which includes the Basic Information Form (BIF) and Self-Administered Questionnaire (SAQ). OMB approved these forms under the previous request for clearance (OMB No. 0970-0397), although this submission seeks approval for a modified BIF that collects limited information about the study participants’ children in order to establish a child roster at baseline. Program staff enters information from the BIF into a web-based system developed specifically for the evaluation. Staff then use the system to conduct random assignment to the treatment or control group. Those assigned to the treatment group are offered the provided services while those assigned to the control group are not able to participate in the program but can access other services in the community. Exhibit B-1 summarizes the general process above.

Exhibit B-1. ISIS Study Participant Recruitment and Random Assignment Process

B.1.3 Target Response Rates

Overall, the research team expects response rates to be sufficiently high in this study to produce valid and reliable results that can be generalized to the universe of the study. The anticipated response rate for the baseline data collection is 100 percent, as completion of the BIF and SAQ is a requirement for enrollment. The expected rate for the 15-month follow-up survey is 80 percent, which is based on experiences in other studies with similar populations and follow-up intervals. For the in-depth interviews with study participants, we will oversample with the expectation of completing the desired number of interviews with treatment group members in each site.

The research team also expects high response rates for the implementation research-related data collection. Because program sites have agreed to participate in the evaluation, the research team anticipates a high response rate for the interviews with the program staff and for instructor and case manager/advisor online surveys. The response rate for other service providers in the community will likely depend on the level of connection to the study program. The team expects a high response rate for staff in organizations that are closely coordinated with the ISIS program (e.g., key referral partners, providers of program services), and a slightly lower rate for more tangential partners.

B.2 Procedures for Collection of Information

B.2.1 Sample Design

The target sample size for the ISIS study is 10,700 individuals. All but one of the nine study sites will each recruit about 1,000 individuals interested in career pathways services and who agree to participate in the study. The ninth site, the Year Up program, will recruit about 2,700 interested individuals. In eight sites, half of the sample members in each site will be assigned to the treatment group to receive the career pathways intervention and the other half will be assigned to a control group. In Year Up, the ratio will be two-thirds and one-third, respectively. Sample members assigned to the control group will have access to all other services provided in the community.

All randomly assigned individuals will be included in participant tracking and follow-up data collection. For the 15 month follow-up survey, the research team will attempt to contact and interview all members of the study sample (10,700 individuals). Therefore, no sampling is required for the tracking or the follow-up survey.

For the in-depth participant interviews, the research team will sample from those study participants who are randomly assigned following the clearance of the instrument. The goal is to interview individuals who are still in program services (treatment group) or other community services (control group) at the time of the first in-depth interview. Assuming clearance in summer 2013, the research team will use the universe of those individuals who were randomly assigned to start program services in the Fall 2013 term. In 7 of 9 sites, the research team seeks interviews with 10 treatment group members and 5 control group members. At least twice that number will be sampled. In the two multiple entity sites, the researchers will sample 10 treatment and 5 control group members in all three sub-sites for one program and in half (four of eight) in the other. .

ACF may exercise the option for additional follow-up data collection, which in addition to a survey of participants could include direct child assessments with one focal child per participant. The sampling method to choose a focal child will be described in the future OMB package that seeks clearance for the additional follow-up survey and direct child assessments.

Data to analyze the impacts of the career pathways interventions will come primarily from the follow-up survey, Unemployment Insurance and quarterly earnings records in the National Directory of New Hires, and college records from the National Student Clearinghouse. Key topics to be included in the follow-up survey are services received, credentials attained, employment outcomes not included in administrative data (such as wage rate, hours, and benefits), and other life circumstances.

B.2.2 Estimation Procedures

The research team will use a variety of estimation techniques. The primary analysis of treatment effects on 15-month outcomes will be intent-to-treat, i.e., they will estimate effects on those who are offered access to the program. Given the rich set of baseline data collected, regression analysis will be used to improve the precision of the estimates while preserving their unbiased character. The estimates of precision presented in the next section assume such regression adjustments, with precision gains based on those obtained in similar studies such as the National Job Training Partnership Act (JTPA) evaluation.³ Plans are also being developed to study sources of variation in outcomes as an exploratory study. Some of the estimation techniques employed for that will not rely on randomization for inference but rather rely on the completeness of baseline covariates to remove selection biases.

B.2.3 Degree of Accuracy Required

The baseline data collected will be used in the future in conjunction with follow-up survey data and administrative data to estimate impacts of career pathways interventions. The research team has estimated the minimum detectable impacts (MDIs). As shown in Exhibit B.2 below, the MDI is the smallest true impact that the study will have an 80 percent probability of detecting when the test for the hypothesis of “no impact” has just a 10 percent chance of finding an impact if the true impact is zero.

MDI estimates for two sample sizes are shown in Exhibit B.2—one for the typical ISIS site (treatment and control groups both 500) and one for Year Up (1,800 treatment, 900 control). MDIs are displayed for outcomes treated as confirmatory at the 15-month follow-up point (percent with substantial progress in career pathways training) and at the 36-month point and beyond (average annual earnings).

Exhibit B-2. Minimum Detectable Impacts for Confirmatory Hypotheses

Statistic	Percent with Substantial Educational Progress (Confirmatory @ 15 months)	Average Annual Earnings (Confirmatory @ 36 months)
MDI for Sample Sizes with
500 T: 500 C (most sites)	6.0	$1,384
1800 T: 900 C (Year Up)	3.9	$860
Control Group Mean	50.0	$11,452

Note: MDIs based on 80 percent power with a 10 percent significance level in a one-tailed test, assuming estimated in model where baseline variables explain 20 percent of the variance in the outcome.⁴ We have set the variance for credential attainment conservatively at 25% (p=q=50%). MDIs for earnings in both sample size categories are based on special variance tabulations using survey data for the second follow-up year of a small random assignment test of Year Up.^⁵ The table shows that the research team will be able to detect impacts on credentials as small as 6.0 percentage points in most sites and 3.9 percentage points in the Year Up site. The corresponding MDIs for annual earnings are $1,384 and $860, respectively.

The research team estimates these MDIs are sufficient to detect impacts likely to be policy relevant in each site. Recent evaluations with positive findings provide a range of estimates for impacts on pertinent post-secondary training outcomes and earnings. Estimates are available for two ISIS programs. A non-experimental analysis of I-BEST found impacts of greater than 20 percentage points on certificate/degree receipt but did not find statistically significant impacts on earnings (Zeidenberg et al. 2010). An experimental evaluation of Year Up (Roder & Elliot 2011) found positive second-year impacts on college attendance of six percentage points (insignificant) and on average annual earnings of $3,641 (significant).

Other recent experiments have tested related innovations operated by community colleges and community-based organizations. The community college experiments generally test narrower and typically shorter interventions than ISIS—approaches such as enhanced guidance and student supports, performance-based scholarships, and short-term learning communities aimed at developmental education students—and, as such, impact analyses to date focused on more incremental outcomes such as semester-to-semester persistence and credits earned. Statistically significant findings tended to be in the 5-10 percentage point range, and initial findings in this range have led to broader demonstrations.⁶ The Sectoral Employment Impact Study, an experiment testing short-term customized training by community-based organizations, found $4,011 average impacts on second-year earnings but did not analyze post-secondary training impacts (Maguire et al. 2010).

Looking further back, a 2001 meta-analysis of 31 government-sponsored voluntary training programs from the 1960s to 1990s (Greenberg et al., 2003) collected impacts on annual earnings and converted them into 1999 dollars. Adjusting their figures forward to 2012 with the CPI figure of 37 percent, the average intervention effect was $1,417 for women and just $318 for men (though estimates for men are substantially larger—$1,365—when restricted to random assignment studies).⁷

B.2.4 Who Will Collect the Information and How It Will Be Done

The modified BIF will be given to all eligible respondents to complete during the baseline data collection. Site staff are responsible for ensuring that the data is complete and entered into the study database. For study participants randomly assigned prior to OMB approval of the modified BIF, questions about their children will be included in the 15-month follow-up survey. The study team estimates about 2,750 individuals will have already been randomly assigned as of the expected clearance date. Of these, the study team estimates 1,562 will need to fill out the child roster as part of the 15-month follow-up survey.⁸

The follow-up survey will be administered 15 months following enrollment in the study and random assignment. The ISIS data collection team will contact study participants with a letter (see Appendix G) reminding them of their participation in the ISIS study and informing them that they will soon receive a call from an ISIS interviewer who will want to interview them over the telephone. The letter will remind the sample member that their participation is voluntary and that they will receive $35 upon completion of the interview. Centralized interviewers using computer-assisted telephone interview (CATI) software will conduct the follow-up survey. Interviewers will be trained on the study protocols and their performance will be regularly monitored. The interviewers will first try to reach the sample member by calling the specified contact numbers to administer the 50-minute follow-up survey. For sample members who cannot be reached at the original phone number provided, interviewers will attempt to locate new telephone numbers for the sample members by calling the secondary contacts provided for this purpose. Once the centralized interviewers have exhausted all leads, the case will be transferred to the field interviewers to locate the sample member in-person. When field staff succeed in finding a sample member and convinces them to answer the survey, the field staff will establish contact with a centralized interviewer on a company cell phone. The centralized interviewer will then conduct the interview while the field interviewer waits discretely nearby. With this approach, we hope to minimize mode effects and training requirements for field staff. The research team will attempt to interview all sample members within six months of their release date (15 months following random assignment).

The research team will conduct the implementation research interviews with site staff and local stakeholders during the visits to each program using the interview guides submitted in this clearance package. Two-person teams will conduct the visits to each program. Each team will be led by a senior researcher, joined by a mid-level researcher, all of whom have experience in conducting site visits to educational and employment programs. Also, during the site visit, names and email addresses of case managers/advisors, instructors, and managers/supervisors affiliated with the program will be collected for the on-line surveys. The team expects to survey the universe of case managers/advisors and manager/supervisors in each program as well as the universe of instructors for those programs that have in-house or affiliated trainers.⁹ An email sent to the case managers/advisors, instructors and managers/supervisors will explain the nature of the survey (see Appendix J).

A different team of researchers (one senior and one junior) will conduct the in-depth interviews with a small sample of study participants in each site at two points in time. The first interview will occur while the individuals are expected to still be receiving services (either from the study program or from other community providers), generally within four months of random assignment. The second interview will occur at least one year following random assignment. The number of interviews per site depends on the site structure. In the 7 “single entity” programs (i.e., no sub-sites), the team will interview 10 treatment group and 5 control group members.) In the programs with sub-sites, the team will interview 10 treatment group and 5 control group members in sub-sites.¹⁰ These sample members will be randomly selected and they will receive $40 for their time and expenses to get to the interview site.

Copies of the proposed instruments can be found in the Appendices.

B.2.5 Procedures with Special Populations

The follow-up survey instrument will be available in English and Spanish. Interviewers will be available to conduct the interview in either language. Persons who speak neither English nor Spanish, deaf persons, and persons on extended overseas assignment or travel will be ineligible for follow-up, but information will be collected on reasons for ineligibility. Also, tracking will continue in case they become eligible for future follow-up activities. Persons who are incarcerated or institutionalized will be eligible for follow-up only if the institution authorizes the contact with the individual.

B.3 Methods to Maximize Response Rates and Deal with Non-response

The goal will be to administer the follow-up survey to all study participants in each site, reaching a target response rate of at least 80 percent. To achieve this response rate, the ISIS team developed a comprehensive plan to minimize sample attrition and maximize response rates. This plan involves regular tracking and locating of all study participants, providing tokens of appreciation, and sample control during the data collection period.

B.3.1 Participant Tracking and Locating

The ISIS team will develop a comprehensive participant tracking system, in order to maximize response to the ISIS follow-up surveys. This multi-stage locating strategy blends active locating efforts (which involve direct participant contact) with passive locating efforts (which rely on various consumer database searches). At each point of contact with a participant (through tracking letters and at the end of the survey), the research team will collect updated name, address, telephone and email information. In addition, the research team will also collect contact data for up to three people that do not live with the participant, but will likely know how to reach him or her. Interviewers only use secondary contact data if the primary contact information proves to be invalid—for example, if they encounter a disconnected telephone number or a returned letter marked as undeliverable. Appendix F shows a copy of the tracking letter.

In addition to the direct contact with participants, the research team will conduct several database searches to obtain additional contact information. Passive tracking resources are comparatively inexpensive and generally available, although some sources require special arrangements for access.

B.3.2 Tokens of Appreciation

Offering appropriate monetary gifts to study participants in appreciation for their time can help ensure a high response rate, which is necessary to ensure unbiased impact estimates. Study participants will be provided $35 after completing the first follow-up survey. As noted above, in addition to the survey, every four months the participants will receive a tracking letter with a contact update form, which lists the contact information they had previously provided. The letter will ask them to update this contact information by calling a toll-free number or returning the contact update form in the enclosed postage-free business reply envelope. Study participants who update their contact information will receive $5 in appreciation for their time. The small sample of study participants selected to participate in the in-depth interviews being conducted in each site will receive $40 for their time and expenses to get to the site.

B.3.3 Sample Control during the Data Collection Period

During the data collection period, the research team will minimize non-response levels and the risk of non-response bias in the following ways:

Using trained interviewers who are skilled at working with low-income adults and skilled in maintaining rapport with respondents, to minimize the number of break-offs and incidence of non-response bias.
Using a tracking letter and contact update form to keep the sample member engaged in the study and to enable the research team to locate them for the follow-up data collection activities. (See Appendix F for a copy of the tracking letter.)
Using an advance letter that clearly conveys the purpose of the survey to study participants, the incentive structure, and reassurances about privacy, so they will perceive that cooperating is worthwhile. (See Appendix G for a copy of the advance letter.)
Providing a toll-free study hotline number, which will be included in all communications to study participants, for them to use to ask questions about the survey, to update their contact information, and to indicate a preferred time to be called for the survey.
Taking additional tracking and locating steps, as needed, when the research team does not find sample members at the phone numbers or addresses previously collected.
Requiring the survey supervisors to manage the sample to ensure that a relatively equal response rate for treatment and control groups in each ISIS site is achieved.

Through these methods, the research team anticipates being able to achieve the targeted 80 percent response rate for the follow-up survey.

To maximize response rates for the implementation research, the study team will work with each program to schedule the interviews at a time when the necessary program and partner staff will be available. An agenda will be created, and all respondents will be notified of the purpose and timing of their interviews. If a staff member is not available during the site visit, a telephone interview will be conducted. They will also offer respondents $40 to ensure a high response rate for the in-depth interviews with study participants. In both cases, the amount provided is used to thank the respondent for their time, and in the case of the in-depth interview participants, help pay for any expenses incurred in getting to the interview site.

B.4 Tests of Procedures or Methods to be Undertaken

In designing the follow-up survey, the research team included items used successfully in previous studies or in national surveys. Consequently, many of the survey questions have been thoroughly tested on large samples. The follow-up survey instrument will be pretested with nine adults in occupational programs at local community or technical colleges that are not part of the ISIS study. Experienced interviewers will be used to conduct the pretest, and a debriefing will be held with them to discuss their perceptions of the clarity and flow of survey items, ease of completion, and time requirements. After pretesting, the questionnaire will be revised based on the feedback and will be trimmed if necessary to stay within a 50-minute average administration time, including time to update contact information for possible future follow-up activities. Changes made to the instrument will be submitted to OMB for review.

B.5 Individuals Consulted on Statistical Aspects of the Design

The individuals shown in Exhibit B-5 assisted ACF in the statistical design of the evaluation.

Exhibit B-5. Individuals Consulted on the Study Design

Name	Role in Study
Karen Gardiner Abt Associates Inc.	Project Director
Dr. Howard Rolston Abt Associates Inc.	Principal Investigator
Dr. David Fein Abt Associates Inc.	Principal Investigator
David Judkins Abt Associates Inc.	Principal Statistician

Inquiries regarding the statistical aspects of the study’s planned analysis should be directed to:

Karen Gardiner ISIS Project Director

Dr. David Fein ISIS Principal Investigator

Brendan Kelly Office of Planning, Research & Evaluation

Administration of Children and Families, US DHHS

Emily Schmitt Office of Planning, Research & Evaluation

Administration of Children and Families, US DHHS

References

Creighton, K., King, K., and Martin, E. (2007). The use of monetary incentives in Census Bureau longitudinal surveys. Washington, DC: U.S. Census Bureau.

Duffer, A. et al., (1994). Effects of incentive payments on response rates and field costs in a pretest of a national CAPI survey. Chapel Hill: Research Triangle Institute

Educational Testing Service (1991). National Adult Literacy Survey addendum to clearance package, volume II: Analyses of the NALS field test, pp. 2-3.

Greenberg, D., Michalopoulos, C., and Robins, P. (2003). A Meta-Analysis of Government-Sponsored Training Programs. Industrial and Labor Relations Review.

Kennet, J., and Gfroerer, J. (Eds.). (2005). Evaluating and improving methods used inthe National Survey on Drug Use and Health (DHHS Publication No. SMA 05-4044,

Methodology Series M-5). Rockville, MD: Substance Abuse and Mental Health

Services Administration, Office of Applied Studies.

Maguire, S., Freely, J., Clymer, C., Conway, M., and Schwartz, D. (2010). Tuning In to Local Labor Markets: Findings From the Sectoral Employment Impact Study. Philadelphia, PA: Public/Private Ventures.

Orr, L.L., Bloom, H.S., Bell, S.H., Lin, W., Cave, G., and Doolittle, F. (1996). Does Job Training for the Disadvantaged Work? Evidence from the National JTPA Study. Washington, DC: Urban Institute Press.

Roder, A., and Elliott, M. (2011). A Promising Start: Year Up’s Initial Impacts on Low-Income Young Adults’ Careers. New York, NY: Economic Mobility Corporation.

Zeidenberg, M., Cho, S., and Jenkins, D. (2010). Washington State's Integrated Basic Education and Skills Training Program (I-BEST): New Evidence of Effectiveness (CCRC Working Paper No. 20). New York: Community College Research Center, Teachers College, Columbia University.

1 The Health Profession Opportunity Grants (HPOG) program provides education and training to Temporary Assistance to Needy Families recipients and other low-income individuals for occupations in the health care field that pay well and are expected to either experience labor shortages or be in high demand. The HPOG program is administered by the Office of Family Assistance within ACF. In FY 2010, $67 million in grant awards were made to 32 entities located across 23 states, including four tribal colleges and one tribal organization. These demonstration projects are intended to address two challenges: the increasing shortfall in supply of healthcare professionals in the face of expanding demand; and the increasing requirement for a post-secondary education to secure a well-paying job. Grant funds may be used for training and education as well as supportive services such as case management, child care, and transportation. http://www.acf.hhs.gov/programs/opre/welfare_employ/evaluation_hpog/overview.html

2 This is based on trends to date that suggest approximately 71% of ISIS study participants have one or more children.

3 Orr, L.L., Bloom, H.S., Bell, S.H., Lin, W., Cave, G., and Doolittle, F. (1996). Does Job Training for the Disadvantaged Work? Evidence from the National JTPA Study. Washington, DC: Urban Institute Press.

4 Regression models for overall impacts on earnings in the recent Sectoral Employment Impact Study explained 19 percent of the variation (see http://www.ppv.org/ppv/publications/assets/325_publication.pdf, p. 77). An older study of general low-income adult populations, the Job Training Partnership Act evaluation generated similar statistics: 20 percent of the variation for annual earnings for adult females and 30 percent of the variation for annual earnings for adult males was explained by the regression models, calculated using the public use data available for the study (see The National JTPA Study: Title II-A Impacts on Earnings and Employment at 18 Months. U.S. Department of Labor, Research and Evaluation Report Series 93-C, 1993).

5 These earnings variance estimates (standard deviations of $12,748 and $10,160 for treatment and control groups, respectively) were the only available for populations actually served in ISIS sites. Though based on a small survey sample (120 treatment, 44 control), they are very close to estimates P/PV provided ISIS for participants in their Sectoral Employment Impact Study, which involved a wider age range than the youth (18-24) targeted in Year Up.

6 Cites for Kingsborough learning community (5-6% on enrollment, 3^rd post-program semester); Louisiana performance scholarship (10% on enrollment in 2^nd year following program); Ohio enhanced guidance (7%, 4% on enrollment in the 2^nd semester of intervention and 1^st post-program); Chafee boosts % w/GPA>2.0 by 12% (all credits) and 7% (degree applicable credits). Both the learning community and performance based scholarship tests led to subsequent multi-site demonstrations.

7 Authors’ calculations based on estimated mean, fraction experimental, coefficient on experimental dummy in Greenberg et al. (2003, Tables 3 & 5).

8 Baseline data collected to date suggests that 71% of ISIS participants have one or more children. Assuming an 80% response rate for the 15-month follow-up survey (2,200 participants), it is expected that 1,562 (or 71%) will need to fill out the child roster.

9 The exception is two programs that use an Individual Training Account model in which program participants can attend any institution in a large geographic area.

10 One program has 3 sub-sites; the research team will interview study participants in all sub-sites. A second program has 8 sub-sites; for cost reasons, the research team will interview study participants in 4 of the 8 sub-sites.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Missy Robinson
File Modified	0000-00-00
File Created	2021-01-29