OMB HtE KS-MO_36M Supp Statement- Part A FINAL

Contract No.: HHS 233-01-0012

Contract Amount: $23.78 million

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART A

DHHS/ACF/ASPE/DOL

ENHANCED SERVICES FOR THE HARD-TO-EMPLOY (HtE)

DEMONSTRATION AND EVALUATION PROJECT

KANSAS-MISSOURI 36-MONTH DATA COLLECTION INSTRUMENTS

June 21, 2007

Prepared for:
U.S. Department of Health and Human Services
	Office of Assistant Secretary for Planning
Administration for Children and Families	and Evaluation
370 L’Enfant Promenade, SW	200 Independence Avenue, SW
Washington, DC 20447	Washington, DC 20201
Phone: 202-401-5070	Phone: 202-260-0384
Project Officer: Girley A. Wright	Project Officer: Flavio Menasce
U.S. Department of Labor	Prepared by:
Employment and Training Administration	MDRC
200 Constitution Avenue, NW	16 East 34th Street, 19th Floor
Washington, DC 20210	New York, NY 10016
Phone: 202-693-3654	Phone: 212-532-3200
Project Officer: Roxie Nicholson	Project Directors: David Butler/Barbara Goldman

TABLE OF CONTENTS

A. JUSTIFICATION 1

A1. Circumstances Necessitating Data Collection 1

A1.1 Background 1

A1.2 Description of the Kansas and Missouri Sites in the Hard-to-Employ Evaluation 3

A1.3 Research Contribution of the 36-month Parent-Reported Survey and Direct Child Assessments 5

A2. How, by Whom, and for What Purpose Are Data to be Used 6

A3. Use of Information Technology for Data Collection to Reduce Respondent Burden 12

A4. Efforts to Identify Duplication 12

A4.1 Reasons Why Available Information Cannot Be Used 12

A5. Burden on Small Business 12

A6. Consequences if Data Collection is not Conducted 12

A7. Special Data Collection Circumstances 13

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission 13

A9. Justification for Respondent Payments 14

A9.1 The Use of Incentives 15

A10. Confidentiality

A10.1 Consent 17

A10.2 Protections for Individuals’ Confidentiality 17

A11. Questions of a Sensitive Nature 18

A12. Estimates of the Hour Burden of Data Collection to Respondents 18

A13. Estimates of Capital, Operating, and Start-Up Costs to Respondents 19

A14. Estimates of Costs to Federal Government 19

A15. Changes in Burden 19

A16. Tabulation, Analysis, and Publication Plans and Schedule

A16.1a Assessment of Data Quality and File Construction 19

A16.1b Analysis Plans 20

A16.2 Publication Plans and Schedule 23

A17. Reasons for Not Displaying the OMB Approval Expiration Date 24

A18. Exceptions to Certification Statement 24

LIST OF APPENDICES

A.1: Parent Consent Form

A.2: Consent Form for Videotaping Direct Child Assessments

B: 36-Month Follow-Up Parent-Reported Survey

C: 36-Month Follow-Up Protocol for Conducting Direct Child Assessments

D.1: Federal Register Published 60-Day Notice

D.2: Federal Register Draft 30-Day Notice

E: Statute/Regulation Authorizing Evaluation and Data Collection: Social Security Act, Section 1110

F: References

A. Justification

The Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project (HtE) seeks to learn what services improve the employment prospects of low-income persons who face serious obstacles to steady work. The project is sponsored by the Office of Planning, Research and Evaluation (OPRE) of the Administration for Children and Families (ACF), the Office of the Assistant Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services (HHS), and the U.S. Department of Labor (DOL).

The HtE project is a multi-year, multi-site evaluation that employs an experimental longitudinal research design to test four strategies aimed at promoting employment among hard-to-employ populations. The four include: 1) intensive care management and job services program for Rhode Island Medicaid recipients with serious depression; 2) job readiness training, worksite placements, job coaching, job development and other training opportunities for recent parolees in New York City; 3) pre-employment services and transitional employment for long-term participants receiving Temporary Assistance to Needy Families (TANF); and 4) two-generational Early Head Start (EHS) services providing enhanced self-sufficiency services for parents, parent skills training, and high quality child care for children in low-income families in Kansas and Missouri.

This document requests OMB approval for the 36-month data collection activities in the Kansas and Missouri sites of the HtE evaluation. These data collection elements consist of a parent-report survey and direct child assessments and are intended to expand our understanding of the longer-term effects of a two-generational Early Head Start program with enhanced employment and self-sufficiency services on hard-to-employ parents and their children. This wave of data collection builds upon the 15-month follow-up effort, for which the data collection instruments were previously approved by OMB (OMB Control Number: 0970-0276).

A1. Circumstances Necessitating Data Collection

This section provides a brief summary of the literature discussing the interplay between hard-to-employ parents’ barriers to employment and economic self-sufficiency and young children’s developmental risks in low-income families. We then provide a short description of the Hard-to-Employ Evaluation in Kansas and Missouri and discuss key components of the evaluation, sources of data, and constructs of interest. This section concludes by highlighting the research contribution of the 36-month data collection effort and instruments, namely – the 36-month parent-reported survey and direct child assessments.

A1.1. Background

A wealth of research indicates that the rate of child poverty remains high (e.g., almost 20% of children under 6 lived in poverty in 2003¹) and that children living in poverty have worse health, behavioral, cognitive, and academic outcomes than their more affluent counterparts.² Many of the factors that place low-income children at developmental risk are indeed the very characteristics that interfere with their parents’ employability and ability to achieve economic self-sufficiency. For example, maternal depression – a notable barrier to employment – is linked with children’s mental health, social and behavior problems, as well as difficulties in school, in part because such mothers tend to use unresponsive and/or harsh parenting styles.³ Parents’ employment and economic circumstances can place children at developmental risk and some children’s developmental difficulties can impede parents’ ability to find and sustain employment. For instance, children’s physical disabilities or behavior problems may make it difficult for parents to find adequate and stable child care in order to go to work.⁴ This bi-directional interplay between low-income children’s developmental difficulties and hard-to-employ parents’ employment and economic circumstances is gaining recognition in the literature.

Numerous programs and policy initiatives have sought to address the needs of hard-to-employ parents and their children.

Evidence from research on two-generational services. A number of programs and policy initiatives have sought to address the needs of hard-to-employ parents and their children. Earlier research of such programs highlights the value of two-generational services for meeting the developmental needs of low-income children and their parents.⁵ The Early Head Start Research and Evaluation Project found that EHS improved both parenting behaviors and children’s cognitive development.⁶ This evaluation identified a combination of home-based and child care-related services as one of the most effective strategies for enhancing young children’s cognitive and social outcomes. Similarly, a review of early childhood programs highlights the advantages of two-generational approaches on child development. This review suggests that home-based interventions might improve family factors, such as parenting, while center-based interventions might improve children’s behavioral and cognitive development.⁷ These findings suggest that combining home- and center-based services might be a powerful approach affecting the broadest range of outcomes. Yet, despite evidence of the positive effects for children, the EHS evaluation found quite small impacts on parents’ employment, suggesting that the effects of a two-generational approach might be enhanced by a more proactive programmatic focus on parental employment and economic self-sufficiency needs.

Implications. Taken together, the prior research suggests that a two-generational approach, particularly if the program focuses on parents’ employment and economic self-sufficiency, could have wider-ranging effects than a program focused on either parents or children, but not both.⁸ Directly addressing young children’s developmental needs can help parents overcome obstacles to sustained employment and economic self-sufficiency. Likewise, directly addressing parents’ employment and economic needs can improve their ability to better their own financial circumstances while indirectly benefiting their children, as earlier research suggests that living in poverty places children at developmental risk.⁹

No study to date has examined the impacts of a two-generational program, which explicitly focuses on parents and children’s needs, on a large-scale using a random assignment research design, even though the literature leads us to expect that child services coupled with enhanced services targeting parents’ employment and self-sufficiency needs might have promising effects on parents and children, particularly in low-income families. For these reasons, the Hard-to-Employ evaluation includes an experimental test of EHS programs with programmatic enhancements to existing employment and self-sufficiency services.

A1.2. Description of the Kansas and Missouri Sites in the Hard-to-Employ Evaluation

The Kansas and Missouri sites of the Hard-to-Employ Evaluation include a test of a two-generational program that dually addresses the needs of low-income children and their parents who are at risk for unemployment. The model of EHS being evaluated consists of all the early developmental childhood and parent education services that are associated with EHS (e.g., high quality child care, intensive home visits, parent education classes) with an expanded self-sufficiency component aimed at assisting parents of participating children work towards their education, employment and self-sufficiency goals.

The target population for EHS is prenatal women and children 3 years old and younger.¹⁰ To qualify for EHS services, pregnant women and families with infants or toddlers must reside within the boundaries of an EHS program’s designated service area, and families must meet EHS income eligibility requirements by having a family income that is at or below the federal poverty threshold.¹¹

Two participating programs,¹² Southeastern Kansas Community Action Program, Inc. in Girard, Kansas and Youth In Need in St. Charles, Missouri, were selected based upon their established histories of delivering high-quality EHS services; the use of a mixed-approach services model (a combination of services that the National Evaluation points to as being most effective for enhancing young children’s developmental outcomes);¹³ their capacities to build sufficient waiting lists to sustain and justify random assignment; and, the support of the EHS policy councils regarding the use of random assignment and programmatic enhancements to existing EHS services.

This evaluation is a multi-year evaluation consisting of several components. The three main components of the evaluation are:

A process and implementation analysis focusing on program operations and challenges encountered. The goals of this analysis are to: 1) describe how the EHS program operates and how programmatic enhancements to employment and self-sufficiency services have been put into place; 2) generate data that will help explain program impacts and costs; 3) provide feedback to HHS and to the sites on program performance; and, 4) assess the feasibility and replicability of the program model. The data sources include administrative records, observations of program activities, field research (on-site observations of program activities and formal interviews and discussions with program administrators, line staff, and other informants), case file reviews, focus groups with program staff and parents, and parent-reported surveys.

An impact analysis examining net effects of EHS with enhanced employment and self-sufficiency services on parental employment, educational and economic outcomes, parental psychological well-being and health, family functioning and routines, parenting, father involvement, as well as children’s emotional and behavioral adjustment, cognitive and language developmental outcomes. This analysis will compare outcomes for parents and children in the experimental group (i.e., those randomly assigned to receive EHS) with their counterparts in the control group using data from administrative records and parent surveys and direct child assessments collected 15 months after random assignment, as well as parent surveys and direct child assessments, which are being proposed for the 36-month data collection effort.

Data for the impact analysis are collected on the following key constructs:

• Baseline demographic and descriptive data. Baseline demographic information for the sample is drawn from common information across all of the programs’ in-take forms and assessments that are completed as part of the application process. The assessments generally have two components. The first is the program eligibility determination and priority score assignment. The second is an in-depth interview with parents covering certain aspects of family life.

• Parental employment. Data on parental employment is collected from several sources. MDRC is currently obtaining Unemployment Insurance (UI) quarterly data from the Kansas and Missouri State Departments of Labor. This data shows quarterly employment in UI-covered jobs held in Kansas or Missouri for each sample member. Administrative data records will be supplemented with survey information collected 15 months after random assignment. The survey data that is being proposed for the 36-month data collection effort will also include parental employment measures. Finally, MDRC is currently looking to access wage data from the National Directory of New Hires. This is a national database maintained by the Office of Child Support Enforcement, and therefore would provide information on earnings from employment both within and outside of Kansas and Missouri.

• Income, earnings, and public assistance receipt. Data from state administrative records track parents’ income, earnings, and public assistance receipt in Kansas and Missouri for each sample member. These data are maintained by the Kansas and Missouri Departments of Human Services. This information is being proposed for the 36-month data collection effort and will be supplemented with survey information, which was collected 15 months after random assignment on parental income, earnings and public assistance receipt.

• Parental psychological well-being, parenting, family functioning, and child care use. Key aspects of parental psychological well-being, parenting, family functioning, such as activities with children (e.g., play and discipline) and family routines, and child care use that might account for the effects of EHS on young children’s development will be assessed using survey information collected 15 months after random assignment. MDRC is also proposing that this information be collected on the parent-reported survey collected 36 months after random assignment.

• Children’s developmental outcomes. Children’s well-being will be measured by direct child assessments and survey data collected 15 months post-random assignment. The 15-month survey will be administered to children’s primary caregivers and includes measures of children’s social/emotional, cognitive development, academic achievement, and health and safety outcomes. The direct child assessments and parent-reported measures of child outcomes are also being proposed for the 36-month data collection effort.

• Child welfare involvement. MDRC is currently looking into the availability of this data from the Kansas and Missouri child welfare administrative records. These data provide information about sample members’ referrals to and involvement with state child welfare systems, including substantiated and unsubstantiated cases of child abuse and neglect.

• Program and services participation data. MDRC will be obtaining administrative participation records from the EHS programs. These data provide information on each family’s participation in EHS, such as number and frequency of home visits and attendance at parent training workshops. In addition, the 15-month survey is designed to tap program and control group members’ services receipt. The proposed 36-month parent-reported survey will also include measures of services receipt.

A cost-benefit study that compares the financial costs and benefits of the program to examine whether any positive impacts achieved by the program are commensurate with program costs. Some of the data sources will be program fiscal records, time studies, automated records on child care, transportation and other support services, the 15-month parent-reported survey, and the proposed 36-month parent-reported survey.

Timeline for the current evaluation. Random assignment was conducted from July 2004 to December 2006. The process and implementation analysis is on-going. The summary of the findings from the implementation analysis will be included in the final report for the evaluation. The 15-month follow-up data collection effort is currently being fielded and is expected to be in the field until mid-2008. The 36-month follow-up data collection activities are scheduled to begin in late 2007 and will be fielded until 2009. Preliminary results from the impact analysis are expected in 2007 and the final report is expected in 2010.

A1.3. Research Contribution of the 36-Month Parent-Reported Survey and Direct Child Assessments.

The purpose of this document is to request OMB approval of the data collection instruments for the 36-month follow-up. For this wave of data collection, we propose collecting a parent-reported survey and direct child assessments. Data collected with these instruments will allow the study to address the following key research questions:

• What are the longer-term effects of a two-generational program with enhanced employment and self-sufficiency services on young children and their parents across a variety of outcomes, including children’s early literacy and math skills, school readiness, and social, emotional and behavioral adjustment, as well as parental psychological well-being and health, family functioning, parenting, parental involvement, and parental employment and economic outcomes?

• To what extent does a two-generational program with enhanced employment and self-sufficiency services affect children’s intra-individual growth, parental employment, earnings and income trajectories, and changes in child care use, family functioning and parenting over time?

• What service receipt differential is experienced by program and control groups over time as a function of program participation?

From a policy perspective, understanding the extent to which a two-generational program like the enhanced EHS program influences parents and children has important implications for early childhood education programs aimed at addressing young children’s developmental needs. To the extent that we find positive impacts on parental employment and economic outcomes and children’s developmental outcomes, the findings from the HtE evaluation may argue in favor of strengthening linkages between child-focused services and other agencies, such as welfare agencies and employment and job training services, aimed at addressing hard-to-employ parents’ education, employment and self-sufficiency needs.

From a developmental science perspective, the HtE evaluation has the potential to make a significant contribution to our understanding of children’s developmental processes from a dynamic systems perspective. The experimental data gathered as part of the HtE evaluation can help us understand how experimentally-induced changes in parental employment, education and self-sufficiency are linked statistically with changes in child development and well-being.¹⁴ Similarly, we will be able to examine how experimentally-induced, or program-driven changes, in children’s well-being affect parents’ ability to sustain employment and improve their financial circumstances, helping us to understand more deeply the bi-directional and transactional processes that influence parent and child well-being. Such processes have rarely been tested in prior developmental research.

A2. How and by Whom, and for What Purpose Are Data to be Used

We plan to collect a parent-reported survey and direct child assessment 36 months post-random assignment from all sample members in Kansas and Missouri. This includes children assigned to the program and control groups and their primary caregivers or parents. Children at the 36-month follow-up will range in ages from 2.5 and 6.75 years old. For the most part, the survey respondents will be mothers, but in some cases, fathers will respond to the survey. The data collected at 36 months post-random assignment will be linked with other sources of data already approved for collection (e.g., administrative data, 15-month parent-reported survey and direct child assessments, and EHS program participation data).

The 36-month parent-reported survey will include measures about family life, parental outcomes and child well-being to understand how enhanced EHS services affect participating families. Direct child assessments at 36 months will provide a snapshot of children’s early literacy and math skills, as well as emotional and behavioral adjustment. Together, the survey and direct child assessments will assess key outcomes and will be complemented with data gathered through administrative records on outcomes, like employment, earnings, receipt of welfare benefits, and involvement with the child welfare system. The direct child assessments and survey data will provide important sources of information, since much of the information on children’s developmental outcomes and family functioning cannot be assessed through school records for the age range of children in the sample, administrative records, or other secondary data sources.

36-month parent-reported survey

The follow-up survey will be administered to primary caregivers of children in the full study sample and will be administered by interviewers in the home or over the phone. The survey will likely be conducted using a mixed mode methodology that may consist of a combination of computer-assisted telephone interviews (CATI), computer-assisted in-person interviews (CAPI), and/or paper and pencil interviews (PAPI). The approximate administration time for this survey is 45 minutes. However, the survey has yet to be formally pre-tested. Should the administration time be longer than anticipated, we will delete content as necessary in order to stay within our budget; our current budget requires that the survey be administered in 45 minutes or less. We base our calculations of respondent burden in Section A12 on a 45-minute survey for the 36-month follow-up data collection effort.

Because both children and parents are targeted in two-generational interventions, we believe that it is important to collect information about children, as well as parents. As such, the 36-month parent-reported survey will consist of questions in the following areas:

Parental employment outcomes. The survey aims to capture a variety of work conditions, such as wages,¹⁵ benefits (i.e., the availability of flextime, sick leave, vacation time, and health benefits),¹⁶ work hours,¹⁷ work schedules,¹⁸ and occupational complexity¹⁹ that are not readily available through secondary data sources. These employment conditions have been shown in the non-experimental literature to be linked with children’s cognitive, social and emotional development, in part through changes in family income, parental stress, family routines and functioning, and parenting practices.²⁰

The developmental literature also suggests that job instability (transitions from jobs to unemployment), and periods of unemployment are linked with less positive child outcomes, whereas job mobility, which is generally characterized by job transitions accompanied by earnings growth, tend to be associated with more positive child outcomes.²¹ Accordingly, the 36-month survey includes questions tapping parents’ employment history since the last interview at the 15-month follow-up to understand how enhanced EHS services affect parental job stability, mobility and loss over the follow-up period. These measures are intended to tap parental formal and informal employment. The self-reported parental employment histories will be used in conjunction with administrative data to understand the impacts of enhanced EHS services on longitudinal parental employment patterns over time. In the case of two-parent families, the respondent will report on both parents’ employment outcomes, since it could be that the program affects both the primary and secondary parents’ employment.

Persistence of barriers to employment. Barriers to employment, such as parental depression, substance abuse, poor parental physical health, and educational attainment, have been shown to impede parents’ abilities to sustain employment and achieve economic self-sufficiency, and are linked with children’s mental health, social and behavioral problems, as well as difficulties in school.²²

Total income and financial and material hardship. We propose asking parents to report their total income on the survey in order to capture formal and informal sources of income, as prior research suggests that many low-wage workers rely upon off-the-books jobs, side-work, or cash contributions from family members or friends to supplement their income. The survey data on income will be used to supplement administrative records of parental income, welfare receipt and earnings over the follow-up period. To the extent that EHS with enhanced employment and self-sufficiency services influence rates of employment among participating parents, the program could also have important impacts on parents’ economic circumstances, and, in turn, children’s outcomes. Prior research suggests that increases in income are linked with more favorable child outcomes.²³ At the same time, even though the program may increase parental work efforts, such changes may not necessarily lead to improved financial circumstances, particularly when parents work in low-wage labor markets.²⁴ Some studies have shown that income effects on children are due in part to whether or not parents perceive that income as being sufficient for meeting their needs.²⁵ Thus, we propose including measures of parental financial and material hardship as well in order to understand how the enhanced EHS services might affect parental psychological well-being and children’s developmental outcomes.

Child care use. Enhanced EHS services might affect children through improvements in children’s non-parental care environments. High quality and stable child care during infant and toddler years, in turn, has been associated with better cognitive functioning and language development.²⁶

Participation in services. Survey measures of program participation and receipt of services, such as job training, child care assistance, housing assistance, educational services, prenatal care, early educational services will be used to assess the service differential between program and control group families. The survey is the only source of information about control group members’ receipt of services, and will serve as a key source of information for the impact and implementation analyses.

Subsequent pregnancies and child bearing. Research has shown that delaying subsequent pregnancies and childbearing can have implications for parental educational, employment and economic outcomes, as well as children’s cognitive development and well-being. According to human capital theories of development,²⁷ for instance, resources available for each child in the family necessarily become diluted with increasing family size and number of children in the family. At the same time, rapid subsequent childbearing, particularly during adolescence, has been linked with maternal life course outcomes, such as less favorable economic well-being and lower educational attainment, and compromised child well-being.²⁸

Residential instability and family and household structure. Prior research suggests that residential instability in children’s lives places them at risk for less optimal developmental outcomes.²⁹ The 36-month survey includes measures of residential instability and changes in family and household structure because these factors may be influenced by enhanced EHS services; families who are participating in enhanced EHS services may be able to achieve more stable financial circumstances, and in turn more stable housing and family arrangements than those in the control group.

Parent-child relationships and parenting practices. EHS was designed as an intervention to promote supportive parent-child relationships and positive parenting practices. The quality of parent-child interactions, in turn, has been shown in numerous non-experimental studies to be an important predictor of children’s cognitive, language, social and emotional competencies.³⁰ Yet, because the model of EHS being tested includes an enhanced programmatic focus on addressing parental employment and educational needs, it is possible that program-driven increases in parental employment may reduce the amount of time that parents have to engage in activities and have warm, cognitively stimulating and nurturing interactions with their children.³¹ Moreover, particularly in low-income families with fewer resources and supports, parents may struggle to balance the demands of work and family.³² As stress builds, parents may be less likely to engage in nurturing interactions with their children.³³ Thus, we propose including measures of parenting stress in addition to parenting practices and parent-child relationships.

Family functioning and routines. A goal of EHS services is to help parents establish regular routines and supportive home environments to foster young children’s development. Therefore, we propose including measures of family functioning and routines in the survey. However, a growing body of research suggests that many parents – particularly, single parents in low-income and hard-to-employ families with few financial resources and supports, may encounter difficulties in negotiating the balance between work and family routines.³⁴ Disruptions to family routines have also been linked with unfavorable child outcomes, such as lower school achievement and externalizing behavior problems.³⁵ These disruptions may also be exacerbated by unstable parental work patterns or irregular work schedules, which are common in low-wage labor markets.³⁶

Father involvement. Father involvement has been a relatively neglected area of research with regard to infant and toddler development. However, understanding how enhanced EHS services affect father involvement and parenting practices for resident and non-resident fathers may be important for describing intervening processes linking program impacts on parental employment and child outcomes. To the extent that the enhanced EHS services increase maternal work efforts, fathers may be called upon to fulfill a more prominent role in their children’s lives. Thus, we propose collecting measures of father involvement in the 36-month survey.

Children’s academic, social, emotional, behavioral and health outcomes. Based on prior research, the child-focused and enhanced self-sufficiency services provided by the EHS programs are expected to have both direct and indirect effects on children across multiple domains of development, including emotional, behavioral, cognitive, and language outcomes. Therefore, the study integrates parent reports of child development with a number of direct assessment techniques to measure multiple domains of children’s functioning. The following aspects of child well-being will be assessed through the survey: social and emotional adjustment, child health and safety outcomes, and preschool and schooling outcomes. Different outcomes are more relevant for children of particular age ranges. Parents will be asked to report on their children who will be between the ages of 2.5 and 6.75 at the 36-month follow-up. Parents and primary caregivers will be asked to assess children’s emotional and behavioral adjustment and social competence on items drawn from the FACES survey and adapted from the CBCL and SSRS,³⁷ children’s school readiness and performance in preschool settings, including academic functioning, participation in special education or gifted/talented programs, grade repetition, and school suspensions/expulsions. These items are intended to provide overlapping information with the direct child assessments of children’s early academic skills and school readiness, and emotional and behavioral adjustment. Parents and primary caregivers will also be asked to assess children’s health and safety outcomes, such as child’s overall health, whether the child has any physical, medical, learning, emotional or behavioral conditions, child’s receipt of health care and immunizations, and child’s receipt of supervised treatment for any disabilities or health conditions.

Direct child assessments

The assessments aim to capture children’s school readiness skills (e.g., early literacy and numeracy skills), emotion and behavioral regulation, a construct that is targeted by the EHS program, and is critical to children’s development during the early childhood years that also appears to be associated with later academic outcomes as children enter school.³⁸

Having both parent-reported measures and direct assessments of children’s functioning enhances our ability to appropriately measure these constructs of interest, as well as to explore how different sources of measurement of child well-being relate to each other. Direct assessments and independent observations can provide a more objective measure of child development than parent reports alone because parent reports may be more open to biases based on the parents’ own characteristics, such as parents’ psychological well-being or financial circumstances. In contrast, parent reports may result in better information and more complete data about how children function in particular contexts and across time than a brief assessment conducted at a specific point in time.

Selecting age-appropriate assessments presents a unique challenge, because very few assessment tools can reliably measure children’s functioning across this broad age range. Direct assessments were prioritized and selected based on the following criteria: 1) direct assessments will be selected for comparability to the assessments used at the 15-month follow-up (this allows for measurement of the same construct at two time points in order to model within-person developmental growth as a function of program participation); 2) direct assessments will be selected for comparability to those used in prior experimental evaluations of child-focused services and two-generational programs, particularly the Early Head Start Research and Evaluation Project;³⁹ and, 3) direct assessments that can reliably measure children’s functioning for age groups, which prior research suggests are most likely affected by program participation, will be prioritized.

Direct assessments of children’s emotion and behavior self-regulation. For children younger than 4 years old at the 36-month follow-up, we will be using a parallel set of tasks as those used at the 15-month follow-up. The interviewer will perform several tasks to assess children’s motor control, attentional skills, and impulsivity. These tasks include: 1) drawing a circle at varying speeds; 2) walking a line at varying speeds; and, 3) waiting, and not peeking, while the interviewer “wraps” a gift that will later be given to the child. The appropriateness of these tasks for child 2 to 6 years old and sample of low-income and racially and ethnically diverse populations has been established by a number of studies.⁴⁰

For children who are 4 years old and older at the 36-month follow-up, a slightly different group of tasks will be used, so that the measures are age-appropriate for older children in the study sample. The developmental literature provides a number of age-appropriate direct child assessments, which tap children’s emotional and behavioral regulation.⁴¹ It is from this literature that we will be selecting a set of four tasks to be used with children who are 4 and older at the 36-month follow-up. In the attached draft of the protocol for administration of the direct child assessments and in the description below, we provide illustrative examples of these tasks from this literature. The set of tasks for the 36-month data collection effort will be finalized once pre-testing has been carried out to determine which tasks can be carried out in the allotted timeframe. These tasks may include: 1) drawing a star at varying speeds; 2) a snack delay in which the child is presented with a snack, but is then asked to wait for a period of time before given the snack to eat; 3) presenting the child with a target picture and an array of other pictures, which include the target, and then asking the child to identify as many of the target items as possible in a fixed amount of time; 4) walking a line at varying speeds; 5) a pencil tapping task, in which the child is asked to tap twice when the interviewer taps once and tap once when the interviewer taps twice; 6) asking the child to take turns with the interviewer while building a tower of blocks; and/or 7) asking the child to sort and put away small toys without playing with them. The appropriateness of these self-regulation tasks for children 3 years old and older has been tested and validated with low-income and racially and ethnically diverse populations.⁴²

Direct assessments of children’s cognitive development and school readiness. To assess all children’s cognitive development and early literacy and math skills, all children at the 36-month follow-up will be assessed using the broad math and reading subscales of the Woodcock-Johnson III (WJ-III).⁴³ For the broad math tests, the child might be asked to write a single number, solve simple arithmetic problems, or count and identify numbers, shapes and sequences. For the broad reading tests, the child might be asked to identify a printed letter, listen to and recall details of a story, or identify an object. This assessment was also used at the 15-month follow-up and was selected for its appropriateness for children 2 years old and older. The WJ-III has been widely used to assess children’s school readiness in a number of large-scale studies with national samples including the Early Head Start Research and Evaluation Project.⁴⁴ The WJ-III has been tested and validated with low-income and racially and ethnically diverse samples.⁴⁵

The direct child assessments will be administered by interviewers in the homes of study participants. The administration time of the assessments will vary with child’s age (longer for older children who will be participating in more assessments and shorter for younger children who will not participate in as many activities). Prior research suggests that the administration time averaged across children of different ages will be 30 minutes. Should the administration time be longer than anticipated based on pretests, we will delete tasks as necessary in order to stay within the 30 minutes allocated for the direct child assessments. We base our calculations of respondent burden in Section A12 on a 30-minute direct child assessment for the 36-month follow-up data collection effort.

Observational assessment of children’s emotional and behavioral adjustment

The interviewer will also assess the child’s emotional state during the administration of all the direct child assessments. The interviewer will be asked to fill out a brief assessment describing the child’s attention, emotion, and behavior skills throughout the interviewer-child interaction using the Adapted Leiter-R Assessor Report.⁴⁶

A3. Use of Information Technology for Data Collection to Reduce Respondent Burden

The use of improved technology has been incorporated into the data collection design wherever possible to reduce respondent burden. When information is available from a centralized, computerized source, such information has not been included in the data collection instruments described in this submission. For example, historical cash assistance (TANF), Food Stamps, and UI data will be obtained through administrative records.

It is also important to note that the CATI/CAPI technology that is likely to be used to administer the surveys can reduce respondent burden. Computer programs enable respondents to avoid inappropriate and non-applicable questions. This technology allows for “individualized” question phrasing, and thus for more streamlined administration. For example, male respondents can automatically be routed past questions about pregnancy because the program stores respondents’ gender. Respondents in single-family households will be skipped over the series of questions about the other parent’s employment, other characteristics and well-being. Also, depending upon the gender of focal child in the surveys with more in-depth child well-being sections, the appropriate “he” or “she” pronoun will automatically be inserted into the stems of questions.

A4. Efforts to Identify Duplication

The survey will focus on information that cannot be found in administrative records or other existing sources. The survey will facilitate the collection of data on, for example, respondents’ experiences in accessing program services, their physical and emotional well-being, their children’s health and behavior problems, other barriers to employment, family functioning, parenting, and father involvement. These types of information are not available routinely or systematically in program or administrative records.

A4.1 Reasons Why Available Information Cannot Be Used

Comparable information from other sources does not exist for the variables covered in 36-month parent-reported survey and direct child assessments for this study’s population. MDRC will use administrative data as the primary source for earnings, TANF payments, and Food Stamp payments. However, administrative data are not available for most of the other outcomes described earlier and, even when available, present problems. The collection would be very costly; many of these data sources are replete with different types of missing records and are maintained by different types of systems in each state, resulting in data which are not comparable across sites. Further, for some data, administrative records – such as program tracking data – are only available for the program group and not the control group. The lack of comparability would make it difficult to estimate differences between the research groups or to pool information across sites and potentially difficult to compare impacts across sites.

A5. Burden on Small Business

Does not apply. All respondents are individuals.

A6. Consequences if Data Collection is not Conducted

If the parent-reported survey and direct child assessments are not collected at the 36-month follow-up, we will not be able to adequately evaluate the longer-term impacts of EHS with enhanced employment and self-sufficiency services. The analysis of the short- and long-term impacts would be limited because changes in many important outcomes, such as barriers to employment (like depression or substance use), the experience of program services, job quality, job duration, wages, family functioning and routines, parenting, father involvement, and child well-being, cannot be captured in administrative records data.

If the data are not collected, program operators and policy makers will also receive less information about whether these particular enhancements to existing child-focused or two-generational services can lead to impacts on parents and children in low-income families. The implementation and process study also depends on the collection of survey data at the 36-month follow-up to obtain information on the services that are received by members of the program and control groups, as well as their views on the helpfulness of these services and assistance received. The survey is the only way of obtaining this data (particularly for members of the control group), and this information is critical to fully understanding the service receipt differential between members of the program and control groups, as both groups receive the same survey instrument.

A7. Special Data Collection Circumstances

No such circumstances.

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission

The 60-day Federal Register notice soliciting comments for the KS-MO 36-month data collection instruments was posted in the Federal Register on March 27, 2007 (Volume 72, Number 58, Page 14278-14279). Copies of the 60-day and draft of the 30-day Federal Register notices are located in Appendices D.1 and D.2.

Although the parent-reported survey and direct child assessments at the 36-month follow-up represents an effort to break new ground in assessing programs specifically designed to assist hard-to-employ parents and their children, these instruments do build upon previous research. We have consequently developed instruments that incorporate items, scales and measures from other major studies, in addition to the 15-month parent-reported survey and direct child assessments used in the Kansas and Missouri sites of the HtE evaluation. To the extent possible, measures included in the survey instrument and the direct child assessments allow for useful comparisons between the data from this project and that from other large-scale studies. We draw measures from prior research across various disciplines and prior experimental and non-experimental research. For the most part, measures on the parent-reported survey and direct child assessments were included in the current evaluation exactly as they appeared in prior research, while others were modified to reflect the goals of the HtE initiative and the current evaluation as fully as possible and also to reflect the low literacy and comprehension skills of the current evaluation’s study population.

Instruments that were used in the development of survey questions and selection of direct child assessments for the 36-month follow-up are as follows:

• MDRC surveys, including those used in the following projects: The HtE baseline survey for Rhode Island; The Employment, Retention, and Advancement (ERA) project; the Project on Devolution and Urban Change; the Minnesota Family Investment Program (MFIP); the National Evaluation of Welfare-to-Work Strategies (NEWWS); the New Hope Project; and Connecticut's Jobs First Evaluation;

• Surveys and direct child assessments used in evaluations done by Mathemetica Policy Research (e.g., The Early Head Start Research and Evaluation Project) and by Westat (the Head Start Family and Child Experiences Survey [FACES] and the Head Start Impact Study and Follow-up);

• Surveys and direct child assessments done in connection with child development studies, including the Infant Health and Development Program; the Fragile Families and Child Well-Being Study; the Welfare, Children, & Families: A Three City Study; and, the Early Childhood Longitudinal Study (ECLS); and

• National surveys, i.e., the National Longitudinal Survey of Youth (NLSY), the National Health Interview Survey (NHIS), and the National Co-Morbidity Study.

To select the measures for various components of the survey and direct child assessments, we consulted with a number of individuals outside MDRC, including: Aletha Huston of the University of Texas at Austin; Cybele Raver of University of Chicago; Grazyna Kochanska at the University of Iowa; Jeanne Brooks-Gunn of Columbia University; Karen Bierman of Pennsylvania State; Lisa McCabe of Cornell University; and, Margaret Briggs-Gowan of University of Connecticut Health Center.

In addition, representatives from the sites will also be asked to review the survey instrument and protocol for direct child assessments.

We also wish to remind readers that in all of the work on which we have drawn to build this survey and direct child assessments, we have worked, and continue to work, with many leaders in the social policy research field, including people working in academic, government and nonprofit settings. This long tradition of collaborative work will certainly influence the refinement, implementation and analysis of these data collection instruments.

A9. Justification for Respondent Payments

Parents who agree to participate in the survey will receive a payment of $35. Children who attempt the direct child assessment will also receive a toy valued at $5 as incentive for participation. This child’s gift is intended to thank children for their time in participating in the direct child assessments and will be given to children, regardless of whether they successfully complete the assessment tasks.

The purpose of the payment to parents for the survey and direct child assessments is to improve response rates by decreasing the number of refusals, enhancing respondent retention, and providing a gesture of goodwill to acknowledge respondent burdens. The payments are being proposed in addition to many of the techniques suggested by OMB to improve response rates that have been incorporated into our data collection effort and are described in Section B3, because our experience has shown that small monetary incentives are useful when fielding data collection instruments with hard-to-employ populations as part of a complex study design.

The best statement of current thought on incentives is the Symposium on Providing Incentives to Survey Respondents convened in October 1992 by the Council of Professional Associations on Federal Statistics (COPAFS) for OMB. COPAFS asked Richard Kulka of NORC to write a review of the literature in light of what was learned at the symposium. Kulka concluded, “the greatest potential effectiveness of monetary incentives appears to be in surveys that place unusual demands upon the respondent, require continued cooperation over an extended period of time, or when the positive forces on respondents to cooperate are fairly low.” Kulka also wrote, “there is evidence that increasing the size of a monetary incentive will result in increases in survey response and/or response quality, although there is also consistent evidence that this benefit may rather quickly reach 'diminishing returns', whereby large incentives no longer result in appreciable increases in survey response.”⁴⁷ We have based the amount of the incentive to be paid for these data collection elements on prior research conducted in this area, and MDRC’s and the survey firm’s prior experience interviewing similar populations.

In addition, more than two decades of survey research support the benefits of offering incentives. Hazard, citing evidence from a 1974 study by Ferber and Sudman found that the effects of incentives are contingent upon respondent burden (i.e., the effort needed to cooperate), the amount of the incentive, and the economic level of the respondent.⁴⁸ A study by Berlin, et al. found that incentives increased the response rates of respondents with low levels of literacy, as well as lowering interviewer costs.⁴⁹ James also found that an incentive was effective in lowering non-response rates and that any incentive lowered the number of interviewer visits per case.⁵⁰ The Mack et al. study of responders to the Survey of Income and Program Participation (SIPP) found that incentives reduced non-response rates in initial and subsequent interviews, and were particularly effective in reducing non-response rates in poor and African-American households.⁵¹ Moreover, the use of incentives has been found to be efficacious for increasing the response rates of in-home and sensitive subject matter surveys.⁵²

Finally, our prior experience fielding data collection instruments with economically disadvantaged and TANF-receiving populations also supports the evidence that incentives increase response rates. For example, in a follow-up interview with Jobs Corps applicants, experimental evidence showed that incentives increased response rates and greatly increased search efficacy. Experience in these and similar studies of disadvantaged populations suggest that incentives can help convince reluctant respondents to participate.⁵³

We believe that the studies summarized here, and MDRC’s previous experiences with fielding survey and direct child assessments with low-income populations, make a strong case for the use of respondent payments for completing the survey and direct child assessments.

A9.1 The Use of Incentives

To be effective, the amount of the incentives must fit the burden of the survey and the direct child assessments. We have based the amounts of the incentives for the 36-month data collection effort based on what was previously paid to HtE sample members for their participation in the 15-month follow-up, prior research, and MDRC’s and the survey firm’s prior experience interviewing similar populations. We propose that parents who agree to participate in the 36-month survey receive a payment of $35 and children who attempt the direct child assessments receive a small toy valued at $5. For parents, the proposed incentives may take on forms other than a cash payment, such as a transportation voucher, gift card, or telephone calling card for the given value. The child’s gift is intended to thank children for their time in participating in the direct child assessments and will be given to children, regardless of whether they successfully complete the assessment tasks.

These amounts reflect current practice in fielding surveys and direct child assessments using similar instruments. For example, the proposed incentive is in line with the size of the incentive found to be effective for the Project on Devolution and Urban Change survey efforts. For this study, a $20 incentive was given to respondents who completed the 90-minute interview in 2001. The gift will be valued at about $5 for children’s participation in the direct child assessments. This amount reflects the current practice for direct child assessments using a similar set of assessments.

Each instrument that will be used to collect follow-up data from HtE sample members has unique aspects that make administration difficult and threaten response rates. We are therefore requesting clearance to offer a small incentive to all sample members who complete each survey and who attempt the direct child assessments. Aspects of the data collection effort that also make it more difficult to obtain high completion rates are:

• The surveys include questions that could be perceived as intrusive and therefore could make respondents uncomfortable (i.e., questions about their mental health, drug and alcohol use, and experience of domestic violence).

• The subject matter of the interview is not intrinsically interesting to respondents. Moreover, many participants may have negative feelings about the other services received that are of interest, such as welfare, Medicaid, job training, etc.

• Other difficulties in administering the surveys come from the population itself. Educationally and economically disadvantaged groups, such as those in the HtE sample, have been found to be more difficult than the general population to convince to participate in surveys.

• Some children may be wary of interacting with the interviewer. The offer of the gift, not only thanks the child for his or her time, but also gives the child and the interviewer a common topic to talk about as the child gets to know the interviewer.

Thus, we are requesting clearance to offer small incentives to those who complete the survey and to children who attempt the direct child assessments to obtain response rates that will yield credible results, to avoid the bias that could result from selective non-response, and to reduce item non-response. We are aiming to achieve an 80 percent survey completion rate for the follow-up survey and direct child assessments. Even with the best data collection practices, it would be very difficult, if not impossible, to obtain such a high completion rate without incentives. In addition, providing parents an incentive for completing the survey not only increases the likelihood that they will complete the 36-month survey, but also increases the likelihood that these parents will be responsive to requests for their child to participate in the direct child assessments.⁵⁴ At the same time, the small toy is meant to be appealing to parents and children and will be given to children to thank them for attempting to participate in the direct child assessments.

A10. Confidentiality

A10.1. Consent

As indicated in the attached consent forms collected prior to random assignment, parents were asked to provide their consent, and consent on behalf of their children, to participating in subsequent waves of data collection, including a parent-reported survey and direct child assessments. Parents will be given the opportunity to agree to have their child be videotaped during the administration of the direct child assessments for checking the quality of the data collected (see detailed description below in section A16.1a) separately at the time of the 36-month follow-up. See attached consent form in Appendix A.2.

Participation in the parent survey, the direct child assessments, or the videotaping of the direct child assessments will be voluntary. At the time of data collection for the 36-month follow-up, parents, children, and/or parents on behalf of children, can choose not to participate in the survey or the assessments, or can choose not to be videotaped at any time.

The direct child assessments are not likely to constitute an undue burden or risk to children. However, it is possible that participating in the direct child assessments will increase the stress experienced by already at-risk study participants. If a child becomes unduly distressed during the direct child assessments, the interviewer will end the assessment.

A10.2. Protections for Individuals’ Confidentiality

MDRC and the survey firm – HumRRO – will protect against breach of confidentiality of parents and children participating in the 36-month data collection effort. These procedures for assuring and maintaining confidentiality will be consistent with the provisions of the Privacy Act and with ethical guidelines of professional organizations. Interviewers will attempt to conduct the interview at a time and place that allows the utmost privacy for respondents. Respondents will receive information about confidentiality protections at the outset of the interviews. They will be informed that all of the information they provide will be kept strictly confidential and that study results will be presented only in aggregate form.

MDRC’s and HUMRRO’s in-house records of names, addresses, Social Security numbers, and tracing information for all sample members (including parents and children) will not be attached to interview or assessment data and will not be made available to anyone outside appropriate staff of MDRC and HUMRRO. All records identifying respondents will be kept in locked storage at MDRC, and respondents will be identified solely by a code number. The videotaped information will not be edited or copied and will be destroyed upon completion of the study. Any coding, data entry and analysis requiring identification of individuals or households will use code numbers only, and a secret password will be necessary to access the data file. No data will ever be reported in such a way that individuals can be identified.

The importance of maintaining confidentiality will be emphasized during interviewer training, and any interviewer who knows a respondent will not be permitted to interview or conduct direct child assessments. All staff, including coders and computer programmers, will be required to sign a confidentiality pledge.

While conducting the parent survey or the direct child assessments, the interviewer may observe or become aware of situations where there is potential harm to the respondent, a child, or someone else. Some areas of inquiry on the parent survey also address sensitive issues; thus, completion of this survey may increase the stress experienced by already at-risk study participants. An introductory script will inform all study participants that information may be revealed to the appropriate authorities if the person appears to be a serious threat to anyone. MDRC will work with the survey contractor to develop a process for reporting potentially threatening situations to the appropriate authorities. Interviewers will be trained to address situations where there is a concern about harm to the child.   

A11. Questions of a Sensitive Nature

Questions in all components of the parent-reported survey are potentially “sensitive” for respondents. Respondents are asked about highly personal topics, some even stigmatizing. The questions we have included were selected in part because they have been widely used in previous research and are respected among experts. Moreover, all will be pilot tested prior to the survey’s full implementation, and if problems arise in regard to any specific items, their inclusion will be reconsidered. Also, all survey forms will contain instructions that explain questions before they are posed. Finally, respondents will be informed by program staff prior to the start of the interview that their answers are confidential, that they may refuse to answer any question, that results will only be reported in the aggregate, and that their responses will not have any effect on any services or benefits they or their family members receive. As mentioned in Section A10, MDRC and its contracted survey firm employ numerous safeguarding procedures to ensure confidentiality.

The collection of direct child assessments and the videotaping of these assessments are potentially “sensitive” for children and their parents. The data we have proposed collecting are being selected in part because they have been widely used in prior research and are respected among experts. MDRC and the survey firm will develop a list of “frequently asked questions” to provide to interviewers so that they are trained to respond to parents’ and children’s questions before they are posed. As mentioned in Section A10, MDRC and its contracted survey firm employ numerous safeguarding procedures to ensure confidentiality. Finally, parents and children will be told that they can refuse to participate in any portion of the direct child assessments, and that this will not have any bearing on the services or benefits that the parents or the child receive.

A12. Estimates of the Hour Burden of Data Collection to Respondents

Participation in all the survey and direct child assessments at the 36-month follow-up is completely voluntary. No sanction or penalty will be applied to respondents receiving state or federal assistance who choose not to provide information. Respondent payments, as described in Section A9, will be offered to each sample member who participates in the survey or direct child assessments.

The estimated response burden by instrument/component was calculated based on the time budgeted for the administration of the survey and the direct child assessments. Assuming a response rate of 80 percent, the total number of respondents for the survey is expected to be 488 parents and the total number of respondents to the direct child assessments is expected to be 488 children. These numbers were then multiplied by the average length of the survey and the administration time for the direct child assessments, respectively, and divided by 60 to determine the total burden in number of hours. The response burden breakdown for all instruments is shown in the table on the next page.

Instruments	Expected Number of Respondents	Number of Responses per Respondent	Average Burden per Response	Total Burden (Hours)
Parent-Reported Survey488	1	45 minutes (or .75 hours)	366.00
Direct Child Assessments488	1	30 minutes (or .5 hours)	244.00
TOTAL PERSON HOURS				610.00

A13. Estimates of Capital, Operating, and Start-Up Costs to Respondents

Not applicable. The 36-month follow-up data collection will be conducted by a subcontracted survey firm.

A14. Estimates of Costs to Federal Government

ACF, ASPE and DOL are funding these activities. The estimated cost for designing, administering, processing, and analyzing follow-up data is $1,050,000. On a year-by-year basis, these expenses are estimated to be:

Year Cost

2007 $200,000

2008 $425,000

2009 $425,000

A15. Changes in Burden

The 36-month follow-up is a new data collection effort and do not involve a change in burden.

A16. Tabulation, Analysis and Publication Plans and Schedule

A16.1a Assessment of Data Quality and File Construction

Assessing and monitoring the quality of the data from the direct child assessments. All interviewers will undergo training prior to the administration of the direct child assessments. To ensure that the interviewers are qualified to conduct the direct child assessments, they will undergo a certification process, in which they must meet select administration criteria on two separate practice assessments⁵⁵ and inter-rater reliability will be established prior to fielding of the direct child assessments. Intra-Class Correlations will be used to assess inter-rater reliability on continuous measures and Cohen’s Kappa statistics used to assess inter-rater reliability on all categorical measures. In addition, 20 percent of all direct child assessments will be videotaped and reviewed by MDRC in order to monitor reliability and to ensure that a high quality of data is being collected once fielding of the assessments has begun. Upon reviewing the videotapes, MDRC will be able to provide interviewers with feedback and guidance in order to trouble-shoot administration problems and deviations from the prescribed procedures and protocols for administering the direct child assessments. This will allow MDRC to make any mid-course corrections within a relatively short timeframe. Participation in the direct child assessments and videotaping is voluntary. At the time of the data collection effort, children, and/or parents on behalf of their children, can choose not to be videotaped, or can choose not to participate in the assessments (see consent form for videotaping in Appendix A.2).

Assessing and monitoring the quality of the data from the parent-reported survey. The follow-up survey will go through a rigorous series of tests for completeness and quality. Staff at the survey firm will review the initial cases completed by each interviewer as well as perform occasional spot checks after that. Editing/coding staff will review questionnaires for quality and consistency after this initial period. Interviewers will be apprised of any problems found and retrained as needed. During the coding of data, coder reliability checks will be undertaken repeatedly to verify that coding procedures are being followed correctly. Data entered into computer files will be assessed for missing information, outliers, and other data problems according to standard procedures. If necessary, questionnaires will be recoded. The survey firm will deliver to MDRC data sets of completed cases at agreed-upon intervals, along with marginal frequencies. The data and frequencies will be reviewed for outliers, unusual distributions and inconsistencies between data items.

Data file construction. Data from the 36-month survey and direct child assessments will then be merged with data from other sources. That is, data from the 36-month follow-up will be combined with previously collected data, including that routinely collected by welfare departments and administrative records information relating to welfare receipt, earnings, child welfare agency, and program tracking (if available) and data collected from the 15-month follow-up parent-reported survey and the direct child assessments.

Tabulation. None of the tables will present individual-level data, all of the results and sample characteristics will be presented in aggregate form.

A16.1b. Analysis Plans

As previously indicated, the HtE evaluation in Kansas and Missouri incorporates a random assignment analytic design. We offer a brief outline of how we will address the project’s long-term analytical goals, with a focus on how the follow-up survey and direct child assessment data will be useful in that process.

Estimating overall impacts. Although the use of a randomized design will ensure that simple comparisons of experimental and control group means will yield unbiased estimates of program effects, the precision of the estimates will be enhanced by estimating multivariate regression models that control for factors at baseline that also affect the outcome measures. Such impacts are often referred to as “regression-adjusted” impacts. Examples of factors that may affect outcomes are the sample members’ age, number of children, prior employment, and baseline barriers to employment.

Most of the analyses of overall impacts will result in estimation models that, in their basic form, can be expressed as follows:

(1) Y_ij = F (T, X_ni, U_ij)

where

Y is a vector of outcomes (e.g., post RA employment, earnings, welfare receipt, children’s behavioral adjustment and early literacy and math skills)

T is the treatment variable indicating whether the individual is a member of the program group

X is a vector of baseline characteristics to be controlled (e.g., baseline education level, child’s age and gender, household structure)

U is a vector corresponding to the residual (error) term

i is the subscript designating the individuals in the sample

j is the subscript designating the various outcomes of interest

n is the subscript designating the various personal characteristics to be controlled.

A range of outcomes (Ys) will be examined, including economic and employment outcomes, services receipt, child care outcomes, aspects of family functioning and routines, parental psychological well-being and health, parenting, parent-child relationships, father involvement, barriers to employment – depression, substance use, parental educational attainment, and children’s early academic skills and cognitive, emotional and behavioral outcomes.

Program/control group differences in economic and employment outcomes. A range of variables will be constructed to examine parental employment and economic outcomes using the impact model outlined above. Simple measures, like ever employed, number of months employed, number of jobs held over the follow-up, average wages over the follow-up period, wage growth, average earnings, welfare receipt, income over the follow-up period, job quality characteristics, work hours, and work schedules will be examined. In addition, we plan to construct “joint outcomes,” which allow us to examine experimentally the program’s effects on other employment and economic outcomes, like job retention. For example, using the following three outcomes –“not employed within the first six months,” “employed within the first 6 months and did not stay employed for each of the 6 subsequent months,” and “employed within the first 6 months and stayed employed for each of the 6 consecutive months”— allows us to put the entire program group and the entire control group into one of these three categories to examine employment stability over the follow-up period. This type of analysis has been conducted in several recent evaluations, such as NEWWS, MFIP, and SSP, to examine impacts on employment duration and stability.

Program/control group differences in non-economic (including family processes and child development) outcomes. Non-economic outcomes include data on barriers to employment (e.g., depression, substance use, parental educational attainment), participation in education and training activities, as well as child care outcomes, aspects of family functioning and routines, parental psychological well-being and health, parenting, parent-child relationships, father involvement, and child cognitive and emotional developmental outcomes. The program impacts on these outcomes will be examined using the impact model described above. We will construct a range of variables to gauge program impacts on a variety of outcomes. Simple variables representing some non-economic outcomes will be examined, such as average number of months in center-based child care and child’s standardized scores on the broad reading and math subscales of the Woodcock Johnson. We will also construct variables assessing respondents’ participation rates, such as whether respondents ever participated in individual activities and in any activity overall. Furthermore, we will construct variables measuring program impacts on joint outcomes, such as whether the child was in child care while the parent was working or not working and cumulative indicators of developmental disadvantage across multiple domains of child functioning and well-being.

Another key purpose of these non-economic outcomes is to inform the implementation research. Data on participation will be used to measure the variation in implementation strategies across the evaluation sites. We also will use the impact findings on the non-economic measures to provide a context for interpreting the program's basic earnings and welfare impacts, and for interpreting the results of the formal benefit-cost analysis, which can only estimate, in dollar terms, the net present value of the program's measurable economic effects.

For some of these analyses, we will use individual survey items or pre-existing scales and measures. In some cases, however, we may create scales using multiple items. In building these scales, we would use standard social science methodologies.⁵⁶ For example, the first step would be to identify the set of items in the survey that were intended to address the same broad topic, such as skills required on the current or most recent job. We would then examine inter-item correlations for the full set of questions designed to measure this outcome and conduct a factor analysis to determine which items in the set “go together” and appear to be measuring the same underlying construct. Next, we would estimate Cronbach's alpha to assess the reliability of the scale. We would add and delete items as appropriate to maximize Cronbach's alpha. After selecting the final set of items for a given scale, we would then produce an overall scale score for each respondent by summing her scores on each of the items in the scale. The overall scale scores for all respondents would then be used as an outcome measure for the impact analysis. We have used this general approach successfully in several previous evaluations, especially the more recent evaluations with child outcomes data.⁵⁷

Subgroup analyses. Previous evaluations of welfare-to-work, child-focused and two-generational programs have found that, in some cases, impacts can vary for certain subgroups based on their demographic characteristics or circumstances at baseline. For example, in the Early Head Start Research and Evaluation, subgroups of families were considered based on their cumulative risk factors (many demographic risk factors were considered, including income, single parent status, receipt of public assistance, not being employed, in school, or in job training, teenage parent status, lacking a high school diploma or GED) and findings suggest that families with two or three risk factors had the largest positive impacts on children’s cognitive and language development, while impacts on these outcomes in families with greater than three risk factors were unfavorable.⁵⁸ The MFIP program, for instance, produced larger earnings impacts for recipients living in public housing than for those in private housing.⁵⁹ It is easy to imagine that the EHS with enhanced employment and self-sufficiency services might also be more effective for parents with particular characteristics, such as those with relatively modest barriers to employment. For this reason, it is essential to go beyond the examination of overall impacts of the program to examine impacts for subgroups defined by level of disadvantage and other characteristics. Impacts might differ for parents and children according to parents’ level of education at program entry, prior work experience, number and ages of children in the household, and prior welfare receipt. Program impacts might also vary by children’s characteristics, such as gender and age at study entry. Exhibit B1.1, showing minimum detectable effects for various sample sizes, indicates whether the impacts can be estimated with precision when the sample is split into various subgroups. This information will guide our analyses of subgroups.

An analysis of subgroup impacts involves estimating the program’s effects for each subgroup separately, using the regression-adjusted model mentioned earlier, and then comparing the two impacts. The standard errors of each of the impacts are used to assess whether the two impacts are statistically significantly different from each other. Subgroup impacts estimated in this way are referred to as unconditional subgroup impacts because they show the gross effect of a particular characteristic, such as education level, on a program’s impacts. As an example, earnings impacts in a program may be lower for individuals without a high school degree, as compared with their more educated counterparts. However, this difference may arise not because of education per se, but because less educated individuals are also less likely to have recent work experience, which also affects how they benefit from the program. In this case, it would be of interest to estimate conditional subgroup impacts, or impacts by education level that also control for prior work experience. These impacts would be obtained by pooling the sample and estimating one impact model, in which education level and prior work experience are interacted with all of the other variables in the model and with the program group dummy variable (T in the previous model). For example, if the coefficient on the interaction of program status and education is reduced in size once the interaction of program status and prior work experience is included, we can conclude that some part of the effect of education on the program’s impacts is due to its correlation with prior work experience.

Non-experimental analyses. Several types of non-experimental analyses will be conducted to complement the estimation of the program’s impacts. First, using data on participation and contact with caseworkers, we can examine what types of families participated in the program and which families were difficult to contact and engage in certain services. Second, we can examine treatment-on-treated effects. A similar set of analyses was also used in the final report of the Early Head Start Research and Evaluation Project.⁶⁰ For the HtE evaluation, we are primarily interested in whether participation patterns of service receipt shed light on the pattern of impacts on particular outcomes within and between sites. This is an important research question because EHS services are voluntary and program participation may be somewhat lower as a result. Treatment-on-treated impacts for families can be estimated most simply by dividing the overall impacts by the participation rate. Alternatively, a more complex method that attempts to recreate an experimental comparison is “propensity score matching,” in which impacts are estimated by comparing outcomes for families in the program group with outcomes for “matched” individuals from the control group. Third, the survey data will provide useful descriptive information on the circumstances of parents who are hard-to-employ welfare recipients and low-wage workers based on the severity and persistence of a range of barriers to employment. Finally, non-experimental analyses can test whether experimentally induced changes in parental employment can be linked statistically with changes in children’s developmental outcomes.⁶¹

A16.2. Publication Plans and Schedule.

Follow-up survey instruments will be administered to parents or primary caregivers and direct child assessments will be conducted with children approximately 36 months after the family was randomly assigned. Fielding is expected to begin as early as September 2007 and end as late as December 2009.

Findings from the 36-month follow-up data collection instruments will be part of the impact, implementation, and benefit-cost analyses. The results will be published in a series of reports based on the results of these analyses. Preliminary results will be available in 2008 with a final report being produced in 2010, as outlined in section A1.2.

A17. Reasons for Not Displaying the OMB Approval Expiration Date

Not applicable. We intend to display the OMB approval number and expiration data on all data collection instruments and materials.

A18. Exceptions to Certification Statement

Not applicable. We have no exceptions to the Certification Statement.

1 Current Population Survey, 2004.

2 Duncan & Brooks-Gunn, 1997.

3 Lennon, Blome, & English, 2001; McLoyd, 1990.

4 Acs & Loprest, 1994; Bernheimer & Weisner, 2000; Wolfe & Hill, 1995.

5 Shonkoff & Phillips. 2000; Olds et al., 1999.

6 U.S. Department of Health and Human Services, 2002.

7 Yoshikawa, 1994.

8 Smith, Blank, & Collins, 1992.

9 Duncan & Brooks-Gunn, 1997.

10 In order to quality for EHS services, infants and toddlers must be younger than 3 years old, however children can remain in the program until they transition to Head Start at 4 years old.

11 Note that in some cases, the income requirement can be waived if the child or family has special needs (as determined by the individual Early Head Start program). However, no more than 10 percent of the program’s enrolled caseload can exceed the income eligibility requirement at one time.

12 Three programs in Kansas and Missouri were initially identified that met all of the selection criteria and agreed to participate in the Hard-to-Employ Evaluation. Due to programmatic challenges, including difficulties sustaining a waitlist, one of these sites was excluded from the evaluation.

13 U.S. Department of Health and Human Services, 2002.

14 Gennetian, Morris, Bos, & Bloom, 2005.

15 Moore & Driscoll, 1997; Ripke, Huston, & Mistry, 2001.

16 Heymann, Earle, & Egleston, 1996

17 Han, Waldfogel, & Brooks-Gunn, 2001; Presser, 2004; Waldfogel, Han, & Brooks-Gunn, 2002

18 Han, 2005; Henly, Shaefer, & Waxman, 2006; Hsueh & Yoshikawa, 2007.

19 Parcel & Menaghan, 1990; 1994; 1997.

20 for reviews, see Smolensky & Gootman, 2003; Zaslow & Emig, 1997; Yoshikawa, Weisner, & Lowe, 2006.

21 Kalil & Ziol-Guest, 2005; McLoyd, Jayaratne, Ceballo, & Borquez, 1994; Yoshikawa et al., 2006.

22 Conger et al., 2002; McLoyd, 1990.

23 Morris, Huston, Duncan, Crosby, & Bos, 2001.

24 Edin & Lein, 1997; Newman, 1999.

25 Conger & Elder, 1994; Elder, 1999; McLoyd, 1990.

26 Burchinal, Roberts, Nabors & Bryant, 1996; NICHD Early Child Care Research Network. 2000; Phillips, McCartney, & Scarr, 1987.

27 Becker & Thomes, 1986.

28 Seitz, Apfel, & Rosenblum, 1991.

29 Stoneman, Brody, Churchill, & Winn, 1999.

30 e.g., Maccoby & Martin, 1983.

31 Hoffman, 1989

32 Downey & Moen, 1987; Moen, 1982.

33 McLoyd, 1990.

34 Edin & Lein, 1997.

35 Edin & Lein, 1997; Yoshikawa, Magnuson, Bos, & Hsueh, 2003.

36 Henly, Shaefer, & Waxman, 2006.

37 Achenbach & Edelbrock, 1981; Gresham & Elliott, 1990.

38 Alexander, Entwistle, & Dauber, 1993; Ladd & Price, 1987; Ladd, Kochendorfer, & Coleman, 1997; O’Neil, Welsh, Parke, Wang, & Strand, 1997.

39 U.S. Department of Health and Human Services, 2002

40 e.g., Kochanska, Murray, & Harlan, 2000; McCabe, Hernandez, Lara, & Brooks-Gunn, 2000; Raver, Blackburn, Bacroft, & Torp, 1999; The Welfare Reform in Three Cities study (principal investigators: Cherlin, Moffitt, Burton, Wilson and Chase-Lansdale).

41 Kochanska et al., 2000.

42 e.g., Kochanska et al., 2000; McCabe et al., 2000; Smith-Donald, Raver, Hayes, & Richardson, in press.

43 McCrew & Woodcock, 2001.

44 U.S. Department of Health and Human Services, 2002; Zaslow et al., 2003.

45 McCrew & Woodcock, 2001.

46 Roid & Miller, 1997; Wakschlag et al., 2005.

47 Kulka, 1992.

48 Hazard, 2002.

49 Berlin et al., 1992.

50 James, 1997.

51 Mack, Huggins, Keathley, & Sudukchi, 1998.

52 Hazard, 2002.

53 Moffitt, 2004. 

54 Singer, Van Hoewyk, & Maher, 1998.

55 Goyette et al., 2006, June.

56For a discussion of these methods, see DeVellis, 1991.

57 See Gennetian & Miller, 2000.

58 U.S. Department of Health and Human Services, 2002

59 Miller et al., 2000.

60 U.S. Department of Health and Human Services, 2002.

61 Gennetian et al., 2005