Previously Reviewed ISIS Followup Child Roster Justification and Logic Models

Supporting Statement for OMB Clearance Request

Appendix E: Logic Models Justifying the Inclusion of a Child Roster

Innovative Strategies for Increasing Self-Sufficiency (ISIS) – Follow-up Data Collection

OMB No. 0970-0397

March 2013

Submitted by:

Brendan Kelly

Office of Planning, Research
and Evaluation

Administration for Children
and Families

U.S. Department of Health and Human Services

Appendix E: Logic Models Justifying the Inclusion of a Child Roster

This appendix presents two logic models that illustrate how ISIS programming may generate outcomes (both positive and negative) for preschool (ages 0-5) and school-age (5-18) children.

The next two pages present those logic models. The text that follows explains those logic models and the research that supports them.

Logic Model for ISIS Program Impacts on Children

The primary goal of ISIS is to improve career employment and earnings for low-skilled, low-income workers, many of whom are parents. For participants with children, career pathways programs may have indirect effects on children’s cognitive development and academic performance, behavior, health, and socio-emotional development. Programs that improve parental education, employment and earnings may indirectly affect the environments that children experience, which, in turn affect their outcomes. In addition, some programs may have also a direct impact on children’s out-of-home care experiences by providing high quality center-based care for children. It is through changes to children’s material environments, family environments, and out-of-home care and education environments that employment programs for parents are hypothesized to influence children’s outcomes (see Huston, 2002; Tout, Brooks, Zaslow, et al., 2004).

Experimental Findings about Impacts of Welfare Reform on Children. Previous research has found that programs designed to improve the economic circumstances of low-income families can also affect children’s school achievement and behavior (see Morris, Knox, & Gennetian, 2002; Morris, Gennetian, & Duncan, 2005; Morris, Duncan, & Clarke-Kauffman, 2005 for synthesis of findings from more than a dozen random assignment studies of welfare reform strategies). Programs that offered earnings supplements in addition to employment-related services (i.e., education, job-skills training, job search) tended to increase both parental employment and income, and in turn, were also found to have statistically significant impacts on school achievement among preschool and school-age children with effect sizes ranging from .08 to .25, corresponding to an increase in achievement test scores from the 25^th to the 30^th percentile (Morris, Knox, & Gennetian, 2002; Morris, Gennetian, & Duncan, 2005). Programs that provided employment-related services without earnings supplements were found to improve parental employment but not income, and they had few impacts on preschool children’s school readiness, school-age children’s school achievement, and children’s behavioral and health outcomes. Despite findings that there were not consistent positive impacts on children of programs that provided education and training alone, further investigation suggests this may have occurred because programs were not successful at generating an educational advantage among parents compared to the control group. An instrumental variable analysis indicated that increases in maternal education were positively associated with children’s academic school readiness and negatively associated with academic problems (Magnuson and McGroder, 2002).

In contrast to the findings for impacts on preschool and school-age children, there were statistically significant negative impacts on adolescents’ school achievement and progress (i.e., greater grade retention) for both types of programs (Morris, Knox, & Gennetian, 2002; Morris, Duncan, & Clark-Kauffman, 2005). Negative program effects on school achievement and progress were most pronounced for adolescents with younger siblings, who were often responsible for the care of their younger siblings. In addition, for this group of adolescents, suspension, expulsion, and dropout rates were higher in program groups than in control groups (Gennetian, Duncan, Knox, Vargas, Clarke-Kauffman, and London, 2002). Program impacts on children tend to vary for children of different ages and may also vary based on other family characteristics.

Theory of Change for ISIS Program Impacts on Children. The body of experimental research on welfare reform strategies offers little evidence about the pathways from direct impacts on parents, such as increased income, employment, or education, to improved outcomes for children. In this section, we describe a logic model illustrating hypothesized pathways from impacts on parents to impacts on children, which is based on largely correlational research about the associations of parental income, employment, and education with children’s environments and the associations of children’s environments and interactions with their developmental outcomes.

Changes in parents’ employment, education, and income are not expected to have a direct impact on children’s development. However, such impacts on parents are hypothesized to affect the environments in which children are being raised, which in turn influence children’s cognitive, academic, behavioral, socio-emotional, and health outcomes. Specifically, it is children’s environments and experiences that are hypothesized to affect their development. To the extent that ISIS programs lead to changes in parents’ employment, education, and income, which thereby lead to changes in children’s environments, we can expect to see impacts on children’s cognitive, behavioral, socio-emotional, and health outcomes.

Increases in families’ income can lead to improvements in children’s physical and material environments, such as adequate housing, greater safety, adequate nutrition and healthcare, as well as resources for more educationally enriching experiences. Although families, even poor families, budget their money in an effort to benefit children (Edin and Lein, 1997), families with low incomes have less money to spend on children. Poor children are more likely to experience hunger, homelessness, exposure to violence and other threats to physical safety, and inadequate preventive health care than more economically advantaged peers (see Brooks-Gunn and Duncan, 1997). Poor children tend to have a greater risk of health problems, injuries, failure to thrive, and untreated medical and dental conditions (McLloyd, 1998; Federal Interagency Forum on Child and Family Statistics, 2011). In addition, by middle childhood and adolescence, children’s sense of well-being may be influenced by their perceptions of their family’s economic hardship (McLloyd, 1998).

Other aspects of the family environment, beyond the material environment, may also be improved by increased employment, income, and education. Parental well-being, which includes reductions in stress and depression, as well as increases in life management skills and aspirations, may contribute to a more positive home environment—one that is more stable, less chaotic, and characterized by more positive, supportive parent-child interactions. Positive parent-child interactions are related to positive behavior and emotional well-being (see Huston, 2002). Parents with mental health problems, particularly depression, tend to be inconsistent in their interactions with children—alternately withdrawn and punitive—and also tend to provide fewer learning experiences for children then parents without depression or other mental health problems (see Brooks-Gunn & Duncan, 1997; McLoyd, 1998; Duncan, Brooks-Gunn, & Klebanov, 1994).

Improvements in parental education and parental well-being may contribute to more enriching parent-child interactions at home, which support children’s cognitive development and school achievement (Magnuson and McGroder, 2002; Duncan, Brooks-Gunn, & Klebanov, 1994). Similarly, increased education, employment, and income may provide access to better quality child care and afterschool care. High quality out-of-home care, particularly, center-based care for young children, supports positive cognitive development and school readiness (NICHD ECCRN, 2002, 2005; Morris, Gennetian, & Duncan, 2005). ISIS programs may influence access to child care indirectly through increased income for purchasing care in the market, and some ISIS programs may provide direct access to child care as part of the package of services available to families. Evidence from welfare reform research indicates that programs increased families’ use of center-based child care, and increases in center-based care were positively associated with children’s school achievement (Morris, Gennetian, & Duncan, 2005). In addition, increased education, employment, and income may promote access to better schools and neighborhoods, which support positive achievement, behavior, and socio-emotional outcomes (Klebanov, Brooks-Gunn, & Duncan, 1994).

As indicated by the evaluations of welfare reform programs, increases in employment and participation education programs may also yield some negative changes to children’s environments, particularly for adolescents. Regardless of children’s age, the intense demands of employment and education programs could contribute to increased stress levels for parents and turbulence in the home, at least in the short-term, which could, in turn, lead to more negative parent-child interactions. In addition, increased reliance on self-care for school-age children and adolescents may be less enriching than parental care, as unsupervised children spend hours watching television or engaging in risky behaviors.

In summary, children who are preschool age (2 to 5 years) when parents entered programs that offer earnings supplements show improved school achievement, compared to children in a control group. For some programs, these impacts were sustained for at least two years after the program ended. However, as impacts on parents’ economic outcomes weakened, effects on children also faded. Furthermore, consistent positive impacts on children’s school achievement were not evident for children of all ages or for all types of employment programs. There appear to be negative consequences for adolescents. In addition, programs that increased parental employment without increasing income tended not to have an impact on children. However, there is some evidence that increases in education may also contribute to improvements in children’s school readiness and reductions in behavior problems.

Given experimental evidence of impacts of welfare reform programs on children’s school achievement and behavior, as well as the theoretical basis for hypothesized changes in children through the pathway of changes in children’s environments, there is a well-justified basis for conducting a study examining impacts of ISIS programs on child outcomes.

The addition of a household child roster to the BIF will provide information on the number and ages of children for each sample household (be it in the treatment or control group). The roster will provide information needed to draw a sample of preschool and school-age children at the time of longer-term follow-up.

Outcomes. As the logic models indicate, the ISIS programs have the potential to affect child outcomes across a range of domains. The broad set of outcomes, coupled with the potentially wide age range in the child sample, presents a challenge for comprehensive measurement of program effects. Because of the likely challenge of small sample sizes within ages and the resulting need to group children in multi-year age categories, the draft measures privilege assessments that are valid across a broad age range. In addition, because the data collection may not include direct access to the children in the sample, priority is given to measures that are based on parent or teacher report (versus direct administration of a child assessment), or where extant data from schools might be used to measure child academic and social outcomes. The final set of child outcome measures will be selected based on the ages of the children in the sample, the data collection methodologies being implemented, and prioritization of the child outcome domains.

The specific measures are yet to be determined, but are expected to fall in the following domains:

Socio-emotional development. Likely measure candidates include the Social Skills Improvement System Rating Scales (SSIS) subscales regarding Competing problem behaviors and Academic competence; and the Brief Rating Inventory of Executive Functioning (BRIEF) Global Executive Composite score and subscales on Behavioral Regulation and Metacognition.

Academic performance. In this domain, we propose to use a combination of parent or teacher reports as well as extant data and direct assessments, such as the Woodcock Johnson III (WJ III) Tests of Achievement for preschool aged children and age/grade-appropriate aptitude tests.

Risk behavior. This is an important domain to measure for children 12 and older, since previous research has suggested that parent employment is associated with more risk behavior because of reduced parental supervision. Candidate measures include items from the CDC’s Youth Risk Behavior Surveillance System, including measures of (a) behaviors that contribute to unintentional injuries and violence, (b) use of tobacco, alcohol and other drugs, (c) sexual risk behaviors, (d) unhealthy dietary behaviors, (e) physical inactivity, (f) prevalence of obesity, asthma.