OMB HtE RI 36M_Supporting Statement Part A REV_12.05.07

SUPPORTING STATEMENT

FOR OMB CLEARANCE

PART A

DHHS/ACF/ASPE/DOL

ENHANCED SERVICES FOR THE HARD-TO-EMPLOY (HtE)

DEMONSTRATION AND EVALUATION PROJECT

RHODE ISLAND 36-MONTH DATA COLLECTION INSTRUMENTS

November 14, 2007

TABLE OF CONTENTS

A. JUSTIFICATION 1

A1. Circumstances Necessitating Data Collection 1

A1.1 Previous Research on Depressed Individuals 3

A1.2 Overview of the HtE Evaluation - Rhode Island Site 8

A2. How, by Whom, and for What Purpose Are Data to be Used 11

A2.1 The Overall Role of Follow-up Survey Instruments in the HtE Evaluation 12

A2.2 The Role of Specific Follow-Up Survey Components 12

A3 Use of Information Technology for Data Collection to Reduce Respondent Burden 15

A4. Effort to Identify Duplication 15

A4.1 Reasons Why Available Information Cannot Be Used 15

A5 Burden on Small Business 15

A6 Consequences to Federal Program or Policy Activities if Data Collection is Not Conducted…. 15

A7. Special Data Collection Circumstances 16

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission 16

A9. Justification for Respondent Payments 17

A9.1 Prior Research on Respondent Incentives 17

A9.2 The Use of Incentives in the HtE Rhode Island Site Follow-Up Surveys 19

A10. Confidentiality 19

A11. Questions of a Sensitive Nature 21

A12. Estimates of the Hour Burden of Data Collection to Respondents 21

A13. Estimates of Capital, Operating, and Start-Up Costs to Respondents 21

A14. Estimates of Costs to Federal Government 22

A15. Changes in Burden 22

A16. Tabulation, Analysis, and Publication Plans and Schedule 22

A16.1a Assessment of Data Quality and File Construction 22

A16.1b Data Analysis 23

A16.2 Publication Plans and Schedule 26

A17. Reasons for Not Displaying the OMB Approval Expiration Date 26

A18. Exceptions to Certification Statement 26

B. COLLECTION OF INFORMATION USING STATISTICAL METHODS 27

B1. Sampling....... 27

B2. Procedures for Collection of Information 28

B3. Maximizing Response Rates 29

B4. Pre-testing…… 30

B5. Consultants on Statistical Aspects of the Design 30

LIST OF EXHIBITS

A1-1: Major Research Questions in the HtE Evaluation-Rhode Island Site……………… 8

B1-1: Follow-Up Survey Sample Sizes …………………………………….....………………27

B1-2: Minimal Detectable Effects for Key Outcomes,

5 Years of Follow-Up, New Hope Project……......……………………………………27

LIST OF APPENDICES

A.1: Rhode Island 36-Month Interview

A.2: Rhode Island 36-Month Youth Interview

A.3: Parent Consent Form: 36-Month Follow-Up Child Add-On

A.4: Youth Assent Form: 36-Month Follow-Up

B.1: Federal Register Published 60-Day Notice

B.2: Federal Register Draft 30-Day Notice

C: Statute/Regulation Authorizing Evaluation and Data Collection: Social Security Act,

Section 1110

D: References

A. JUSTIFICATION

This document requests OMB authorization for data collection activities related to the Rhode Island 36-month follow-up survey for the Enhanced Services for the Hard-to-Employ Demonstration and Evaluation Project (the HtE evaluation).

A1. Circumstances Necessitating Data Collection

The HtE evaluation targets an issue that has gained increasing prominence in the wake of the welfare reforms of the 1990s: how to improve the employment prospects of hard-to-employ parents who face serious obstacles to steady work. These obstacles include physical and mental health problems, substance abuse, and criminal convictions, as well as human capital deficiencies and situational barriers. Welfare reforms, which contributed to a reduction of over 50 percent in welfare caseloads across the country, have pushed the hard-to-employ population to the top of the nation’s social policy agenda and magnified the need to find effective employment strategies for them. As interest in hard-to-employ adults has increased, there is also a growing concern that the problems that impede parents’ employment may also pose threats to their children. While many existing programs are innovative and exciting, there is little hard evidence about what works best for whom within this population.

Research on public assistance recipients indicates that as many as one-quarter have experienced past-year depression.¹ Moreover, their depression may be one of several barriers that limit their employability.² Although a considerable body of random assignment research has identified various types of efficacious treatment for depression³ and indicates that “treatment for depression can reduce job loss and work-related impairments,”⁴ studies that are specifically applicable to low-income, hard-to-employ populations, in particular Temporary Assistance for Needy Families (TANF) recipients, are not yet available.

Considerable research demonstrates the benefits of therapeutic and pharmacological treatments in reducing depression. Yet, there are few studies on treatment effectiveness for low-income parents, where rates of depression are very high (estimated as high as one-quarter to one-third of welfare recipients in some studies)⁵. Moreover, the critical issue in reducing depression is not about knowing how to treat depression, but rather, how to get depressed parents into evidence-based treatment—even in mixed-income samples, as few as one-fifth of depressed individuals may receive adequate treatment.⁶ This problem is exacerbated in low-income communities, where knowledge of depression treatment is lower and stigma for the receipt of treatment is higher than in higher-income communities. Moreover, even among those individuals who do seek treatment, relapse rates are quite high,⁷ suggesting the importance of strategies that maintain continuity of care.

One promising way to address this problem is through care management, which is designed to support clinical treatment by actively facilitating an individual’s engagement in treatment, with particular emphasis on the quality and continuity of that treatment. Working Toward Wellness (WtW) is a telephone care management intervention designed to help Medicaid recipients who are experiencing major depression seek and remain in evidence-based treatment. Individuals are being offered WtW only as part of the Hard-to-Employ evaluation. The care manager-outreach monitoring model was developed by researchers from the Group Health Cooperative (GHC) in Seattle and is currently being evaluated among a working population in a large-scale study, Outreach and Treatment for Depression in the Labor Force, funded by the National Institute of Mental Health and led by a research team from Harvard Medical School.⁸

Another critical question for policy is how changes in parents’ depression affect the development of low-income children. Such high rates of depression are particularly problematic for low-income children, as a wealth of research has shown that children of clinically-depressed parents face difficulties in their social behavior and psychological functioning, problems that are likely to be compounded by the stresses of living in poverty. However, we know almost nothing about whether treatment for low-income parents’ depression will improve outcomes for their children.

Therefore, in addition to examining the changes in maternal depression brought about by the GHC care manager-outreach monitoring intervention, we are also interested in examining the effects of this experimental intervention on children. This intervention model builds on a series of studies demonstrating the effectiveness of care management, in general, and telephonic delivery in particular, for patients in primary care settings.⁹ This work represents a significant advance in developmental research and informs both science and policy. The analyses proposed—using experimental-control impact estimates as well as instrumental variables (IV) estimation techniques—are the first to make use of a random assignment design to address the ways in which changes in parents’ depression affect psychosocial (behavioral problems as well as skills and socioemotional competencies), clinical (psychiatric diagnosis of affective disorders), and cognitive (verbal functioning and achievement) outcomes for children and adolescents in low-income families.

Our goal is to collect and analyze data on outcomes for parents and their children at the 36-month follow-up, to provide long-term findings of the intervention on effects on health and employment for the parents, as well as effects on children of any changes in depression among parents. The data collection strategy is multi-pronged, and includes a combination of in-depth parent surveys, in-depth surveys with the adolescents, and direct child assessments with the young children. Analysis will be conducted to assess differences between families assigned to the program group and those assigned to the control group, as well as analyses to tease out the direct effects of changes in maternal depression on outcomes for children. In addition to the follow-up surveys discussed in this submission, the evaluation includes a baseline survey, a 6-month follow up survey, 18-month follow up surveys (approved as 15-month follow-up by OMB No. 0970-0276, however fielding was delayed by three months) and administrative records data collected from the TANF, Medicaid, food stamps, Social Security, and Unemployment Insurance (UI) systems.

The HtE evaluation is the most ambitious, comprehensive effort to learn what works in this area to date and is explicitly designed to build on previous and ongoing research by rigorously testing a wide variety of approaches to promote employment and improve family functioning and child well-being. The HtE project will “conduct a multi-site evaluation in up to six sites that studies the implementation issues, program design, net impact and benefit-costs of selected programs”¹⁰ designed to help Temporary Assistance for Needy Families (TANF) recipients, former TANF recipients, or low-income parents who face serious barriers that make them hard-to-employ. The project is sponsored by the Office of Planning, Research and Evaluation (OPRE) of the Administration for Children and Families (ACF), the Office of the Assistance Secretary for Planning and Evaluation (ASPE) in the U.S. Department of Health and Human Services (HHS), and the U.S. Department of Labor (DOL). Following a competitive procurement process, HHS awarded a contract to MDRC to conduct a major, multi-year evaluation of various HtE program approaches in selected sites.¹¹ For the Rhode Island site, MDRC has subcontracted with Group Health Cooperative and United Behavioral Health to assist in site development and analysis and with HumRRO to administer the 18-month and 36-month follow-up surveys (DataStat administered the 6-month follow-up survey).

A1.1 Previous Research on Depressed Individuals

A1.1a Effects of Interventions on Adults

Effects of traditional welfare-to-work programs on depressed welfare recipients. The 1996 Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA) was the culmination of several decades of efforts to promote work and reduce long-term welfare receipt among single-parent families. As a result of these efforts, Aid to Families with Dependent Children (AFDC), which had guaranteed aid for low-income families with children, was eliminated and replaced with Temporary Assistance for Needy Families (TANF), which provided block grants to states, introduced time limits on cash assistance, and imposed work requirements on recipients.  At the same time, these changes highlighted the need for effective employment strategies for parents facing significant barriers to employment, including depression.  Prior to 1996, depressed welfare recipients were typically exempted from work requirements. Today, PRWORA limits exemptions for work requirements or time limits and places greater emphasis on quick employment.  These narrower exemptions have pushed states to work with many clinically depressed individuals who were previously exempted or deferred, while time limits and stricter sanctions have raised the stakes to help these families make the transition to employment. At the same time, the newest welfare reform proposals have raised child well-being to an explicit goal of welfare policy, making the fact that parental depression is a risk factor for employment as well as child well-being¹² a critical target for welfare policy.

Recent studies of welfare recipients and other low-income populations (e.g., recipients of WIC or family planning services) report a prevalence of depression at one-quarter to one-third,¹³ with rates closer to one-half among more disadvantaged samples such as young mothers.¹⁴  Research on welfare-recipient samples finds that depression is a critical barrier to employment among low-income parents, although research is more limited for this population than for other segments of working-age adults.¹⁵  While clinical depression is only one of a number of barriers that welfare recipients face (including low basic skills and substance abuse), a recent synthesis of the results from 20 mandatory welfare-to-work programs finds that these approaches were moderated most markedly by risk of depression as compared to the many characteristics of recipients measured at baseline.¹⁶  That is, parents facing higher risk of depression did not show increases in earnings over the course of the follow-up periods as a result of these approaches, while those at lower risk for depression did experience such effects.  Experiencing major depressive disorder appears to independently affect the likelihood of working more than part time, even after accounting for the negative effects of other employment barriers on employment.¹⁷ This is one example of a large body of research that links depression to work-related outcomes in non-poor samples.¹⁸  Clearly, innovative strategies for moving this higher risk group of welfare recipients into employment are critical.  One such strategy is to integrate the best clinical treatment practices for depression into welfare-to-work programs.

Using intensive care management to reduce depressive symptomatology. Considerable progress has been made in the field of depression treatment, suggesting the benefits of both therapeutic and pharmacological approaches in reducing depression, and in combination for those with more severe and chronic depression.¹⁹ As mentioned earlier, care management may be a promising way to address high rates of depression in low-income populations. Six and twelve-month follow-up findings from Partners in Care, a groundbreaking randomized, controlled experimental trial designed by Wells and colleagues²⁰ that utilized care management by nurses within primary care settings, suggests that intensive care management can decrease depression and unemployment, at least for some racial/ethnic groups.²¹ Critical for the current study is the finding that these care management approaches appear to be most effective among the most disadvantaged populations.²² In addition, a growing number of studies indicate that telephone-based care management significantly reduces depression among outpatient populations, providing a cost-effective approach to care management.²³  Together, these effectiveness trials have evaluated re-designed systems for the management of depression that include: a) an information system to monitor adherence and outcomes; b) an expert system of consulting specialists or a computer support system; and c) patient materials for self-management support (either educational materials or psychoeducational interventions). Compared to usual care, these systematic interventions have led to improved clinical outcomes and some have demonstrated improvements in functional outcomes or disability measures as well.²⁴

Effects of early life stressors of onset and treatment of depression

There is mounting evidence that certain types of depression may be the outcome of certain gene-environment interactions. Exposure to stress during early development, especially on a chronic basis, is one way that the environment affects susceptibility to depression in those individuals that are particularly vulnerable.²⁵ Various laboratory studies have identified that the effects of early life stressors such as neglect or child abuse persist into adulthood.²⁶ Having an increased sensitization to stress may elevate the risk to develop depression or experience relapse.²⁷ Additionally, patients who suffer from chronic depression and experienced early life stressors may be unique in their response to treatment. Patients with a history of early life stressors have demonstrated better response to psychotherapy alone compared to treatment with antidepressants alone and that combination therapy was the most effective for this population.²⁸

A1.1b Effects of Maternal Depression on Children

A wealth of research has documented the negative effects of maternal depression for children’s development.²⁹  Early studies found that children of depressed parents were at similar levels of risk as children of parents experiencing other forms of psychopathology (e.g., schizophrenia).³⁰ Children of depressed parents show decrements in social behavior and psychological functioning, as well as affective disorders like depression.³¹  Other work has found that children of depressed parents show a more negative attributional style that results in a more negative self-concept.³²

A1.1c Effects of Interventions for Depressed Parents on Children

Controlled intervention studies on the effects of maternal depression on children. A careful review of the literature indicates that only two randomized controlled intervention programs have studied the experimental effects of programs on mothers’ depression and on their children³³ and only one with low-income families.³⁴  Although based on small samples, the results from these intervention efforts are encouraging.  Beardslee finds greater benefits using a clinician-facilitated condition relative to a lecture format for delivering critical information to children intended to improve family communication and increasing children’s understanding of parental depression.³⁵  Cicchetti finds improvements in cognitive outcomes for young children for an intervention aimed at improving parenting skills.³⁶ The family-focused nature of these interventions makes them more powerful in their likely effects on children.  At the same time, because these interventions are targeted at family processes more so than depression per se, they do not provide information critical to understanding whether changes in outcomes among children are linked with changes in parental depression.  

Varying effects by child age. There is evidence to support the hypothesis that the pattern and pathway of the effects of maternal depression on children may vary according to child age. Children’s development can be characterized by the achievement of successive stage-salient developmental tasks.³⁷  Across developmental periods, children may vary in their responses to changes in maternal depression because maternal depression may to a greater or lesser extent interfere with the attainment of stage-salient tasks in ways that affect later development. Moreover, because parents’ roles in children’s development change over the course of development, as the proximal and distal systems affecting development change over time,³⁸ the pathways by which depression is likely to affect children’s development may differ over the childhood age span.

While studies have found negative effects of depression for all stages of childhood, the effects appear to be most pronounced during particular stages of development.³⁹  Research on the effects of maternal depression on children have marked early childhood and adolescence as two periods in which maternal depression interferes with the stage-salient tasks most central to development.⁴⁰ Moreover, because transition periods are marked by qualitative reorganization of individuals’ organizing systems,⁴¹ transitions in development may represent sensitive periods.⁴² Recent findings suggest that developmental transition periods (in this case, the transition to middle childhood and the transition to adolescence) are most sensitive to welfare reform policy change.⁴³

The early childhood period has been the focus of considerable research on the negative effects of maternal depression.  This period is thought to be particularly sensitive to the negative effects of maternal depression for two reasons.  First, self-regulatory competencies develop during this period, and these competencies may be disrupted by affective changes in the mothers’ behavior brought about by maternal depression.⁴⁴  Second, early childhood is a period in which the child is primarily embedded in the family system and only indirectly affected by community and neighborhood contexts through interactions with their family.⁴⁵

For young children, depressed parents are thought to display noncontingent affective cues that result in difficulties in emotion regulation skills and maladaptive parent-child relationships.  Even short-term experimental manipulations that try to mimic the emotional unavailability of the depressed parents, by subjecting children to a “still face” condition on the part of the parent, result in increased gaze aversion and distress by infants of nondepressed parents.⁴⁶ Depressed parents are thought to have difficulty modeling emotions appropriately for their children, leading to impaired knowledge of emotions among children and difficulty in peer relations.⁴⁷ Moreover, maternal depression impedes the development of a secure attachment relationship, the basis for competent socio-emotional development in young children.⁴⁸ 

In addition to the early childhood period, the adolescent period may also be particularly sensitive to the negative effects of depression.  Research has found that maternal depression contributes to increased risk during adolescence in low-income families,⁴⁹ as well as depression among the adolescents themselves.⁵⁰ The tasks of gaining independence and developing identity central to adolescent development⁵¹ may be impeded by the lack of autonomy in the parent-child relationship characteristic of depressed parent-child dyads.⁵²  The negative cognitions and beliefs characteristic of parental depression⁵³ may color adolescents’ own views of their worlds, as well.

The transition to adolescence has been characterized by significant biological as well as socio-emotional growth,⁵⁴ and research has pointed to the challenges to self-system development of the combination of school and familial transitions, along with these pubertal and developmental changes.⁵⁵ Indeed, puberty itself has been implicated in the onset of depression, particularly among girls.⁵⁶ It may be that the hormones of puberty interact with any biological tendencies and social stresses, making children particularly vulnerable during this period.

Implications. The main purposes of the study are the following: (1) to determine whether a telephone care management model focused on low-income parents can be successfully implemented and, if so, (2) to determine whether the model is effective at alleviating depression, increasing employment and earnings, and reducing the use of public assistance. The study thus provides a unique opportunity to determine whether this relatively inexpensive type of outreach can be an effective model for state systems.

In addition, this evaluation will also address important gaps in the scientific knowledge about the effects of experimentally-induced changes in maternal depressive symptomatology on the psychosocial development of low-income children and youth.  Few, if any, studies have leveraged a random assignment research design in this way to examine these questions in the context of low-income families.   Lastly, the results of this study can help guide policy about effective options for low-income clinically depressed mothers and their young children and adolescents in the context of a new era of welfare policy.  Typically, welfare policy and mental health treatment are separate and distinct areas of research and policy.  Bringing the two together represents an innovation in policy and practice and has the potential to address prior shortcomings of policies for depressed parents on welfare.  In this way, the proposed research will inform our understanding of the impact of public policy, aimed at parents, and on youth development, within the context of increasing interest in children’s development in this newest phase of welfare reform.

A1.2 Overview of the HtE Evaluation – Rhode Island Site

This document requests OMB authorization for an amendment to data collection activities related to the HtE follow-up survey collected in Rhode Island 36-months after random assignment. This submission covers data captured through the following two surveys and direct child assessment. Each will be administered in person, and the adult follow-up survey will be paper and pencil with an interviewer present. The youth survey will be administered using Audio-CASI to ensure confidentiality of the youth’s responses. The interviewer will be present in order to provide any assistance to the youth.

These follow-up surveys should be seen as supplementing the automated administrative records data (such as TANF, Medicaid, and food stamp records and earnings records from state Unemployment Insurance (UI) data systems) that will continue to be collected over the course of the HtE evaluation.

• Rhode Island 36-Month Adult Survey: To supplement the administrative records data, respondents will be administered a follow up survey. The first half of it is referred to as the core portion of the survey. To assess changes in depression over time, we will use the Quick Inventory of Depressive Symptomatology – Self-Report (QIDS-SR), which assess depressive symptomatology concurrently (past 7 days), can be coded to construct a cut-point for “caseness” (i.e. of major depression), and allows for assessments of severity. This survey will also include assessments of the following important outcomes: health coverage; service and medication use; general health; early life experiences; asthma; work performance; and general child well-being.

Additionally, the second half of the survey, referred to as the child add-on portion, is administered with those female respondents who had children between the ages of 0-18 at random assignment, questions in the survey will be devoted to gathering additional, more specific, data on their children, including some data collected only on “focal” children of particular relevance to the intervention (e.g., social/emotional well-being). These focal children will belong in two different age groups, between the ages of approximately 0-5 or 8-14 at the time of random assignment. School outcomes are assessed only for school-aged children. Respondents will also be asked about aspects of their personal and family lives, particularly about their parenting, including questions regarding: parental activities with children (e.g., play, discipline, monitoring); parental involvement; parent-child relationship; family routines; household composition; child care history; and parental emotional well-being.

• Rhode Island 36- Month Youth Survey: Because part of this study aims to understand the impact of depression on child outcomes, it is important to interview the adolescents themselves. Relying solely on parental reports of child outcomes could bias our results; any effects observed using parental report data may reflect changes in parent perception of child behavior, rather than actual child behavior, since depression alters parents’ perceptions of themselves, their children, and their environments. Adolescents will be asked specific questions aimed to supplement the 36-Month Survey (child add-on portion), regarding: academic performance and outcomes; youth activities; parental involvement; parent-child relationship; autonomy granting; monitoring; social skills; self-esteem; mental health (e.g., depressive symptomatology, agency, and anxiety); risky behaviors; and adolescent health.

• Rhode Island 36- Month Child Direct Assessment: Direct child assessments of “focal” children will be administered to provide objective assessments of children’s well-being since no administrative data on developmental outcomes for children is available.⁵⁷ Children (ages 3-8) will be asked by an interviewer to perform several self-regulation tasks, which assess children’s working memory, motor control, impulsivity, and set shifting skills at the time of the assessment. These tasks might include 1) repeating a list of words in backwards order; 2) walking a line at various speeds; 3) a general waiting task; or 4) sorting cards along either color or shape categories. For these same children, we will include a cognitive assessment such as the portions of the Woodcock Johnson-III focused on broad reading and math skills or the Peabody Picture Vocabulary Test, Version Three (PPVT-III). Additionally, we may include a task which involves showing children different vignettes about peer-related hostility and conflict resolution.

Site selection. Rhode Island is one of four sites in the larger HtE evaluation. Please refer to the February 2005 HtE Follow-Up Surveys OMB Supporting Statement for a description of all sites.

Evaluation components. The evaluation of the HtE programs will include three components:

• A process and implementation analysis focusing on Working toward Wellness program operations and the challenges encountered;

• An impact analysis that studies whether the Working toward Wellness treatment affects parents’ depression, employment, earnings, income, welfare dependence, and child development outcomes (see below for more detail); and

• A cost study that compares the financial costs of the Working toward Wellness program.

Impact analysis. This analysis component will examine the net effects of the Working toward Wellness program approaches on economic outcomes, barriers to employment such as depression, family functioning, and child well-being. Impacts in the Working toward Wellness program will be evaluated using a random assignment design. The study will compare outcomes for individuals in the experimental group versus those in the control group, using data from administrative records follow-up for up to 36-months and from follow-up surveys. In Rhode Island, surveys will be administered at 6 months after random assignment, at 18 months after random assignment, and then again at 36-months after random assignment, for sub-samples of the full research sample. The 6- and 18-month surveys will provide an important source of data on service receipt for the program and control groups, as well as a measure of program impacts on such behavior as episodes of recovery/relapse, subsidized and unsubsidized employment, and other critical issues. The 36-month survey will allow for even longer-term assessments of these outcomes. Administrative records will be used for such outcomes as employment, earnings, and receipt of welfare and other benefits. Survey data will be used to study impacts on job characteristics (e.g., hourly wages, hours worked, fringe benefits, wage progression), total income, material hardships, persistence of barriers to employment, family well-being, parent-child relationships, and children’s academic, social, and behavioral outcomes. The study will also assess whether results differ for key subgroups of the HtE population (e.g., people with depression at baseline, those with multiple barriers). Follow-up survey data at 6-, 18-, and 36-months (for which OMB authorization is being sought) will play an important role in the impact study, as described in Section A2.

Evaluation schedule. Random assignment to research groups began in Rhode Island on November 2004 and ended in October 2006. A total of 507 families were recruited. Data and findings will be issued and shared over the course of the multi-year evaluation through:

• On-going and frequent feedback to sites,

• Early Assessment memo (completed in September 2005),

• Initial report on program start-up (to be published in late 2007),

• Interim reports (2009),

• A final report (2011),

• Periodic synthesis and other cross-site documents,

• A web-based community, and

• Public use files.

The next section provides more detail on the follow-up survey data collection activities currently requiring OMB authorization, and the role that the resulting data will play in the HtE evaluation.

A2. How, By Whom, and For What Purpose Are Data to be Used

Follow-up survey data collected at the Rhode Island site 36-months after random assignment will be used for a variety of purposes by MDRC and its expert consultants. These purposes include the following:

• To study the effects of a telephonic care management intervention on depression, employment, earnings, and use of public assistance;

• To collect data on a wider range of outcome measures than is available through automated administrative records in order to understand how individuals were affected by the telephonic care management intervention; job retention and job quality; educational attainment; interactions with care managers; household composition; childcare, health coverage and service use; health and other barriers to employment; and income;

• To study the effects of a telephonic care management intervention for parents’ depression on children’s behavioral, socioemotional, and verbal functioning, and presence of psychiatric disorders;

• To study the extent to which intervention effects on children’s development may be attributed to changes in maternal depressive symptomatology that result from the intervention;

• To study the effects of a telephonic care management intervention for depression on parental mediators of the effects on children—parents’ expression on emotion, their parenting behaviors, and relations with their children;

• To study the extent to which these effects on parents’ behavior may be attributed to changes in maternal depressive symptomatology; and

• To obtain participation information important to the evaluation’s cost component.

A2.1 The Overall Role of Follow-Up Survey Instruments in the HtE Evaluation

The 36-month Adult and Youth Surveys will yield important data not available through administrative records, providing information on, for example: depressive symptomatology, a range of health-related outcomes, parenting, social-emotional well-being of parents and children, academic achievement, family relationships, child care history. The surveys will be analyzed in conjunction with the administrative records data (such as TANF and Food Stamp payment records and earnings records from state Unemployment Insurance (UI) data systems) to understand the impacts on parent and child outcomes.

A2.2 The Role of Specific Follow-Up Survey Components

Whenever possible, the questions in the follow-up surveys were taken or adapted from existing instruments with national samples or from instruments used in other HHS evaluations, so that comparisons with national or other evaluation findings will be possible. Section A8 provides more information about instruments used in the development of survey questions. We will work with the survey firm, HumRRO – and if necessary with firms specializing in translation – to ensure that these surveys are translated for administration with non-English-speaking populations as needed.

A2.2a Rhode Island 36-Month Adult Survey

The core portion of the 36-month Adult Survey will be administered to all adults in the study and will ask many of the same items that respondents were asked in the 18-month survey. This survey is devoted to gathering additional, and more specific, data regarding health coverage, service use, and health for those experiencing major depression. These data enhance our ability to assess program effects over time. To assess changes in depression over time, we will use the Quick Inventory of Depressive Symptomatology – Self-Report (QIDS-SR), which assesses depressive symptomatlogy concurrently (past 7 days), can be coded to construct a cut-point for “caseness” (i.e., of major depression), and allows for assessments of severity.⁵⁸ These surveys will also include assessments of other important outcomes:

• Health coverage and service and medication use, general health: including questions about insurance status of respondents and family members, use of preventive and emergency health care, anti-depressants, self-rated general health, activities of daily living, and asthma symptoms;

• Work performance: including full and partial days missed at work due to physical or mental health problems and job performance; and

• Child well-being: including questions about all children (e.g., health, allergies and asthma, child care, school outcomes, activities, and problem behaviors).

Much like the 18-month survey, the 36-month survey contains brief series of questions on alcohol and recreational drug use, early life experiences, and a broader range of symptoms of physical and emotional health problems.

Since we are interested in understanding whether intervention for depression has any impact on child outcomes, information on children, as well as parents, needs to be collected at the 36-month follow up point. Since no administrative data on developmental outcomes for children is available, we rely on parents’ reports and direct assessments of child well-being.

The child add-on portion of the Rhode Island 36-month Adult Survey will only be administered to female respondents who had children between the ages of 0-18 at the time of random assignment. The survey will provide information about how children and parents are faring 36 months after random assignment. It includes more detailed sections – in relation to a common component on child well-being – on children’s social/emotional development, parenting, cognitive development, academic functioning, and health and safety.

Different outcomes are more relevant to certain age groups of children, and the greatest amount of data will be collected for “focal” children, for whom we expect strongest intervention effects. These focal children will belong in two different age groups, between the ages of approximately 0-5 or 8-14 at the time of random assignment. For both age groups, respondents will be asked to assess their children’s social/emotional well-being as well as their own. Respondents will also be asked questions regarding parenting. Parents will also be asked to assess their children’s social-emotional well-being using the Positive Behavior Scale (revised) and the Social Skills Rating System (externalizing/internalizing dimensions). Data assessing child health and safety will be collected about all children. Questions about child care history will only be asked of children who are between the ages of three and four at the 36-month follow-up, while after-school care will be asked of school-age children. Additionally for school-age children, school outcomes data are collected with questions to assess: academic functioning, participation in special education or gifted/talented programs, grade repetition, and school suspensions/expulsions.

The approximate administration time for the survey is estimated to be around 60 minutes, though it may be longer. Our budget requires that this survey be administrated in 60 minutes or less. In the formal pre-test conducted before fielding the survey and other assessment techniques, we will gain a more precise estimate of administration time and delete content if necessary. Therefore we base our calculations of respondent burden in Section A12 on a 60 minute parent survey for the Rhode Island 36-month follow-up. Should the survey instrument take longer than 60 minutes to administer, the research team has identified several items or sections in the following areas of inquiry as possible candidates for streamlining and cutting the survey: section J (parent psychological well-being & stress), section K (employment and educational activities), and if needed components of section BB (parenting).

A2.2b Rhode Island 36-Month Youth Survey

The Youth Survey will be devoted to gathering unbiased information regarding child outcomes of the older focal children who will be between the ages of 11-17 at the 36-month follow-up. This data will enhance our ability to assess program impacts on child outcomes. This survey will include assessments of the following important outcomes:

• Academics: including questions about school performance and outcomes, youth activities;

• Social Skills: including questions about social skills;

• Self-Esteem: including questions about agency and self-perception;

• Parent-Child Relationship: including questions about parental involvement, child’s evaluation of relationship to mother, monitoring, parental acceptance and psychological autonomy granting, connectedness and time spent together with mother;

• Mental Health: including questions about anxiety, loneliness and social dissatisfaction and depressive symptomatology;

• Risky Behavior: including questions about deviant behavior, alcohol and drug use;

• Health: including questions about asthmatic symptoms, body weight, pubertal development, sexual behavior and suicide.

The approximate administration time for the Youth Survey is estimated to be around 45 minutes. Regardless, we will conduct a formal pre-test before fielding the survey and other assessment techniques, and we will gain a more precise estimate of administration time and delete content if necessary. However, we do not anticipate going over the time limit because this Youth Survey is essentially the same as the Youth Survey collected at the 18-month follow-up which is currently taking approximately 30 minutes to administer in the field. Therefore, we do not anticipate needing to delete any content. We base our calculations of respondent burden in Section A12 on a 45-minute survey for the Rhode Island 36-month follow-up Youth Survey.

A2.2c Rhode Island 36-Month Direct Child Assessment

Direct child assessments of younger focal children will also be used at the 36-month follow-up to provide objective assessments of children’s well-being.⁵⁹ Children (ages 3-8) will be asked by an interviewer to perform several self-regulation tasks, which assess children’s working memory, motor control, impulsivity, and set shifting skills at the time of the assessment. These tasks might include 1) repeating a list of words in backwards order; 2) walking a line at various speeds; 3) a general waiting task; or 4) sorting cards along either color or shape categories. For these same children, we will include a cognitive assessment such as the portions of the Woodcock Johnson-III focused on broad reading and math skills or the Peabody Picture Vocabulary Test, Version Three (PPVT-III). Additionally, we may include a task which involves showing children different vignettes about peer-related hostility and conflict resolution.

The administration time of the direct child assessments vary with the age of the child (longer for the older children for whom we will be administering more assessments), however prior research suggests that the approximate administration time averaged across the differing ages of children is 30 minutes. Our budget requires that this portion of the data collection be administrated in 45 minutes or less.

In the formal pre-test conducted before fielding the survey and other assessment techniques, we will gain a more precise estimate of administration time and delete content if necessary. However, we will pull together a battery of assessments similar in length to what was administered in the 18-month follow-up, which is currently averaging at 37 minutes in the field. Therefore, we do not anticipate needing to delete any content. We base our calculations of respondent burden in Section A12 on a 45-minute direct child assessment for the Rhode Island 36-month follow-up.

A3. Use of Information Technology for Data Collection to Reduce Respondent Burden

The Youth Survey will be administered using Audio-CASI since the survey contains sensitive content and the survey will typically be administered in the respondent’s home. Other non-technology efforts to reduce burden include training interviewers extensively, sections in the survey with lead questions to enable skip patterns, and limiting the questions to the age groups of children most likely to be affected by the intervention, with a maximum of two children per family.

A4. Efforts to Identify Duplication

The surveys focus on information that cannot be found in administrative records or other existing sources. They will facilitate the collection of data on, for example, general health and mental health, household composition, job retention and job quality, educational attainment, parenting, parent and child socio-emotional well-being, children’s health and behavior problems, and other child outcomes, and these types of information are not available routinely or systematically in program records.

A4.1 Reasons Why Available Information Cannot Be Used

Comparable information from other sources does not exist for the variables covered in the 36-month surveys for the populations included in this HtE evaluation site. MDRC will use administrative data as the primary source for earnings, TANF payments, and Food Stamp payments. However, administrative data are not available for all the other outcomes described earlier.

A5. Burden on Small Business

Does not apply. All respondents are individuals.

A6. Consequences to Federal Program or Policy Activities if Data Collection is not Conducted

If the survey data are not collected, we will not be able to adequately evaluate the long-term impact of the Rhode Island site. The analysis of the short- and long-term impacts of the telephonic care management intervention would be limited because changes in many important outcomes cannot be captured in administrative records data such as depression or substance use, parenting, and child well-being. Therefore, having a 36-month follow-up is a logical addition to a 18-month follow-up. In addition, this data has never been collected in a randomized experiment and various fields would benefit from this analysis.

If the data in this submission are not collected, program operators and policy makers will get less information on whether particular strategies for assisting the HtE lead to impacts on depression, job attainment and retention, parenting and child well-being, and whether there is a long-term effects of this kind of intervention. Surveys are the only way of obtaining these data and are required in order to fully understand the treatment. Information on outcomes for children is an important element of the cost analysis, as well.

A7. Special Data Collection Circumstances

No such circumstances.

A8. Form 5 CFR 1320.8(d) and Consultations Prior to OMB Submission

The 60-day Federal Register notice soliciting comments for the HtE follow-up survey instruments was submitted by MDRC to HHS and posted in the Federal Register, Volume 72, Number 117, pages 33761-33762 on June 19, 2007. A copy of the published 60-day Federal Register notice and a draft of the 30-day Federal Register notice are located in Appendix B.1 and B.2.

Although this survey represents efforts to break new ground in assessing programs specifically designed to assist the depressed hard-to-employ population, it inevitably builds on previous survey research. We have consequently developed instruments that incorporate items and scales from other major studies. To the extent possible, the questions included in the follow-up survey instruments allow for useful comparisons between the data from this project and that from other large-scale surveys.

Surveys previously fielded by MDRC provided a natural starting place for the development of these instruments. However, because the HtE Rhode Island intervention targets a population that disproportionately struggles with social and health issues, many items selected for inclusion have been developed in various other fields of study, namely those pertaining more directly to health and social behavior (e.g., developmental psychology, health services research, psychiatric epidemiology). Some questions were included exactly as they were in previous surveys, while others were modified to reflect the goals of the HtE initiative as fully as possible, and also to reflect the population’s low literacy and comprehension skills.

Instruments that were used in the development of survey questions are as follows:

• MDRC surveys, including those used in the following projects: The Employment, Retention, and Advancement (ERA) project; the Project on Devolution and Urban Change; New Hope 8-Year Follow Up Study; and the Self-Sufficiency Project (SSP);

• Surveys used in evaluations done by Harvard Medical School and Group Health Cooperative (Workforce Depression Study);

• Surveys used in evaluations done by NYU Center for Research on Culture, Development and Education (Project RAP);

• Surveys done in connection with child development studies, including the Infant Health Development Program, the Early Childhood Longitudinal Study (ECLS), the 2005 State and Local Youth Risk Behavior Survey, the Prenatal/Early Infancy Project, Abstinence Study, and the Adult Literacy Study;

• National surveys, i.e., the National Longitudinal Study of Adolescent Health;

• Early Trauma Inventory – Adult Version

• Baseline survey used in Allies Against Asthma;

• Home Environment Checklist from the Healthy Homes-II Study.

To select these measures for the various components of the survey, we consulted with a number of individuals outside MDRC who provided insights on specific issues, including: Greg Simon and Evette Ludman of Group Health Cooperative; Francisca Azocar of United Behavioral Health; Sheldon Danziger of the University of Michigan; Ron Kessler of Harvard Medical School; Jeanne Miranda of UCLA; Aletha Huston of University of Texas at Austin; Adrian Angold of Duke University; and Robin Weersing of the Yale Child Study Center. We also wish to remind readers that in all of the work on which we have drawn to build this survey, we have worked, and continue to work, with many leaders in the social policy research field, including people working in academic, government and nonprofit settings. This long tradition of collaborative work will certainly influence the refinement, implementation and analysis of this survey.

A9. Justification for Respondent Payments

We propose to provide small incentive payments to respondents who complete each of the follow-up surveys. The purpose of these payments is to improve response rates by decreasing the number of refusals, enhancing respondent retention, and providing a gesture of goodwill to acknowledge respondent burdens. Although many of the techniques suggested by OMB to improve response rates have been incorporated into our survey effort and are described in Section B3 (i.e., use of a survey contractor, specifying that data is not individually identifiable, increased fielding time, interviewer training, follow-up contact after initial mailing), it has been our experience that small monetary incentives are useful when surveying hard-to-employ populations as part of a complex study design. In the following section, prior research on respondent payments is discussed, and subsequently we discuss the rationale for respondent payments for these HtE surveys.

A9.1 Prior Research on Respondent Incentives

We are aiming to achieve an 80 percent survey completion rate for each follow-up survey. Even with the best survey practice, it would be very difficult, if not impossible, to obtain such a high completion rate without incentives. Prior research and survey experience indicate that incentive payments are warranted when "the task given to respondents is more complex and makes greater demands than is usually considered."⁶⁰ According to Groves’ evaluation of the use of incentives, "the larger the burden the larger the effect [on response rates] of incentives." ⁶¹

More than two decades of survey research and experience overwhelmingly support the benefits of offering incentives. Hazard, citing evidence from a 1974 study by Ferber and Sudman found that the effects of incentives are contingent upon respondent burden (i.e., the effort needed to cooperate), the amount of the incentive, and the economic level of the respondent.⁶² A study by Berlin, et al. found that incentives increased the response rates of respondents with low levels of literacy, as well as lowering interviewer costs.⁶³ James also found that an incentive was effective in lowering non-response rates and that any incentive lowered the number of interviewer visits per case.⁶⁴ The Mack et al. study of responders to the Survey of Income and Program Participation (SIPP) found that incentives reduced non-response rates in initial and subsequent interviews, and were particularly effective in reducing non-response rates in poor and African-American households.⁶⁵ Moreover, the use of incentives has been found to be efficacious for increasing the response rates of in-home and sensitive subject matter surveys.⁶⁶

The best statement of current thought on incentives is the Symposium on Providing Incentives to Survey Respondents convened in October 1992 by the Council of Professional Associations on Federal Statistics (COPAFS) for OMB. COPAFS asked Richard Kulka of NORC to write a review of the literature in light of what was learned at the symposium. Kulka concluded, “the greatest potential effectiveness of monetary incentives appears to be in surveys that place unusual demands upon the respondent, require continued cooperation over an extended period of time, or when the positive forces on respondents to cooperate are fairly low.” Kulka also wrote, “there is evidence that increasing the size of a monetary incentive will result in increases in survey response and/or response quality, although there is also consistent evidence that this benefit may rather quickly reach 'diminishing returns', whereby large incentives no longer result in appreciable increases in survey response.”⁶⁷

Experience with surveys of economically disadvantaged and TANF-receiving populations also supports the evidence that incentives increase response rates. For example, in a follow-up interview with Jobs Corps applicants, experimental evidence showed that incentives increased response rates and greatly increased search efficacy. Experience in these and similar studies of disadvantaged populations suggest that incentives can help convince respondents to participate.⁶⁸

A9.2 The Use of Incentives in the HtE Rhode Island Site Follow-up Surveys

Each instrument that will be used to collect follow-up data from HtE Rhode Island sample members has unique aspects that make administration difficult and threaten response rates. We are therefore requesting clearance to offer a small gift to all sample members who complete each survey. Aspects of the survey effort that also make it more difficult to obtain high completion rates are:

• The surveys include questions that could be perceived as intrusive and therefore could make respondents uncomfortable (i.e., questions about their mental health, employment, income, pubertal development, drug and alcohol use, and sexual experiences).

• The subject matter of the survey is not intrinsically interesting to respondents. Moreover, many participants may have negative feelings about the programs of interest, Medicaid, etc.

• Other difficulties in administering the surveys come from the population itself. Educationally and economically disadvantaged groups, such as those in the HtE Rhode Island sample, have been found to be more difficult than the general population to convince to participate in surveys.

These difficulties interact to make this survey of HtE Rhode Island sample members much more difficult to conduct than surveys of the general population.

Thus, we are requesting clearance to offer small incentives to those who complete the surveys to obtain response rates that will yield credible results, to avoid the bias that could result from selective non-response, and to reduce item non-response. We believe that the studies summarized here, and MDRC’s previous experiences with surveys of welfare recipients and other disadvantaged populations, make a strong case for the use of gifts or respondent payments for completing these surveys.

To be effective, the amount of the incentives must fit the burden of the survey. We have based the amount to be paid to HtE respondents on prior research, and MDRC’s and the survey firm’s prior experience interviewing similar populations. The following incentive amounts were previously approved by OMB and used at the 18-month follow-up point. We propose that the same monetary amount be used for the 36-month follow-up and include $20 for the core portion of the adult survey and $30 for the child add-on, a gift card for the youth survey valued at $20, and a toy for the younger focal children. These amounts reflect current practice in surveys using similar instruments. These incentives may also take forms other than a cash payment, such as a transportation voucher or telephone calling card for the given value.

A10. Confidentiality

MDRC will follow procedures for assuring and maintaining confidentiality that are consistent with the provisions of the Privacy Act and with ethical guidelines of professional organizations. Respondents will receive information about confidentiality protections at the outset of the interviews. They will be informed that all of the information they provide will be kept strictly confidential and that study results will be presented only in aggregate form. (They will also be told that completion of the survey is voluntary and that they may choose not to answer any question.)

The following safeguards are routinely employed by MDRC regarding confidentiality assurances:

• All staff who have access to data at MDRC and the survey subcontractor firm sign an agreement to abide by corporate policies on data security and confidentiality. This agreement affirms each individual's understanding of the importance of maintaining data security and confidentiality and abiding by procedures that implement these policies.

• All data, both paper files and computerized files, are kept in secure areas. Paper files are stored in locked storage areas with limited access on a need-to-know basis. Computerized files are managed via password control systems to restrict access as well as physically secure the source files.

• Merged data sources have identification data stripped from the individual records or encoded to preclude identification of individuals.

• All reports, tables, and printed materials present aggregate numbers only.

• Compilations of individualized data are not provided to participating agencies.

• Confidentiality agreements are executed with any participating research subcontractors, partners, and consultants who obtain access to data files.

Specific confidentiality procedures for the follow-up survey instruments are described below.

MDRC and the HumRRO survey firm will maintain in-house records of names, addresses, Social Security numbers, and tracing information for all sample members. This information will not be attached to survey or assessment data or made available to anyone outside appropriate staff of MDRC and the survey firm. All records identifying respondents will be kept in locked storage at MDRC, and respondents will be identified solely by a code number. Any coding, data entry and analysis requiring identification of individuals or households will use code numbers only, and a secret password will be necessary to access the data file. No data will ever be reported in such a way that individuals can be identified.

The importance of maintaining confidentiality will be emphasized during interviewer training, and any interviewer who knows a respondent will not be permitted to interview him or her. All staff, including coders and computer programmers, will be required to sign a confidentiality pledge.

At the beginning of each interview, respondents will be informed of their rights. In addition, interviewers will attempt to conduct the interview at a time and place that allows the utmost privacy for respondents. In many cases this will be in private areas at the program sites, while in others it will be in respondents’ homes.

Additionally, a Certificate of Confidentiality was obtained from the National Institute of Child Health and Human Development (NICHD) for the full study on March 26, 2007 and expires on December 31, 2009, which covers the majority of the project. We will be making modifications to this Certificate to include all data collection components of the 36-month follow-up and therefore, our proposed consent forms include the statement that explains the Certificate of Confidentiality. We anticipate submitting our modifications in late November 2007 to Steven Hirschfeld, MD PhD, Associate Director for Clinical Research at NICHD. Our last submission took less than a month to be processed so we are confident that we will obtain approval for our modification in time to begin fielding in spring 2008.

Our proposed consent forms request permission to collect the child add-on portion of the adult survey with parents and to give permission to do the youth survey and direct child assessments (if applicable) with their children. Respondents gave permission for follow-up surveys when they were recruited at baseline. Adolescents are asked to provide assent in order to conduct the youth survey.

A11. Questions of a Sensitive Nature

Questions in all components of the HtE follow-up surveys are potentially “sensitive” for respondents. Respondents are asked about highly personal topics, some even stigmatizing. The questions we have included were selected in part because they have been widely used in previous research and are respected among experts. Moreover, all will be pre-tested prior to the survey’s full implementation, and if problems arise in regard to any specific items, their inclusion will be reconsidered. Also, all survey forms will contain instructions that explain questions before they are posed. Finally, respondents will be informed by program staff prior to the start of the interview that their answers are confidential, that they may refuse to answer any question, that results will only be reported in the aggregate, and that their responses will not have any affect on any services or benefits they or their family members receive. As mentioned in Section A10, MDRC and its contracted survey firms employ numerous safeguarding procedures to ensure confidentiality.

A12. Estimates of the Hour Burden of Data Collection to Respondents

Participation in all the survey data collection activities is completely voluntary. No sanction or penalty will be applied to respondents receiving state or federal assistance who choose not to provide information. Respondent payments, as described in Section A9, will be offered to each sample member who participates in the survey.

The estimated response burden by instrument/component was calculated based on information on survey length obtained during the pretests (see Section B4). Assuming a response rate of 80%, the total number of respondents for Rhode Island Parent (400) and Youth surveys (298) and Direct Child Assessment (164) at 36-months were multiplied by the average length of the surveys/assessment, divided by 60, then summed to determine the total burden in number of hours. These estimates are based on current proportions of children in each group. The response burden breakdown for all instruments is shown in the table below.

In this submission we are seeking approval for the Rhode Island 36-month survey.

Instrument	Expected Number of Respondents	Number of Responses per Respondent	Average Burden per Response	Total Burden (Hours)
RI 36-month, parent survey	400	1	60 minutes or 1 hr	400
RI 36-month, youth survey	298	1	45 minutes or .75 hrs	223.50
RI 36-month, direct child assessment	164	1	45 minutes or .75 hrs	123
TOTAL PERSON HOURS				746.50

A13. Estimates of Capital, Operating, and Start-Up Costs to Respondents

Not applicable. The follow-up surveys will be conducted by a subcontracted survey firm.

A14. Estimates of Costs to Federal Government

The estimated cost for designing, administering, processing, and analyzing this 36-month follow-up data is $1,550,000. On a year-by-year basis, these expenses are estimated to be:

Year Cost

2008 $750,000

2009 $650,000

2010 $150,000

A15. Changes in Burden

This is a new collection.

A16. Tabulation, Analysis, and Publication Plans and Schedule

A16.1a Assessment of Data Quality and File Construction

These follow-up surveys will go through a rigorous series of tests for completeness and quality. Professional staff at the survey firms will review the initial cases completed by each interviewer as well as perform occasional spot checks after that. Editing/coding staff will review questionnaires for quality and consistency after this initial period. Interviewers will be apprised of any problems found and retrained if needed. During the coding of data, coder reliability checks will be undertaken repeatedly to verify that coding procedures are being followed correctly. Data entered into computer files will be assessed for missing information, outliers, and other data problems according to standard procedures. If necessary, questionnaires will be re‑coded. The survey firms will deliver to MDRC data sets of completed cases at agreed-upon internals, along with marginal frequencies. The data and frequencies will be reviewed for outliers, unusual distributions and inconsistencies between data items.

Data from the surveys will then be merged with data from other sources. That is, data collected through follow-up survey interviews will be combined with previously collected data, including that routinely collected by welfare departments and administrative records information relating to welfare receipt, earnings, and Medicaid eligibility and program tracking (when available) and data collected from earlier surveys (e.g., the Rhode Island 36-month data will be merged with the Rhode Island 6- and 18-month data).

A16.1b Data Analysis

As previously indicated, the HtE evaluation- Rhode Island Site will incorporate a random assignment analytic design. We offer a brief outline of how we will address the project’s long-term analytical goals, with a focus on how the follow-up survey data will be useful in that process.

Estimating Overall Impacts. Although the use of a randomized design will ensure that simple comparisons of experimental and control group means will yield unbiased estimates of program effects, the precision of the estimates will be enhanced by estimating multivariate regression models that control for factors that also affect the outcome measures. Such impacts are often referred to as “regression-adjusted” impacts. Examples of factors that may affect outcomes are the sample members’ age, number of children, prior employment, and baseline barriers to employment.

Most of the analyses of overall impacts will result in estimation models that, in their basic form, can be expressed as follows:

(1) Y_ij = F (T, X_ni, U_ij)

where

Y is a vector of outcomes (e.g., post RA employment, earnings, welfare receipt)

T is the treatment variable indicating whether the individual is a member of the HtE intervention group

X is a vector of baseline characteristics to be controlled (e.g., baseline education level)

U is a vector corresponding to the residual (error) term

i is the subscript designating the individuals in the sample

j is the subscript designating the various outcomes of interest

n is the subscript designating the various personal characteristics to be controlled.

Ultimately, a range of outcomes (Ys) will be examined. Many will be economic outcomes (e.g., employment, job stability, earnings, welfare receipt, and material hardships). Others are noneconomic outcomes that reflect progress regarding specific barriers to employment (e.g., depression, substance use, criminal activity), or child outcomes (e.g., children’s cognitive and emotional development).

Program/Control Group Differences in Child Outcomes. Child outcomes include data on academic achievement, social-emotional well-being, perceptions of parent-child relationship, depressive symptomatology, risky behaviors, and health.

For parts of the analysis, we will use individual survey items or pre-existing scales as child outcome measures or implementation measures. In some cases, however, we may create scales that include multiple items. In building these scales, we would use standard social science methodologies.⁶⁹ For example, the first step would be to identify the set of items in the survey that were intended to address the same broad topic, such as depressive symptomatology in children. We would then examine inter-item correlations for the full set of questions designed to measure this outcome and conduct a factor analysis to determine which items in the set “go together” and appear to be measuring the same underlying construct. Next, we would estimate Cronbach's alpha to assess the reliability of the scale. We would add and delete items as appropriate to maximize Cronbach's alpha. After selecting the final set of items for a given scale, we would then produce an overall scale score for each respondent by summing her scores on each of the items in the scale. The overall scale scores for all respondents would then be used as an outcome measure for the impact analysis, or for computing each evaluation site's ranking on an implementation measure, depending on the analysis. We have used this general approach successfully in several previous evaluations, especially the more recent evaluations with child outcomes data.⁷⁰

Subgroup analyses. There is considerable evidence that interventions for low-income populations are more effective for some subgroups than for others. For example, in MDRC’s New Hope study it was found that the program had large employment and earnings effects on people with just one barrier to employment, but little effect on those with two or more.⁷¹ Thus, it is expected that strategies for the hard-to-employ will be more effective for certain types of participants, so it is essential to go beyond the examination of overall impacts of the HtE programs to examine subgroup impacts.

The baseline survey will produce data on a diverse array of background characteristics and barriers to employment. These will be used to create subgroups to further our understanding of how baseline barriers influence economic outcomes and program impacts. We will construct a range of variables that can be used to create subgroups reflecting physical and mental health, familial caregiving obligation, and a host of characteristics on their children’s health and social behavior. In some instances, individual survey items will allow for the creation of analytical subgroups (e.g., self-rated health, familial caregiving). In others, multiple items or scales will be used to identify them (e.g., QIDS-SR for major depression). In such instances, we will rely on established criteria for determining cutoff points for subgroup identification. Additionally, subgroups will be created based on the respondents’ total number of initial barriers.

These non-economic measures, and the subgroups we will create from them, will vastly enrich the mental health study and the HtE evaluation as a whole, increasing the comprehensiveness of the information available for assessing the evaluation’s overall effects and its costs. Similarly, they promise to inform the implementation research, potentially illustrating, for example, some of the key ways in which programs successfully remove certain barriers.

An analysis of subgroup impacts involves estimating the program’s effects for each subgroup separately, using the regression-adjusted model mentioned earlier, and then comparing the impacts of the two (or more) subgroups. The standard errors of each of the impacts are used to assess whether the impacts are statistically significantly different from each other. Subgroup impacts estimated in this way are referred to as unconditional subgroup impacts, because they show the gross effect of a particular characteristic, such as major depression at baseline, on a program’s impacts.

As an example, earnings impacts in a program may be lower for individuals who were severely depressed at baseline, as compared with their counterparts who were less severely depressed. However, this difference may arise not because of the severity of depression per se, but because individuals who have suffered more severe bouts of depression are also less likely to have had recent work experience, which also affects how they benefit from the program. In this case, it would be of interest to estimate conditional subgroup impacts, or impacts by baseline severity of depression that also control for prior work experience. These impacts would be obtained by pooling the sample and estimating one impact model, in which severity of depression and prior work experience are interacted with all of the other variables in the model and with the program group dummy variable (T in the model). For example, if the coefficient on the interaction of program status and severity of depression is reduced in size once the interaction of program status and prior work experience is included, we can conclude that some part of the effect of severity of depression on the program’s impacts is due to its correlation with prior work experience.

Exhibit B1-2 (presented in Section B1), shows detectable effects for various sample sizes. This table indicates the size of impacts that would be needed to be detected at significant levels when the sample is split into subgroups of various sizes. This information will guide our analyses of subgroups.

Additionally, experimental analysis will be conducted to examine the moderating role of child age on the relation between the intervention and outcomes for children.  Differences in subgroup impacts will be tested by conducting split sample regression analyses and estimating differences using an HT statistic.  The HT statistic is the weighted sum of squares of the impact estimates for the subgroups and has a chi-squared distribution.⁷²  Unlike more standard interaction terms, split sample approaches have the advantage of not assuming homogeneity of variance across the subgroups examined.  Other moderating effects to be tested include site and parent characteristics, and child gender. Differences in impacts on parental characteristics may help to tease out differences in effects of the intervention on children of parents who respond easily to treatment, as compared to parents whose depression relapses, and parents whose depression is unresponsive to the intervention, while still maintaining the strength of the experimental design. That is, we can compare impacts across parental subgroups and investigate patterns of impacts on both parental depression outcomes and child outcomes to determine the extent to which differing impacts on child outcomes appear related to differing impacts on parents’ outcomes.

Non-experimental analyses. Non-experimental analyses can help complement the estimation of HtE’s impacts. These analyses can test whether experimentally induced changes in depression can be linked statistically with changes in children’s outcomes.⁷³

A16.2 Publication Plans and Schedule

In Rhode Island, surveys have been administered at 6 and 18-months post-random assignment. For the latest survey effort, fielding will occur 36-months after each cohort was randomly assigned, beginning as early as January 2008, pending OMB approval, and ending as late as February 2010.

Findings from the follow-up survey instruments will be part of the impact, implementation, and cost analyses. The results will be published in a series of reports based on the results of the implementation, impacts, and cost analyses. The reports will be produced during 2009 and 2011, as outlined in section A1.2.

A17. Reasons for Not Displaying the OMB Approval Expiration Date

Not applicable. We intend to display the OMB approval number and expiration data on all survey materials.

A18. Exceptions to Certification Statement

Not applicable. We have no exceptions to the Certification Statement.

1Corcoran, Danziger, and Tolman (2003).

2Danziger et al. (1999).

3Katzelnick et al. (2000).

4Mintz, Mintz, Arruda, and Hwang (1992).

5 See Danziger et al., 2000; Polit, London, & Martinez, 2001; for data on teen parents, see Quint, Bos & Polit, 1997.

6 Kessler et al., 2003

7Belsher and Costello (1988).

8The model has been adapted for the WtW intervention, given the considerably different target population. Outreach and Treatment for Depression in the Labor Force is focused on active employees of large corporations, whereas WtW is focused on nondisabled Medicaid recipients. Nonetheless, both are based on telephonic outreach and care management for depression offered by master’s level clinicians.

9 Hunkeler et al., 2000; Katzelnick et al., 2000; Simon, VonKorff, Rutter, & Wagner, 2000; Wells et al., 2000

10 From the Department of Health and Human Services RFP No.: 233-01-0012. This RFP also included an option of an additional four sites, for a total of up to 10 sites, but this option has not been exercised by HHS.

11 The authorization to conduct this evaluation and data collection is Section 1110 of the 1993 Social Security Act (see Appendix C).

12 Knitzer, 2000

13 Danziger et al., 2000; Miranda et al., 2003; Polit, London, & Martinez, 2001

14 Quint, Bos, & Polit, 1997

15 Danziger et al., 2000; Kessler & Frank, 1997

16 Michalopoulos & Schwartz, 2001

17 Danziger et al., 2000

18 Mintz et al., 1992; Ormel et al., 1990, 1993; Von Korff et al., 1992

19 Keller et al., 2000

20 Wells et al., 2000

21 Miranda et al., 2003; Wells et al., 2000

22 Araya et al., 2003; Miranda, et al., 2003, Smith et al., 2002

23 Hunkeler et al., 2000; Katzelnick et al., 2000; Simon, VonKorff, Rutter, & Wagner, 2000

24 Katzelnick et al., 2000; Unutzer et al., 2003; Wells et al., 2000

25 Gillepsie & Nemeroff, 2007

26 Gutman & Nemeroff, 2003

27 Gillepsie & Nemeroff, 2007

28 Nemeroff, et al. 2003

29 Beardslee, Versage, & Gladstone, 1998; Cicchetti, Rogosch & Toth, 1998; Downey & Coyne, 1990

30 Downey & Coyne, 1990

31 see Cummings & Davies, 1994; Downey & Coyne, 1990; Goodman & Gotlib, 1999, 2002 for reviews

32 Hammen, 1988; Hammen, Adrian, & Hiroto, 1988

33 see Beardslee et al., 1997; Cicchetti, Rogosch, & Toth, 2000

34 Cicchetti et al., 2000

35 Beardslee et al., 1997

36 Cicchetti et al., 2000

37 Sroufe, 1979; Waters & Sroufe, 1983

38 Bronfenbrenner & Morris, 1998

39 Radke-Yarrow & Klimes-Dougan, 2002

40 Beardslee, 1986; Gelfand & Teti, 1990

41 Cicchetti, 1991; Sroufe, 1990

42 Graber & Brooks-Gunn, 1996

43 Morris, Duncan, & Clark-Kauffman, 2003

44 Eisenberg & Fabes, 1992

45 Bronfenbrenner & Morris, 1998

46 Field, 1994

47 Field, 1994; Goodman & Gotlib, 1999

48 see van Izjendoorn, Schuengel, & Bakermans-Kranenburg, 1999; Lyons-Ruth, Lyubchik, Wolfe, & Bronfman, 2002 for review

49 McLoyd, Jayartne, Ceballo & Borquez, 1994

50 see Beardslee et al., 1998 for a review

51 Erickson, 1963

52 Radke-Yarrow & Klimes-Dougan, 2002

53 Barnett & Gotlib, 1988

54 Brooks-Gunn & Petersen, 1983; Hamburg, 1974

55 Seidman, Allen, Aber, Mitchell, & Feinman, 1994; Simmons & Blythe, 1987

56 Angold, Costello, & Worthman, 1998

57 Kochanska, Murry, & Harlan, 2000; McCabe, Hernandez, Lara, Brooks-Gunn, 2000; Mather & Woodcock, 2001a; Mather & Woodcock, 2001b; McGrew & Woodcock, 2001; Reynell & Gruber, 1990

58 Rush, A.J., M.H. Trivedi, H.M. Ibrahim, T.J. Carmody, B. Arnow, D.N. Klein, J.C. Markowitz, P.T. Ninan, S. Kornstein, R. Manber, M.E. Thase, J.H. Kocsis and M.B. Keller, 2003

59 Kochanska, Murry, & Harlan, 2000; McCabe, Hernandez, Lara, Brooks-Gunn, 2000; Mather & Woodcock, 2001a; Mather & Woodcock, 2001b; McGrew & Woodcock, 2001; Reynell & Gruber, 1990.

60 Cannell and Henson, 1974.

61 Groves, 1989. Other major research on incentives includes a study by Miller, Kennedy, and Bryant (1972) of the 1971 Health and Nutrition Examination Survey -- in which offering a financial incentive increased the response rate from 70% to 82% -- and Chromy and Horvitz (1978) which also found that the use of an incentive payment increased the response rate.

62 Hazard, 2002.

63 Berlin et al., 1992.

64 James, 1997.

65 Mack et al., 1998.

66 Hazard, 2002.

67 Kulka, 1992.

68 Moffit, 2004.

69For a discussion of these methods, see DeVellis, R.F., 1991

70 See Gennetian, L., and C. Miller, 2000.

71 Bos et al., 1999.

72 Cooper & Hedges, 1994; Greenberg, Meyer, & Wiseman, 1993

73 Gennetian, Morris, Bos & Bloom, 2005