Evaluation of Child Care Subsidy Strategies
Evaluation of Child Care Subsidy Strategies; Massachusetts, Illinois, and Washington
orginial omb packet3 (2)final
Evaluation of Child Care Subsidy Strategies
OMB: 0970-0306
Evaluation of Child Care Subsidy Strategies
Request for OMB Clearance: Massachusetts, Illinois, and Washington
June 2006
Prepared for
Ivelisse Martinez-Beck
Child Care Bureau
Administration for Children and Families, HHS
1250 Maryland Avenue, SW
Portals Building, Suite 800
Washington, DC 20024
Prepared by
Ann Collins
Charles Michalopoulos
Jean I. Layzer
Table of Contents
Summary i
Part A Justification 1
A1 Explanation of the Circumstances That Make the Collection of Information Necessary 1
a.Massachusetts……………………………………...…………………………………...3
b. Illinois……….…..………………………………………………..……………...…….3
c. Washington……….………………………………………………...………………….5
A2 How the Information Will Be Used, by Whom, and for What Purpose 6
a. Massachusetts………………………………………………………………………….6
b. Illinois………………………………………………………………………………...10
c. Washington…………………………………………………………………………...15
A3 Use of Improved Technology to Reduce Burden……………………………………….23
A4 Efforts to Avoid Duplication 24
A5 Efforts to Minimize Burden on Small Businesses or Other Small Entities 24
A6 Consequences if the Information is Not Collected 24
A7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5 (D) (2) 26
A8 Efforts to Consult with Persons Outside the Agency 26
A9 Payments to Respondents 27
A10 Assurances of Confidentiality Provided to Respondents 28
A11 Justification for Questions of a Sensitive Nature 30
A12 Estimates of Respondent Burden 30
A13 Estimates of the Cost Burden to Respondents 33
A14 Estimates of the Cost to the Federal Government 34
A 15 Reasons for Any Program Changes or Adjustments 35
A 16 Plans for Tabulation, Statistical Analysis and Publication 35
a. Massachusetts 35
b. Illinois………………………………………………………………………………..38
c. Washington…………………………………………………………………………..47
A17 Display of Expiration for OMB Approval 53
A18 Exception to the Certification Statement Identified in Item 19.0 of Form OMB 83-I 53
Part B Collection of Information Using Statistical Methods 54
B1 Sample Universe, Sampling Method and Expected Response Rates 54
a. Massachusetts 54
b. Illinois 55
c. Washington…………………………………………………………………………...56
B2 Data Collection Strategy 59
a. Massachusetts 59
b. Illinois 61
c. Washington 64
B3 Methods to Maximize Response Rates 68
B4 Tests of Procedures 71
B5 Individuals Consulted on the Statistical Aspects of the Design 71
References……………………………………………………………………………………………73
Summary
The Evaluation of Child Care Subsidy Strategies is designed to provide Federal, state, and local policymakers with information about the role of subsidy programs and policies in helping low-income families obtain and retain work and in improving outcomes for children. The goal of the study is to determine how differences in certain aspects of child care subsidy policies or quality-improvement efforts are related to outcomes for parents, children, and child care providers. The Evaluation of Child Care Subsidy Strategies consists of three studies, one in Massachusetts, one in Illinois, and one in Washington. OMB clearance is sought for each study site.
a. Massachusetts
The Massachusetts experiment will test the effects of a curriculum designed to address language development and the development of pre-literacy skills for very young children in family child care settings. Study participants are approximately 350 family child care providers who belong to one of 16 state-supported family child care networks and care for subsidized children. Each of these providers has approximately two children under the age of three in their homes. Half of the providers (i.e., the treatment group) will be randomly assigned to use Learningames, a research-based early childhood curriculum that can be easily adapted for use in family child care homes. These providers will be supported by network home visitors who are trained in the Learningames approach. The other half of the providers (i.e., the control group) will continue to offer care in their usual manner and be supported by the network’s standard training and technical assistance.
The major research questions include:
What is the effect of a research-based developmental curriculum (Learningames) on provider’s behavior and interactions with children on the language and literacy environment of the home?
What is the impact of the intervention on children’s language and pre-literacy skills?
The study will include an implementation analysis and an impact analysis. Data sources include observations of the family child care setting, at baseline and three other points in time, assessments of children taken at two points in time, questionnaires for providers at two points in time, and a one-time questionnaire for home visitors. Providers began using Learningames in the fall of 2005. The intervention and evaluation will be completed in the fall of 2007. Reports will be issued in 2008.
b. Illinois
The Illinois experiment will test the effects of subsidies on parental employment and selection of child care for families with incomes between 50% and 65% of State Median Income (SMI). Such families currently have incomes slightly over the state’s income eligibility ceiling. Approximately 2,000 families living in Cook County will be in the study; half will receive subsidies (i.e., the treatment group) and the other half will remain ineligible for subsidies (i.e., the control group). Families in the treatment group will be eligible for subsidies for a two-year period. These families will be randomly assigned to a six-month certification period, at which point they must reapply for subsidies in order to continue receiving them, or a 12-month certification period.
The major research questions are:
What is the impact of receiving a child care subsidy on parental employment, family income, and the receipt of public benefits?
What is the impact of receiving a subsidy on the type of care chosen, the stability of the arrangement and on parents’ satisfaction with the child care?
Does the length of the certification period for subsidies affect how long families receive a subsidy?
The study will include process analysis, impact analysis, and cost-benefit analysis. Data sources include (1) a 35-minute telephone interview of parents, conducted three times, (2) administrative records from the child care subsidies, other public benefits, and unemployment insurance programs, (3) 30-minute interviews, asking the same questions to fewer than nine state and local staff to understand contextual issues for the process analysis, and (4) policy manuals and other documents. Recruitment for the study began in the spring of 2005 and will be concluded in the spring of 2006. The intervention and evaluation will be completed in the spring of 2008. Reports will be issued in 2008 and early 2009.
c. Washington
The Washington state experiment focuses on parental co-payments for families receiving child care subsidies. In the fall of 2005, approximately 5,150 families who applied or reapplied for subsidies throughout the state during a three-week time period were assigned to either the state’s standard co-payment scale or an alternative scale. The alternative scale requires either the same or a lower co-payment amount for families at every income level. The intervention will continue for two years. If families leave the subsidy system and return to it, they will still be assigned co-payments according to the standard co-payment scale (if they are in the control group) or the alternative scale (if they are in the treatment group).
The major research questions are:
What is the effect of the alternative co-payment scale on parental employment, family income, length of receipt of subsidies and the receipt of other public benefits?
What is the effect of the alternative scale on the type of child care chosen, the stability of the arrangement, and on parents’ satisfaction with child care?
Is the effect of the alternative co-payment amount on employment and child care outcomes different for families who are new subsidy recipients than for families who are already receiving a subsidy and wish to continue?
Does the impact of the alternative co-payment differ for families at different income levels at the beginning of the study?
The study will include process analysis, impact analysis, and cost-benefit analysis. Data sources include (1) a 35-minute telephone interview of parents, conducted three times, (2) administrative records from the child care subsidies, other public benefits, and unemployment insurance programs, (3) 30-minute interviews with state and local staff to understand contextual issues for the process analysis, and (4) policy manuals and other documents. The intervention and evaluation will be completed in the fall of 2007. Reports will be issued in 2008.
P
art
A
Justification
A1 Explanation of the Circumstances That Make the Collection of Information Necessary
Investment in child care by the Federal government and by individual states increased substantially in the years after the passage of the Personal Responsibility and Work Opportunity Reconciliation Act. The legislation provided greatly increased Federal resources to states to provide child care assistance, authorizing some funds specifically for child care and also allowing states to transfer Temporary Assistance for Needy Families (TANF) funds to the Federal child care program (and to spend TANF funds directly on child care subsidies). The Child Care and Development Fund (CCDF), created as part of the legislation, combined four of the existing child care funding programs into a single block grant to states, giving them much more flexibility to decide how child care funds should be expended. In FY 2004, Federal and state spending on child care totaled $9.4 billion: Federal CCDF funding (including TANF funds transferred into CCDF) reached $6.9 billion; state spending totaled $2.5 billion. In addition, direct TANF spending on child care was $1.4 billion. As a result of increased Federal funding combined with steady increases in the states’ contributions to the subsidy program, many more low-income families with working parents are able to receive help in paying for child care. In addition, the CCDF stipulates that states must set aside 4% of their CCDF funding for efforts to expand the supply or improve the quality of child care; many states allocate more than this minimum amount. States face a considerable challenge in trying to use their child care funds as effectively as possible, both to support parent’s employment and to improve child care quality to ensure children’s safety and enhance their development.
In related efforts, states are working to meet the goals of President Bush’s initiative, Good Start, Grow Smart, to enhance the school readiness of young children. Universal pre-kindergarten (UPK) is one of the strategies that many states use to meet the presidential mandate. Typically, states are implementing UPK through the existing system of schools and child care centers. However, much of the care for young children, especially infants and toddlers, is provided in family child care homes. States can and do use the CCDF quality-set aside funds to attempt to improve the quality of family child care, but they do so with scant information about the effectiveness of their efforts. The majority of the research available about efforts to enhance the school readiness of children from low-income families focuses on center-based early childhood programs that serve primarily preschool-age children. A research focus on center-based programs for three- and four-year old children does not reflect the widespread recognition that the very early years are also a critical period of development. A focus on the school readiness of children in center and pre-school classrooms leaves out the many children who are cared for in family child care, before they reach the age where they may be in settings supported by UPK. The work of Hart and Risley (1996), among others, suggests that efforts need to be made early in children’s lives to enrich their language environments because this are of development is key to school readiness and later success in school.
Furthermore, despite the substantial increases in funding for subsidies over the last seven years, more recently some states have faced budget constraints that have had an impact on the subsidy program. Regardless of their fiscal situation, states must still make choices about how to allocate resources and target subsidies to meet multiple objectives. They do so directly and indirectly through a host of decisions about child care policies and their implementation. These include: determining the level of state resources and matching requirements for counties (where applicable); setting eligibility guidelines and setting priorities for subsidies (including priorities attached to serving TANF vs. non-TANF families); deciding how and to what extent the availability of subsidies will be publicized; developing co-payment scales; and developing fee schedules and payments for providers. None of the child care research conducted over the past three decades has systematically examined the effectiveness of different child care subsidy policies or programs.
Recognizing the need for carefully-designed research that would provide useful information to states and communities, the Child Care Bureau and the Office for Planning, Research and Evaluation of the Administration for Children and Families (ACF) within the US Department of Health and Human Services commissioned a research effort designed to expand our knowledge about child care subsidies and quality-improvement efforts. In September 2001, a contract was awarded to Abt Associates Inc. to conduct a multi-site, multi-year Evaluation of Child Care Subsidy Strategies. To carry out the study, Abt Associates and its subcontractors—Manpower Demonstration Research Corporation (MDRC), Columbia University’s National Center for Children in Poverty (NCCP), and Moore & Associates, Inc.— worked closely with state and local partners in four sites to design and implement experimental studies that are tailored to their needs and interests, as well as the interests of policymakers in general.
The Evaluation of Child Care Subsidy Strategies will provide Federal, state, and local policymakers with information about the role of subsidy programs and policies in helping low-income families obtain and retain work and in improving outcomes for children. The goal of the study is to determine how differences in certain aspects of child care subsidy policies or quality-improvement efforts are related to outcomes for parents, children, and child care providers/caregivers. Outcomes of interest include the stability of parental employment and earnings, parent and child well-being (especially the development of children’s language development and literacy skills), availability of care, and child care quality. The study will address this goal through rigorous evaluation using a set of three random assignment experiments that will test aspects of subsidy policies or evaluate quality-improvement efforts. Three states and one locality have agreed to participate in the study—Illinois, Washington, Massachusetts, and Miami-Dade County in Florida. This request for OMB approval is for the experiments that will be conducted in Massachusetts, Illinois, and Washington.1
Authorization for the CCDF is part of the Personal Responsibility and Work Opportunity Reconciliation Act (PRWORA), Public Law 104-193 (42 USC 1305). Authorization for research related to the CCDF is part of the appropriations legislation for the Departments of Labor and Health and Human Services (PL 109-149).2
a. Massachusetts
In Massachusetts, many licensed family child care providers who accept subsidies for children in their care operate under the auspices of family child care networks (called “systems” by the state). There are more than 50 such networks in the state, varying greatly in size and geographic coverage. Networks in Massachusetts receive state funds to provide technical assistance to their member providers in part through regular visits by home visitors/mentors. These visits are intended to provide general support and information to providers to improve the quality of the care they offer. In 2003, the state developed family child care regulations that stipulated that all licensed family child care providers must use a developmental curriculum. However, little guidance has been offered about how to identify and select a curriculum, and state agency staff believes that the mandate has largely been ignored or overlooked, although mentors could, in principle, help providers do so. Further, little research, thus far, has focused on the linkages between the use of CCDF quality funds and state activities aimed at improving the quality of programs and of the care children experience.
To address the need for tested approaches that enhance children’s language development and pre-literacy skills, the Commonwealth of Massachusetts has identified a strong research-based curriculum—Learningames— that can easily be adapted for use in a family child care setting. The state is interested in experimentally testing the effectiveness of Learningames in family child care homes linked to family child care networks. The state views the proposed experimental test of Learningames as an opportunity to provide both system administrators and providers with evidence-based guidance on how to address the state mandate as well as complement their current efforts on a new UPK initiative, by focusing on younger children in family child care settings. The need for such a study is clear. In Massachusetts the most recent research on the quality of child care (Marshall, et al, 2001 and 2003) indicates that, on average, the quality of care in family child care is considerably lower that of center-based programs. Indeed, only 30% of family child care homes met accepted standards for good quality child care.
The evaluation in Massachusetts includes 350 child care providers who are members of 16 family child care networks. Each provider is caring for at least two children under the age of 26 months at the outset of the study. Half of the providers were assigned to receive training on the implementation of Learningames, materials and mentoring support; the remaining providers will continue to receive the usual ongoing training and support. Recruitment for the study began in the spring of 2005 and random assignment was completed by June 30th. Outcomes of interest are changes in caregiver behavior and changes in children’s language development and literacy skills. The data collection includes observations of the family child care home environment and child assessments. It will occur at three points in time: in June 2006 (after receiving OMB approval); in January 2007; and January/February 2008. In addition, a short caregiver survey will be administered twice; in June 2006 and January 2008. Baseline data will include observations of homes and extant child assessment data collected by family child care network staff. Finally, home visitors will be asked to complete a brief questionnaire once, in June 2006.
b. Illinois
The study in Illinois will consist of a random assignment evaluation to assess the effects of subsidies on parental employment and selection of child care for families with incomes between 50% and 65% of the state median income (SMI). Such families currently have incomes slightly over the state’s income eligibility ceiling. Approximately 2,000 families residing in Cook County will be in the study; half will receive subsidies (i.e., be in the treatment group) and the other half will continue to remain ineligible for subsidies (i.e., serve as the control group). Recruitment for the study began in March 2005 and is expected to occur over a 12-month period. The study has three components: an Implementation Study, an Impact Study, and Cost-Benefit Study. We are seeking clearance for a parent interview that will be used to inform the impact and cost-benefit studies (this interview will also be used in the Washington study). The survey will be conducted three times over the two-year period. Baseline data on study participants will be obtained through extant administrative records. The Implementation Study will rely on documents, researcher notes made during the planning and start-up period, and brief interviews conducted with fewer than nine people in the state.
The State of Illinois currently sets the income eligibility ceiling for initial and continuing receipt of subsidy at 50% of State Median Income (SMI), and is committed to serving all eligible families that apply. In 2003, all families receiving subsidies in Illinois had an average annual income of $10,744, and 50% of SMI was equivalent to an annual income of $21,819 (Anderson, Ramsburg, Rothbaum, 2003). The CCDBG eligibility limit is set at 85% of SMI and the average state subsidy eligibility ceiling is set at 62 % of SMI (Collins, et al., 2000), thus the new Illinois income limit of 65% is comparable to the state SMI average. Allowing a group of families between 50 and 65% of SMI to receive subsidies would allow us to examine the impact of child care subsidy receipt on employment, on achieving self-sufficiency, and on child care outcomes for low-income families currently not receiving TANF assistance, but who include former TANF recipients or families at risk for TANF receipt.
A study of the dynamics of child care subsidy receipt, conducted in Illinois and four other states, found that the median length of the first spell of subsidy receipt was six months in Illinois (Meyers et. al., 2002). Because most families currently are certified to receive subsidies for six months, the state is interested in knowing whether a longer certification period would increase the use of subsidies and the stability of care, as well as make subsidy administration more efficient. Results regarding the impact of changing recertification periods will serve to inform administrative decisions aiming to increase administrative efficiency and strengthen fiscal and administrative accountability.
The study will test the impact of subsidies on the employment outcomes, child care choice and stability, and child care satisfaction. The study group will be composed of two subgroups. Subgroup 1 will include those families who are new applicants to the subsidy system.3 For this subgroup, families who apply for subsidies with incomes over the current eligibility ceiling but under 65% of SMI will be randomly assigned to a treatment group (approved to receive subsidies) or a control group (not approved to receive subsidies). The second subgroup will be composed of families who were using subsidies, but, at re-determination, have incomes between the current eligibility ceiling and 65% of SMI. As with the first group, these families will be randomly assigned to a treatment group (recertified to receive subsidies) or a control group (not recertified to receive subsidies). With the subgroups, it will be possible to test the effect of subsidy receipt on families with incomes between 50% and 65% of median income and whether there is a difference in impacts for a family newly receiving a subsidy (Subgroup 1) versus a family that had a subsidy but lost it (Subgroup 2).
In addition, the study will test the effects of longer certification periods by randomly assigning those approved to receive subsidies in the treatment group into two groups: a) eligibility certified for six months; and b) eligibility certified for one year. Families in the two treatment groups will retain eligibility for subsidies over the two-year study period, provided their income remains below the experimental limit and they comply with other requirements (e.g., continue to be employed, have children under 13 years of age, etc.). Outcomes will be measured through administrative records and interviews with parents.
The experiments will result in three related sets of analyses: an implementation evaluation, an analysis of impacts on families, and a cost-benefit analysis. The experiment began in spring 20054; the final report for the evaluation will be completed in early 2008. Near the end of the 24-month period during which families in both experiments receive subsidies under enhanced eligibility guidelines, a letter will be mailed to families reminding them of the date that the enhanced eligibility guidelines will no longer be in effect.
c. Washington
The CCDF is a major source of funds for child care subsidies in Washington. The law requires states to establish sliding fee scales for subsidies but gives little further instruction. Little is known about the effects of varying co-payment amounts on outcomes related to parents’ employment and on their selection of and use of child care.
In 2000, the average co-payment amount, among those families that had a co-payment, was approximately six percent of family income (Administration for Children and Families, 2003).
However, individual state co-payment policies vary greatly. Differences in co-payments across states are quite pronounced for eligible families who are at the higher end of the income scale. For instance, in 1999, at 33 percent of state median income, among the 17 states in the National Study of Child Care for Low-Income Families, the required co-payment ranged from 2 percent of a family’s income (Minnesota) to 17 percent (Massachusetts) (Collins et al, 2000). In addition, in their 2002-2003 CCDF state plans, 14 states reported that they prohibit child care providers from charging fees in addition to the co-payments established by the state (Administration for Children and Families, 2002).
In some states, the co-payment schedule produces large jumps, or “notches,” as families earned more. In Washington State, for instance, the 1999 co-payment was 6 percent of a family’s income when its income was 33 percent of the state median, but climbed to 20 percent of the family’s income when it reached 50 percent of the state median. In other states, the co-payment was consistent across income levels; for example, in North Carolina, the co-payment always accounted for nine percent of a family’s income (Collins et al., 2000).
How do co-payments affect families’ work and child care decisions? A large literature in economics has tried to address how child care subsidies affect employment and child care use by examining the effects of reduced child care costs on these outcomes. Two useful summaries of this literature are Chaplin et al. (1999) and Blau (2000). Chaplin et al. (1999) summarize studies of the effects of child care costs on child care use patterns, and find that the average results of 13 studies of child care use indicated that a 1 percent reduction in child care costs would cause a 0.5 percent increase in the number of families using market forms of child care. This suggests that a lower co-payment would help families use non-parental care, which might help them stay at work.
Blau (2000) summarizes studies of the effects of child care costs on maternal employment. He finds that results from 12 studies of the effects of child care costs on employment imply that a 1 percent reduction in child care costs would encourage .4 percent of parents to work. However, few of these studies focus on low-income families and Blau points out that in studies that do focus on such families, the price elasticities of employment tend to be larger. Despite the large number of studies, there is a critical need for experimental studies of employment responses of low-income families in the new environment created by welfare reform.
The need for such a study of the effects of different child care co-payments on parental employment and child care choice is clear. To date, there is little research to guide states in structuring co-payment schedules. The only research on co-payments is descriptive and often shows how states co-payment policies compare. In Washington State, the current parent co-payment schedule begins with $15 per month for families whose incomes are below 82% of the federal poverty level (FPL), jumps to $50 for families with incomes between 82% and 137.5% of the FPL, and then rises very steeply. There is concern that these “notch” effects in the schedule, as well as relatively high co-payments for some families, may cause parents to drop the subsidy, change their child care arrangements to more informal arrangements in which the co-payment may not be demanded, or refuse additional hours of work if that would raise their co-payment so much that they would see no net benefit.
The study in Washington will test the effects of an alternative co-payment structure that decreases the co-payment burden for some families. The alternative co-payments were formulated so that amounts required are either the same as or less than current co-payment amounts at every point on the fee scale. That is, all participating families will either pay less or the same amount than they would under the current schedule for the same level of income. From October 18, 2005 through November 7, 2005, 5,142 families throughout the state who were applying for or reapplying for subsidies were randomly assigned to the alternative co-payment schedule (approximately 2,000) or to the standard schedule (approximately 3,142). The study will include an Implementations Study, an Impact Study, and a Cost-Benefit Study. Baseline data will come from extant records. Data collection will include 35-minute parent interviews at 8, 16 and 24 months after random assignment to gather detailed child care and employment information. The study will also rely on a 30-minute interview with the state’s coordinators from regional offices as well as eligibility staff. Finally, the study will draw from extant administrative records on use of public benefits and to obtain wage data reported to the unemployment insurance system.
A2 How the Information Will Be Used, by Whom, and for What Purpose
a. Massachusetts
In cooperation with the State Lead Child Care Agency, the Massachusetts experiment will test the effects of the Learningames curricula to address language development and the development of pre-literacy skills for very young children in family child care settings. Learningames is an approach built on evidence that children learn best in one-on-one interactions with a caregiver who is nurturing and responsive to the child and who also provides rich language stimulation. If caregivers in home-based settings are given appropriate tools, the intimate interactions that are possible in home-based care provide an ideal opportunity for promoting children’s oral language, communication skills, and early phonological awareness. The curriculum is well suited to support the caregiving practices that have been linked to improving and providing good quality care, a goal of the CCDF. An earlier version of Learningames was first used in the highly-successful Abecedarian program, as an approach for teachers to use with children and their parents. It has been adapted for use in Even Start, a two-generation family literacy program, both with the teachers in the center-based early childhood programs and with parents to use at home with their children from birth to eight years. In its revised form, Learningames expands the number of games and activities to reflect the newest research on the importance of language and emergent literacy. It is appropriate for use with parents, family child care providers and early childhood teachers.
What makes the Learningames curriculum suitable for family child care is the fact that it is less concerned with a particular set of lessons and content than with the relationship between the caregiver and the children. It focuses on teaching caregivers to be (a) responsive and nurturing and, (b) capable of using observation of the child and appropriate stimulation to move the child from his/her developmental level to a higher level of functioning. The stimulation is organized around a set of simple games that the provider can play with the child one-on-one, as a way to encourage the provider to listen to the child, talk to the child, respond to the child’s questions and actions, and help the child develop.
Learningames supports caregivers in providing an effective learning environment for children. The model is organized around three strategies that are used sequentially, forming a responsive interaction between caregiver and child. First, the caregiver notices what the child is doing as well as the child’s interests and developmental level. Second, based on what he or she notices, the caregiver then nudges the child in an appropriate way to help move his/her understanding to a higher level. Once the child responds, the adult begins to narrate, to track what the child is doing and/or to guide the child’s behavior in new directions. As the adult notices changes in the child’s behavior the cycle begins again and may repeat itself many times even during a short interaction. To help caregivers find opportunities to engage in these types of interactions with children, Learningames has a set of about 200 simple activities (each presented on a card with both a picture of a caregiver and child in that activity and easily-understood suggestions about how to initiate and extend the activity). The 200 “Learningames” focus on simple, everyday activities to provide opportunities for the Notice-Nudge-Narrate cycle. In addition, the activities are carefully constructed to promote children’s oral language development, communication skills, and early understanding of sounds and letters.
Mentors are trained over a three-day period in how to work with providers both to implement Learningames and to address other problems and needs they may have. Providers and mentors each attend separate, initial group training sessions after which mentors visit the homes every two weeks, spending one to two hours with each provider. During this visit, mentors introduce new games, model their use and encourage providers to try using the approach with children. They answer questions and discuss problems or issues the provider may have and suggest solutions.
Giving providers simple games to play individually with children, and training providers about the importance of using language and of encouraging children to use language in an interactive cycle, Learningames incorporates the most recent research on what supports children’s development and delivers it in a way that seems well-suited to the family child care environment.
Study Components
The study includes two components:
An Implementation Study to document the process in which Learningames is implemented in family child care homes in Massachusetts and describe challenges and barriers to implementation that are encountered in this setting; and
An Impact Study to estimate the effects of Learningames on the quality of care provided in family child care homes and children’s language and pre-literacy skills.
Implementation Study
When combined with a well-designed impact study, a comprehensive Implementation study is an indispensable evaluation component. In the overall context of the experiments, the research goals of the Implementation analysis are to:
Describe the intervention;
Describe the degree to which the intervention was implemented as planned;
Document relevant contextual factors; and
Help interpret the findings of the impact study.
The principal task of the Implementation Study is to describe the intervention. Because observed impacts are the result of differences in services actually experienced by treatment and control group providers and children (i.e., the fidelity of the implementation), those experiences must be documented. There will be three sources of information on fidelity/degree of implementation: home visitors that are implementing Learningames will use a five-point scale with definitions at each point; as part of the implementation, providers will keep feedback logs on which games they used with which children during the week; and, as part of the independent data collection, Abt observers will also complete a simple observational measure of fidelity.
In addition, we will investigate challenges to implementation and possible reasons for differences in implementation through informed discussions during regular meetings with child care systems staff and home visitors and through a Home Visitor Questionnaire. The Questionnaire, which home visitors for both the treatment and control groups will be asked to complete, will collect information on caseload size, frequency duration and purpose of home visitors, education and training, and language of the home visitor.
Information on actual implementation of Learningames will help in the interpretation of impact findings. For example, if Learningames fails to bring about its expected impacts on providers and children in Massachusetts, the information collected through the Implementation Study should help us distinguish among three possible reasons: Learningames was not implemented as planned and was not a true test of the intervention; contextual factors counteracted the behavioral influence of the demonstration; or the demonstration was implemented well and in a favorable context, but failed to change behavior in expected ways. Clearly, if there are no impacts, each of these reasons conveys different policy information.
Impact Study
The impact study will address the following research questions:
What is the impact of a research-based developmental curriculum (Learningames), designed to enhance the quality of care and tailored to the needs of family child care provider, on providers’ behavior and interactions with children and on the language and literacy environment of the home?
What is the impact of the intervention on children’s language and pre-literacy skills?
Family child care homes participating in the evaluation will be randomly assigned to one of two groups. The treatment group will receive Learningames, in addition to generalized technical assistance provided by family child care networks. Control homes will receive only the generalized technical assistance. We will then estimate the impact of Learningames on both providers and children by comparing the Learningames group and the control group on a set of key outcomes described below. The process of random assignment ensures that the groups of providers are the same across all measured and unmeasured characteristics that could be related to the study outcomes. This means that the estimates of the impacts of Learningames will be unbiased; statistically significant differences on the study outcomes that favor the Learningames group will provide convincing evidence that the implementation of the curriculum caused these positive differences.
The logic of the impact study is that Learningames will change what providers do with children, which in turn will improve children’s developmental outcomes. In addition, it is assumed that there will be differences in how well caregivers implement the Learningames approach, depending on the caregiver’s background characteristics. To assess all parts of this logic model, the study will collect three kinds of data will be collected for the evaluation: observations of provider behavior with children, a brief interview with the provider about her education and experience, and assessments of children’s development.
The provider observations will use the QUEST Caregiver Rating Scale, a standardized coding system for rating the quality of early childhood settings, including either family child care or center care. The Caregiver Rating Scale is based on the most recent research on instructional practices that are associated with children’s development and learning. The rating scale focuses on caregiver warmth/responsiveness and on caregiver support for the child’s development in four critical domains—cognitive development, especially language development and early literacy; emotional development; social development; and physical development.
The QUEST describes six main aspects of caregiver interactions in the home:
Caregiver with Children
Caring and responding (items 1-10)
Using positive guidance and discipline (items 11-19)
Supervision (items 20-23)
Does no harm (items 24-28)
Supporting Social Emotional Development (items 29-36)
Supporting Play (items 37-40)
Supporting Cognitive Development
Instructional style (items 41-45)
Learning activities and opportunities (items 46-56)
Supporting Language Development and Early Literacy (items 57-67)
Television and Computers (items 68-69)
The QUEST is completed based on a minimum of 2.5 to 3 hours of observation in the home. Providers will be observed up to four times over the two years, before the intervention and three additional times. A subset of observations will use an additional rating scale – the Family Day Care Rating Scale (FDCRS) – for the principal purpose of enabling a comparison of the two ratings.
Outcome data on the children will be collected using the Preschool Language Scale-Fourth edition (PLS-4). The PLS-4 measures children’s receptive and expressive language. It is appropriate for children from birth to six years. The norms include children’s total language, auditory comprehension, expressive communication, standard scores, percentile ranks, and language age equivalents. There are English and Spanish language versions. The assessment takes between 20 and 45 minutes per child. Outcome data will be collected at three points in time, at seven, 12 and 24 months after the intervention begins.
After OMB clearance has been obtained, a provider questionnaire will be administered. It will include questions about the provider’s background, the level and types of education and training obtained, languages spoken, and motivation for being a child care provider. Information from the provider questionnaire will be used as covariates for the impact study. A second, shorter questionnaire will be administered at 24 months of the study, to document additional education and training obtained by providers, beyond the Learningames intervention, over the two years. Finally, for baseline information on children’s developmental status, we will use extant data that are currently collected by the home visitors employed by the family child care networks in the course of their regular work.
b. Illinois
The Illinois experiments, which are being conducted in close cooperation with the Illinois Department of Human Services (IDHS) and with Action for Children of Illinois (AFC), will examine the effects of obtaining and losing subsidies on low-income families with employed parents. Exhibit A2.1 shows a conceptual framework that suggests the hypothesized outcomes from these experiments.
In 2003, 51% of families receiving child care subsidies in Illinois had received TANF within the last five years (Anderson, Ramsburg, & Rothbaum, 2003). Families with employed parents apply and re-apply for subsidies within a particular economic and general policy environment. If they are in a treatment group, they will begin to receive or continue to receive subsidies, which may affect both their child care and employment decisions. In particular, subsidy receipt may affect the type and nature of the child care they select. By increasing families’ purchasing power for care, the subsidy may enable families to gain access to child care that they think is more appropriate for their children. The state’s subsidy payments for this care may increase the stability of the care arrangement. Similarly, the subsidy may allow parents to stay at work and the higher income ceiling, may encourage them to take better jobs or work more hours, or stay employed. In turn, more stable employment may lead to more stable child care arrangements. In addition, a longer certification period for subsidies may further stabilize both employment and a child care arrangement if the process of certification is indeed disruptive. Ultimately, child well-being might be affected if extra employment increases family income and if the child’s experiences in care are more positive because of the subsidy. Parental well-being might result from extra income, extra employment, a more stable child care situation, or improved child well-being. Stable employment and income also may reduce the need to rely on cash assistance and other public benefits.
The research will take place in Cook County, Illinois. Two-thirds of the state’s subsidy funding goes to this highly urban area; therefore the study results will have strong implications for state policy. The information may also be useful to other states that have highly urban areas to which the preponderance of subsidy funding flows. In addition, in order to determine the degree to which the characteristics of the study population is representative of other families receiving subsidies in the state, the study team will use data from anonymous statewide subsidy records to compare the study population with 1) families near the top of the eligibility ceiling in Cook County; and 2) families near the top of the subsidy eligibility ceiling in non-Cook County areas.
Exhibit A2.1
Conceptual Framework
|
Community Characteristics (child care market, employment market) |
Family and Child Characteristics |
|
The research questions inherent in this conceptual design follow:
For low-income, working families:
What is the impact of receiving a child care subsidy on parental employment, family income and the receipt of public benefits?
What is the impact of receiving a subsidy on the type of child care chosen, the stability of the arrangement, and on parents’ satisfaction with the child care?
What is the impact of losing a child care subsidy on parental employment, on family income and on the receipt of public benefits?
What is the impact of losing a subsidy on the type of child care chosen, the stability of the arrangement, and on parents’ satisfaction with the child care?
Does the length of the certification period for subsidies affect how long families receive a subsidy?
The information collected in this study will provide policymakers in Illinois with much needed information regarding the effects of raising the income eligibility ceiling for child care subsidies from 50% to 65% of SMI and will inform decisions on whether or not the state should change the income eligibility ceiling. In addition, since all states must deal with the issue of where to set income eligibility for child care subsidies, the information produced by this study will help inform state and local policy decisions across the county.
Study Components
The study includes three components: 1) an Implementation Study which will provide information about the way in which the initiatives were mounted and helps in the interpretation of the findings from the impact and benefit cost analyses; 2) an Impact Study which will identify the outcomes affected by receiving a subsidy; and 3) a Benefit-Cost Analysis which will compare the costs of the initiative (e.g., the amount of funding used for subsidies) with the initiative’s benefits.
Implementation Study
When combined with well-designed impact and cost-benefit analyses, a comprehensive Implementation study is an indispensable evaluation component. In the overall context of the experiments, the research goals of the implementation analysis are to:
Describe the intervention;
Describe the degree to which the intervention was implemented as planned;
Document relevant contextual factors; and
Help interpret the findings of the impact and cost-benefits studies.
The principal task of the Implementation Study is to describe the intervention. Because observed impacts are the result of differences in policies, operations, and services actually experienced by treatment and control group families, those experiences must be documented. Although the interventions in Illinois appear to be well-designed and well-specified on paper and relatively simple to implement, describing what actually happens in the field and over time is a critical factor in assessing the faithfulness of the test, as well as in determining the policy information conveyed by impact findings and their use in guiding decisions about future policy choices.
Another key research goal of the Implementation Study will be to describe contextual factors that may affect the outcomes of interest. For example, understanding the structure of the existing child care market for low-income families, local economic conditions, and local expectations about child care arrangements among low-income families, helps define the playing field in which impacts may happen. Community-level differences in those factors may lead to differences in expected impacts.
Information on actual demonstration program operations, family experiences and attitudes and contextual factors also may help in the interpretation of impact findings. For example, if the demonstration fails to bring about its expected impacts in Cook County, the information developed by the Implementation Study should help us distinguish among three possible reasons: the demonstration was not implemented as planned and was not a true test of the policy innovation; contextual factors counteracted the behavioral influence of the demonstration; or the demonstration was implemented well and in a favorable context, but failed to change behavior in expected ways. Clearly, the three different types of reasons for no impacts convey different policy information.
Another use of Implementation Study results in the impact analysis is to help understand variations in impacts over time. By describing and monitoring site changes over time in program implementation and key contextual factors, the Implementation Study can provide information that can help the study generate reasonable hypotheses about such impact variations.
These various goals lead to a number of research questions for the Illinois experiments, which are included in Exhibit A2.2.
Exhibit A2.2
Research Questions for the Implementation Study |
1. Describing intervention design, rationale, planning, and start up |
|
|
|
|
Did workers require special training, and, if so, how was it conducted? Were special forms and education materials developed? Were administrative systems affected? Were agency staffing or facilities changed in any way to accommodate the demonstration, and, if so, how? What other preparations were required? |
|
2. Describing the operation of the intervention |
|
|
|
|
|
|
|
|
3. Documenting relevant contextual factors for the demonstration |
|
|
|
|
4. Identifying implications of processes for the impact and cost-benefits studies |
|
|
|
Impact Study
The purpose of the Impact Study is to estimate the effects of gaining or losing a subsidy on a number of outcomes. The random assignment design enables us to identify the net effect of subsidies on these outcomes by letting us draw comparisons between the treatment and control group.
The types of outcomes we will assess fall into three general categories:
Employment and income outcomes (e.g., months in which employment occurred, average numbers of weeks worked, stability of employment, average monthly earnings, changes in earnings levels, average change in total family income)
Child care outcomes (e.g., reliability, flexibility, and stability of child care arrangements, numbers of child care problems, cost of the child care, average amount parents spend on child care)
Public assistance outcomes (e.g., percent with any use of TANF assistance, average months of use of TANF, percent with any use of food stamps, average months use of food stamps)
The research design includes three telephone interviews, so that families are reached at Month 8,5 Month 16, and Month 24 after they enter the study. We assume that at least 25 percent of families will be interviewed in person, most because they cannot be reached by telephone and a small percentage because they have more than two children (for which an in-person interview will be needed to collect detailed information on child care use).
In addition to assessing differences in these outcomes between the treatment and control group, the design of the impact analysis will also enable us to identify differences within the treatment group of families who receive subsidies under the enhanced eligibility guidelines with a 6-month certification period and those that receive subsidies for a 12-month period.
Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with expenditure information, to determine whether the costs of the experimental programs or policies are justified, given their impacts.
The benefits and costs of the experiment will be assessed from several perspectives, including families and children, the government, tax payers, and society.
From the perspective of families and children, we will identify the gains and losses to families that result from use or loss of subsidies, in terms of parental employment and child care stability. We also will identify changes in the use of public assistance, families’ child care costs, and total family income.
From the government perspective, we will tally benefits and costs to federal, state, and local governments. The benefits of child care subsidies could include increased tax revenue as a result of increased employment of families that receive subsidies and reduced use of public assistance. The primary costs to government will be the actual subsidy payments and the costs related to administering them. The potential costs are increases in the use of public assistance.
From the taxpayers’ perspective, we will tally the benefits and costs to the general public. The taxpayer will benefit from increased tax revenue since it offers the possibility of greater services or lower taxes.
The perspective of society as a whole combines the perspectives of families and other taxpayers. A net gain to society occurs when a gain to one taxpayer is not a loss to other taxpayers. For example, lower administrative costs resulting from longer certification periods represent a gain to the government budget that does not come out of the pocket of any taxpayer. By contrast, child care subsidy payments themselves represent neither a cost nor benefit to society per se because one taxpayer receives them while other taxpayers pay for them.
One of the challenges of the cost-benefit analysis is that some of the benefits of receiving subsidies are not valued in dollars. This is particularly the case with outcomes related to parental satisfaction with child care.
Overview of Data Collection
The impact, implementation, and cost-benefit studies will rely largely on extant data from documents and administrative records. In addition, a 35-minute interview will be conducted at three points over two years to study participants. Exhibit A2.3 describes the data collection.
Exhibit A2.3
Summary of Data Collection |
|||
|
Study Components to be Informed by Data Collection |
||
Type of Data Collection |
Impact Study |
Cost-Benefit Study |
Implementation Study |
Participant interviews |
x |
x |
|
Extant administrative records (i.e., data from state public assistance and child care data; state unemployment insurance data) |
x |
x |
x |
State memoranda, documents and procedural manuals |
|
x |
x |
Census and Bureau of Labor Statistics data |
|
|
x |
Interviews with select key informants |
|
x |
x |
Researcher notes and memoranda from study development and start-up process |
|
x |
x |
c. Washington
Study Components
Exhibit A2.4 shows the alternative co-payment scale and compares it with the existing scale. On the existing scale – represented by the dark, solid line – families are placed in one of three co-payment tiers. Tier 1 (income less than 82% of the FPL) is $15 per month, Tier 2 (income between 82 and 137.5% of FPL) is $50 per month, and Tier 3 (income above 137.5% of FPL) is $50 per month + 44% of each additional dollar of income above 137.5% of FPL. The alternative fee scale varies by the number of children in care and is depicted by the other three lines. In Tier 1, the amount of the alternative payment is also $15. In Tier 2, the co-payment rises gradually to $50, eliminating the notch; in Tier 3, the marginal co-payment and the overall amount are reduced.
Exhibit A-2.4 Current and Proposed Monthly Alternative Parent Fee Scales |
The study includes three components:
An Implementation Study, which will provide information about the way in which the initiatives were mounted and helps in the interpretation of the findings from the impact and benefit cost analyses.
An Impact Study, which will estimate the effects of the three alternative co-payment schedules on child care subsidy recipients; and
A Benefit-Cost Analysis, which will compare the costs of the initiative (e.g., the amount of funding used for subsidies) with the initiative’s benefits.
The evaluation is guided by a conceptual framework
that hypothesizes the specific outcomes that are likely to be
affected by a change in the co-payment schedule (Exhibit A2.5).
Exhibit A2.5
Conceptual Framework
|
Community Characteristics (child care market, employment market) |
Family and Child Characteristics |
Child’s experience
In care
Subsidy Policy
:
Change in co-payment scale
-
Child care arrangements: Type Stability
Employment characteristics:
Family
Parent
Well
-
being
Child
Well
-
being
Child
Well
-
being
Amount of co-payment Stability
Hours of
employment
Hourly
wage |
Employed parents with children who need child care apply and re-apply for subsidies within a particular economic and general policy environment. If they are in the treatment group for this study, they will be subject to an alternative fee schedule that will likely result in reduced child care costs for them, either immediately or as their income rises over time. The co-payment represents the cost of care for subsidized families so it can be considered their “effective price” for child care. An eligible parent may choose not to receive subsidies but instead select a child care setting with a full price that is less than the required co-payment amount because this choice is more economically advantageous. Therefore, a change in the co-payment amount—the effective price– may influence the type and nature of the child care they select as well as the long-term desirability of subsidies.
In addition, since the alternative co-payment smoothes out the notches in the scale and reduces the overall marginal rate for families at relatively higher income ranges, the alternative fee scale enables families to keep more of their earnings and reduces any sharp changes in the amount of their net earnings as their income rises. These changes may make sustained employment as well as increased earnings more desirable. Therefore, the alternative fee scale may encourage parents to take better jobs or work more hours or stay employed. In turn, more stable employment may lead to more stable child care arrangements, as well as reduced use of public assistance. Ultimately, child well-being might be affected if extra employment increases family income and if the child’s experiences in care are more positive because of the subsidy. Parental well-being might result from extra income, extra employment, a more stable child care situation, or improved child well-being. Stable employment and income may reduce the need to rely on cash assistance and other public benefits.
Research Questions
The research questions inherent in this conceptual framework are:
For low-income, working families receiving subsidies:
What are the effects of the alternative co-payment scale on parental employment, family income, length or receipt subsidy receipt and the receipt of other public benefits?
What are the effects of the alternative co-payment scale on the type of child care chosen, the stability of the arrangement, and on parents’ satisfaction with the child care?
Is the effect of the alternative co-payment amount on employment and child care outcomes different for families who are new subsidy recipients than for families who are already receiving a subsidy and wish to continue?
Does the impact of the alternative co-payment differ for families at different income levels at the beginning of the study (i.e., whether they are in Tiers 1, 2, or 3 under the current parent fee scale)?
Implementation Study
When combined with well-designed impact and cost-benefit analyses, a comprehensive implementation study is an indispensable evaluation component. In the overall context of the experiments, the research goals of the implementation analysis are to:
Describe the intervention;
Describe the degree to which the intervention was implemented as planned;
Document relevant contextual factors; and
Help interpret the findings of the impact and cost-benefits studies.
The principal task of the Implementation Study is to describe the intervention. Because observed impacts are the result of differences in policies, operations, and services actually experienced by treatment and control group families, those experiences must be documented. Although the intervention in Washington appears to be well-designed and well-specified on paper and relatively simple to implement, describing what actually happens in the field and over time is a critical factor in assessing the faithfulness of the test, as well as in determining the policy information conveyed by impact findings and their use in guiding decisions about future policy choices.
Another key research goal of the Implementation Study will be to describe contextual factors that may affect the outcomes of interest. For example, understanding the structure of the existing child care market for low-income families, local economic conditions, and local expectations about child care arrangements among low-income families, helps define the playing field in which impacts may happen. Community-level differences in those factors may lead to differences in expected impacts.
Information on actual demonstration program operations, family experiences and attitudes and contextual factors also may help in the interpretation of impact findings. For example, if the demonstration fails to bring about its expected impacts in Washington, the information developed by the Implementation Study should help us distinguish among three possible reasons: the demonstration was not implemented as planned and was not a true test of the policy innovation; contextual factors counteracted the behavioral influence of the demonstration; or the demonstration was implemented well and in a favorable context, but failed to change behavior in expected ways. Clearly, the three different types of reasons for no impacts convey different policy information.
Another use of Implementation Study results in the impact analysis is to help understand variations in impacts over time. By describing and monitoring site changes over time in program implementation and key contextual factors, the Implementation Study can provide information that can help the study generate reasonable hypotheses about such impact variations.
These various goals lead to a number of research questions for the Washington experiment, which are included in Exhibit A2.6.
Exhibit A2.6 Research Questions for the Implementation Study |
1. Describing intervention design, rationale, planning, and start up |
|
|
|
|
Did workers require special training, and, if so, how was it conducted? Were special forms and education materials developed? Were administrative systems affected? Were agency staffing or facilities changed in any way to accommodate the demonstration, and, if so, how? What other preparations were required? |
|
2. Describing the operation of the intervention |
|
|
|
|
|
|
|
|
3. Documenting relevant contextual factors for the demonstration |
|
|
|
|
4. Identifying implications of implementation for the impact and cost-benefits studies |
|
|
|
Impact Study
The purpose of the Impact Study is to estimate the effects of reducing parent co-payments for child care subsidies on a number of outcomes. The random assignment design enables us to identify the net effect of subsidies on these outcomes by letting us draw comparisons between the treatment and control group.
The types of outcomes, or dependent variables, we will assess fall into four general categories:
employment and earnings;
family income and public assistance;
child care outcomes; and
subsidy and public assistance expenditures.
The independent variable is the family’s assignment to the treatment or control group. The dependent variables are the outcomes that will be assessed as a result of the assignment to the treatment or control condition. Exogenous variables are the characteristics of the family (e.g., family size, racial composition, geographic location) that are collected before random assignment and therefore are unaffected by whether they are in the treatment or control group. The exogenous variables are used to partial out the effects of any pre-existing differences between the treatment and control groups. The inclusion of such variables increases the power of the evaluation to detect changes that result from the treatment. Exhibit A2.7 lists all of the variables that will be used in the impact analysis.
Exhibit A2.7 Independent, Exogenous, and Dependent Variables |
|
INDEPENDENT VARIABLE |
Assignment to Alternative or Existing Co-Payment Schedule |
|
|
EXOGENOUS VARIABLES |
Family characteristics |
|
Household size |
|
Number of parents in household |
|
Number of adults in household |
|
Ages of parents |
|
Number of children |
|
Ages of children |
|
Immigration status |
|
Race/Ethnicity |
|
Education level(s) of parents |
|
Prior receipt of TANF cash assistance |
|
|
DEPENDENT VARIABLES |
Employment characteristics |
|
Number of employers at point single point in time |
|
Total number of employers |
|
Hourly wage per employer |
|
Weekly earnings |
|
Average number of weekly hours worked |
|
Average monthly earnings |
|
Hourly wage at 24 months |
|
Change in hourly wage from baseline |
|
Change in number of weekly hours worked |
|
Average monthly earnings |
|
Number of breaks in employment |
|
Average length of break |
|
Number of work days lost because of child care problems |
|
Number of weeks in which non-traditional schedule occurred |
|
Number of quarters in which any employment occurred |
|
Number of consecutive quarters in which employment did not occur |
|
Number of hours worked per quarter |
|
Number of employers per quarter |
|
Number of hours worked per quarter |
|
|
|
Family income and public assistance |
|
Total annual family income |
|
Income from child support |
|
Dependent Care Tax Credit taken |
|
|
|
Child care characteristics |
|
Type of child care arrangements for all children in family |
|
Number of child care arrangements per family |
|
Duration of individual child care arrangements |
|
Duration of individual subsidized child care arrangements |
|
Monthly amount paid for child care* |
|
Level of child care flexibility |
|
Level of child care reliability |
|
Number of problems with child care arrangements |
|
Average number of breaks/interruptions in child care arrangements |
|
Average monthly co-payment amount |
|
Cumulative co-payment amount over study period |
|
|
|
Subsidy and public assistance expenditures |
|
Any receipt of TANF cash assistance |
|
Average number of months on TANF cash assistance |
|
Average monthly amount of TANF cash assistance |
|
Unit administrative cost of TANF cash assistance |
|
Any receipt of food stamps |
|
Average number of months of food stamp receipt |
|
Average monthly amount of food stamp receipt |
|
Prior receipt of food stamps |
|
Unit administrative cost of food stamps |
|
|
|
Per family cost to state of subsidized arrangements |
|
Per child cost to state of subsidized arrangements |
|
Duration of subsidy receipt by arrangement |
|
Number of subsidized arrangements per family |
|
Unit administrative cost of subsidized arrangement |
|
|
Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with expenditure information, to determine whether the costs of the experimental programs or policies are justified, given their positive impacts.
The benefits and costs of each of the experiment will be assessed from several perspectives, including families and children, the government, tax payers, and society.
From the perspective of families and children, we will identify the gains and losses to families that result from a decrease in their co-payment, in terms of parental employment and child care stability. We also will identify changes in the use of public assistance, families’ child care costs, and total family income.
From the government perspective, we will tally benefits and costs to federal, state, and local governments. The benefits of reduced co-payments could include increased tax revenue as a result of increased employment of families that receive subsidies and reduced use of public assistance. The primary costs to government will be the actual subsidy payments and the costs related to administering them.
From the taxpayers’ perspective, we will tally the benefits and costs to the general public. The taxpayer will benefit from increased tax revenue since it offers the possibility of greater services or lower taxes.
The perspective of society as a whole combines the perspectives of families and other taxpayers. A net gain to society occurs when a gain to one taxpayer is not a loss to other taxpayers.
A3 Use of Improved Technology to Reduce Burden
a. Massachusetts
The use of improved technology has been incorporated into the data collection wherever possible to reduce respondent burden. For instance, information about the number of subsidized children enrolled in the homes and the duration of their enrollment will be obtained from centralized databases maintained by each family child care system.
b. Illinois
As summarized in Exhibit A2.3, the information to be collected for this study will come from documents, existing records and data, in-person or telephone interviews with parents, individual and small-group interviews with providers and agencies. The use of improved technology has been incorporated into the data collection wherever possible to reduce respondent burden. When information is available to the contractor from a centralized, computerized source, such information has not been included in the data collection instruments. Specifically, automated information will be used to provide data about the use of public assistance programs, data on the characteristics of the underlying population will be taken from data collected by the Census Bureau and the Bureau of Labor Statistics Labor Market Data, and CATI systems will be used in the telephone interviewing.
c. Washington
Much of the information to be collected for this study will come from documents, existing records and data, and telephone interviews with parents. The use of improved technology has been incorporated into the data collection wherever possible to reduce respondent burden. When information is available to the contractor from a centralized, computerized source, such information has not been included in the data collection instruments. Specifically, automated information will be utilized to provide data about the use of public assistance programs and CATI systems will be used in the telephone interviewing.
A4 Efforts to Avoid Duplication
There are no other studies currently underway to examine the effects of Learningames or any other curriculum for family child care providers in the Commonwealth of Massachusetts. There are no other studies currently underway to examine the effects of providing subsidies in Illinois to families who would otherwise be ineligible for them according to current eligibility limits in Illinois. There are no other studies currently underway to examine the effects of reducing parent co-payments in the State of Washington.
A5 Efforts to Minimize Burden on Small Businesses or Other Small Entities
a. Massachusetts
Every attempt has been made to reduce the burden placed on family child care providers participating in the experiment. To this end, information on child care quality and fidelity of implementation of Learningames will be collected through direct observation rather than interviews or questionnaires that address provider behavior.
b. Illinois
No small businesses are involved in this study. However, every attempt has been made to reduce the burden placed on families participating in the experiments. The evaluator will:
collect information on benefit receipt from standard applications for child care subsidies and administrative records for TANF, food stamps, and unemployment insurance;
collect information on administrative practices from brief interviews with staff from the Illinois Department of Human Services and Action for Children of Illinois and select other local informants; and
collect information on employment and educational characteristics from a brief parent survey.
c. Washington
No small businesses are involved in this study. However, every attempt has been made to reduce the burden paced on families participating in the experiments. The evaluator will:
collect information on benefit receipt from standard applications for child care subsidies and administrative records for TANF, food stamps, and unemployment insurance; and
collect information on income, employment and educational characteristics from a brief parent survey.
A6 Consequences if the Information is Not Collected
a. Massachusetts
Currently, states expend funds on initiatives to improve family child care, but have little guidance on what constitutes an effective intervention. Compared with center-based care, for which there is an expanding body of knowledge based on rigorous experimental research, family child care has been relatively neglected, except for descriptive studies. This study will provide reliable information on the effectiveness of a research-based intervention that is particularly suited to family child care.
Observational data on caregivers and the home environment is essential in understanding the impact or lack of impact on children of the intervention, since the caregiver’s behavior mediates those impacts. If the intervention results in behavior and interactions that promote children’s development, we might expect to see a positive impact on children’s performance on standardized tests of language and preliteracy skills. If, on the other hand, the intervention fails to make changes in caregivers that are greater than those that occur in the control group as a result of conventional assistance, we would not expect a measurable difference in the outcomes for children in the two groups of homes. We do not expect to see immediate or substantial change in caregiver behavior. The observations are repeated over the course of the study so that we can understand the rate at which and the ways that caregiver behavior changes, with and without an intervention.
Background information on caregivers will be collected once to provide information necessary to construct covariates for the regression models used in the impact analysis.
Child outcome data will be gathered through administration of standardized assessments at three points in time. In this case, the repeated measures are necessary to ensure that we capture information on children who leave the home before the end of the two-year-period of the study. While these data collections will generally occur at set intervals, like the observations, we will ask providers to let us know when a child is leaving so that we can schedule an assessment if the regularly scheduled data collection will miss that child.
b. Illinois
The data for which clearance is being requested for the Illinois experiment will be collected at three time points over a two-year period for each family. Baseline data on family characteristics will be taken from state records collected as part of the application or recertification process. Parents will be interviewed at 8, 16 and 24 months to obtain follow-up data on family characteristics, employment and educational characteristics, and family income. It will be necessary to speak with them at intervals of reasonable lengths because we believe that it will be too difficult for participants to recall work and child care transitions without sufficient detail for a period longer than eight to ten months. For control group members, we will not be able to augment information on child care use collected through the interview with extant data from administrative records because they will not be using child care subsidies unless their incomes decrease and they become eligible for the standard program. Therefore, without these follow-up interviews it will be impossible to estimate the effects of the intervention.
c. Washington
The data for the random assignment experiment in Washington for which clearance is being requested will be collected at three time points over a two-year period. Parents will be interviewed at eight, 16, and 24 months to obtain follow-up data on family characteristics, employment and educational characteristics, family income, and child care characteristics. Baseline data on family characteristics, income, employment, and child care characteristics will be collected from administrative records and these records will be used for the analysis for the two-year period of the study unless parents deny researcher access to them. Parent interviews will be necessary to augment administrative data because the information collected through administrative systems is insufficient to measure fully the impacts of the changed co-payment policy, especially for families that leave the subsidy system during the course of the two-year intervention. It also will be necessary to speak with parents at intervals of reasonable lengths because we believe that it will be too difficult for participants to recall work and child care transitions without sufficient detail for a period longer than eight-10 months. Without these follow-up interviews it will be impossible to estimate the effects of reducing parent co-payments.
A7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5 (D) (2)
All three proposed data collections are consistent with the guidelines set forth in Section 1320.5 (D) (2).
A8 Efforts to Consult with Persons Outside the Agency
An announcement of the Administration for Children and Families’ (ACF) intent to seek approval to collect this information provided an opportunity for public comment on this study. This announcement was published in the Federal Register, September 21, 2005, Volume 70, Number 182, pages 55402-55403 and specified a 60-day period for comment ending November 20, 2005. No comments or suggestions were received in response to this notice. A copy of the relevant Federal Register announcement is provided in Appendix A.
Several individuals were consulted in developing the design for the studies and identifying the types of data to be collected. Their feedback was obtained through telephone conversations, on-site meetings of the full study staff, and meetings with individual consultants. The names and affiliations of these individuals include:
Bobbie Weber
Coordinator
Oregon Child Care Research Partnership
Oregon
State University
236 NW 28th Street
Corvallis, OR 97330-5307
David Edie
State Technical Assistance Specialist
10 S Blackhawk Ave
Madison, WI 53705-3317
Deanna T. Schexnayder
Associate Director
Ray Marshall Center for the Study of Human Resources
LBJ School of Public Affairs
The University of Texas at Austin
3001 Lake Austin Blvd., Suite 3.200
Austin, TX 78703
512-471-2193 (voice)
Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Louise Stoney
NCCIC Technical Assistance Specialist
308 Thais Road
Averill Park, NY 12018
518-674-5635
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA
A9 Payments to Respondents
a. Massachusetts
Family child care homes will be given a $20 gift certificate for children’s educational materials each time researchers conduct an observation and assess children in their home, to thank them for accommodating the researchers in their homes. Every effort will be made to minimize the disruption in their homes caused by the data collection.
b. Illinois
Participants in the two experiments in Illinois will be interviewed three times over the two-year study period. Three-quarters of these interviews will be conducted by telephone; the remainder will be completed in-home by field interviewers. All respondents completing interviews will receive a $20 voucher or gift certificate for each interview. Families who return postcards indicating a change of address also will receive a $5 voucher or gift certificate.
c. Washington
Participants in the Washington experiment will be interviewed three times over the two-year study period. Three-quarters of these interviews will be conducted by telephone; the remainder will be completed in-home by field interviewers. All respondents completing interviews will receive a $20 voucher or gift certificate for each interview.
A10 Assurances of Confidentiality Provided to Respondents
Abt Associates is fully committed to protecting the privacy of respondents at all points in the data collection and analysis Implementation. The following data handling and reporting procedures will be used to maintain the privacy of all individual respondents:
a. Massachusetts
Each family child care provider and child who participates in the study will be assigned a unique identification number which will be used throughout the study. All data collected will be attached to and stored with this number, rather than to any name or other identifying information.
Files linking ID numbers to the names of individuals will be kept in a locked file to which access is restricted. Access to this information will be limited to field staff/interviewers who need this information to schedule interviews and to Abt senior staff who will monitor the data collection.
The importance of maintaining privacy will be stressed during data collection training. All Abt staff will be required to sign a statement that affirms their understanding of the assurance of privacy and pledges to maintain that privacy.
Coding documents and computer files will refer to respondents by their ID numbers only. No names or other identifying information will appear on data files. Access to all data bases will be protected by passwords and restricted to staff involved in the data analysis.
No data will ever be reported by the contractor in any form that can be identified with individual respondents.
b. Illinois
Prior to being enrolled in the study, parents sign a consent form that authorizes the researchers to contact them for the three interviews and indicates all information will be kept private.
Each family who participates in the study will be assigned a unique identification number which will be used throughout the study. All data collected will be attached to and stored with this number, rather than to any name or other identifying information.
Files linking ID numbers to the names of individuals will be kept in a locked file to which access is restricted. Access to this information will be limited to field staff/interviewers who need this information to schedule interviews and to Abt senior staff who will monitor the data collection.
The importance of maintaining privacy will be stressed during interviewer training. All Abt staff will be required to sign a statement that affirms their understanding of the assurance of privacy and pledges to maintain that privacy.
Coding documents and computer files will refer to respondents by their ID numbers only. No names or other identifying information will appear on data files. Access to all data bases will be protected by passwords and restricted to staff involved in the data analysis.
No data will ever be reported by the contractor in any form that can be identified with individual respondents.
c. Washington
Each family who participates in the study will be assigned a unique identification number which will be used throughout the study. All data collected will be attached to and stored with this number, rather than to any name or other identifying information.
The state will only release contact information to Moore & Associates, which will use it for the telephone interview. Abt Associates Inc. and MDRC will only use a study identification number to track participants. Files linking ID numbers to the names of individuals will be kept in a locked file to which access is restricted. Access to this information will be limited to field staff/interviewers who need this information to schedule interviews and to Moore & Associates senior staff who will monitor the data collection.
The importance of maintaining privacy will be stressed during interviewer training. All research staff will be required to sign a statement that affirms their understanding of the assurance of privacy and pledges to maintain that privacy.
Coding documents and computer files will refer to respondents by their ID numbers only. No names or other identifying information will appear on data files. Access to all data bases will be protected by passwords and restricted to staff involved in the data analysis.
No data will ever be reported by the contractor in any form that can be identified with individual respondents.
Public and Restricted Use Data
Abt will create and deliver restricted access and public use data files on a CD-ROM for the impact and cost-benefit analyses. These files must be in ASCII. ACF expects to make restricted access files available at a secure data center, Research Connections (please see the attached memo discussing the release of public and restricted use data files on the Research Connections website in Appendix B). Access to the files will be restricted to researchers whose projects are approved by ACF. These data files will not contain direct identifiers such as names or social security numbers but will contain the data used in the impact and cost-benefit analyses. Abt will employ masking and other strategies as appropriate to ensure the privacy of the sample members is ensured.
The restricted access files must contain sufficient information to allow duplication by other researchers of analyses performed by Abt unless privacy concerns require aggregating or limited detailed data. Abt Associates will also prepare and deliver tables describing each variable contained in the restricted access and public use data files. The tables must include information such as frequencies and means. Abt will also prepare and deliver documentation to accompany the public use files that describe the following: 1) each variable on the file; 2) how to use and access the file; and 3) any editing strategies employed. The CD-ROMS should include all the reports, in Adobe formats.
A11 Justification for Questions of a Sensitive Nature
For all three studies, the interviews that will be conducted do not contain any questions of a sensitive nature.
A12 Estimates of Respondent Burden
a. Massachusetts
Exhibit A12.1a presents estimates of respondent burden for the Massachusetts surveys. Exhibit A12.1b presents estimates of burden to family child care network staff for participating in training to conduct baseline observations and to do record abstraction.
Exhibit A12.1a Respondent Burden for Massachusetts Interviews |
|||||||||||||
Year |
Number of Family Child Care Providers Respondents |
Interviews Per Year |
Hours Per Interview |
Hours Total Burden: Provider Interviews |
Number of Children Respondents |
Assessments Per Year |
Average Hours Per Assessment |
Hours Total Burden: Assessments |
Numbers of Home Visitors Respondents |
Interview Per Year |
Hours Per Interview |
Hours Total Burden: Home Visitor Interviews |
Total Burden: Interviews + Assessments |
2006 |
350 |
1 |
.16 |
56 |
700 |
1 |
.5 |
350 |
64 |
1 |
.16 |
10 |
416 |
2007 |
0 |
0 |
0 |
|
700 |
1 |
.5 |
350 |
0 |
0 |
|
|
350 |
2008 |
350 |
1 |
.16 |
56 |
700 |
1 |
.5 |
350 |
0 |
0 |
|
|
406 |
Total |
700 |
2 |
|
112 |
2100 |
3 |
|
1,050 |
64 |
1 |
|
10 |
1,172 |
Exhibit A12.1a
Respondent Burden for Training Massachusetts
Network Staff for Baseline Observations
and Administrative Data Abstraction
-
Year
Number of Family Child Care Network Staff Respondents
Number of Efforts
Hours Per Effort
Hours Total Burden:
2006
2007
2008
Total
20
20
20
60
1
1
1
3
24
4
4
240
40
40
320
The estimated annual burden for respondents in the Massachusetts study is 497.3 hours (total burden divided by number of years: 1,172+ 320/3). The total annual response is 2,924 (average annual respondents multiplied by the average annual response).
b. Illinois
Exhibit A12.2a presents estimates of respondent burden for the Illinois study. Please note it also includes pre-testing the interview with up to 30 respondents. Exhibit A12.2b presents estimates of respondent burden for obtaining baseline data and administrative records from the state agency as well as the burden for the process study interviews that will be undertaken with a maximum of 15 individuals. The estimated annual burden for respondents is 1,381 hours (total burden hours divided by number of years: 2,627+135)/2). The total annual response is 12,504 (average annual respondents multiplied by the average annual response).
Exhibit A12.2a Estimates of Respondent Burden for Illinois Survey |
|
||||
Year |
No. of Respondents |
Interview Wave |
Interviews per year |
Hours Per Interview |
Burden (in hours) |
2006 |
30 |
Pre-test |
1 |
0.58 |
17 |
|
1,600 |
1 |
1 |
0.58 |
928 |
|
1,500 |
2 |
1 |
0.58 |
870 |
Total |
3,130 |
|
3 |
0.58 |
1,815 |
|
|
|
|
|
|
2007 |
1,400 |
3 |
1 |
0.58 |
812 |
|
|
|
|
|
|
Total Both Years |
4,530 |
|
4 |
|
2,627 |
Exhibit A12.2b Estimates of Respondent Burden for Illinois Records Abstraction and Process Study Interviews |
|
||||
Year |
No. of State Staff Respondents |
Type and # of Efforts |
Interviews per year |
Hours Per Effort |
Burden (in hours) |
2006 |
1 |
Records abstraction |
2 |
20 |
40 |
|
15 |
Process Study Interviews |
1 |
1 |
15 |
Total |
16 |
|
3 |
21 |
55 |
|
|
|
|
|
|
2007 |
1 |
Records abstraction |
4 |
20 |
80 |
|
|
|
|
|
|
Total Both Years |
17 |
|
7 |
41 |
135 |
c. Washington
Exhibit A12.3a presents estimates of respondent burden for the interviews for Washington study. Exhibit A12.3b presents estimates of respondent burden for the administrative records data abstraction. The estimated annual burden for respondents is 1,815 hours (total burden hours divided by number of years: 3,510+120)/2). The total annual response is 8,128 (average annual respondents multiplied by the average annual response).
Exhibit A12.3a Estimates of Respondent Burden for Interviews |
|
|
Y Respondents per Year Interview |
No. State Interviews Hours / Respondents per Year Interview |
TOTAL BURDEN |
2006 2000 1 0.58 |
30 1 0.50 |
1,175 hours |
2007 2000 2 0.58 |
30 1 0.50 |
2,335 hours |
Total 4000 3 |
60 2 |
3,510 hours |
ear
No. of Parent Interviews Hours per
Exhibit A12.3b Estimates of Burden for Data Abstraction |
|
Y State Staff per Year Staff Respondents |
TOTAL BURDEN |
2006 2 1 20 |
40 |
2007 2 2 20 |
80 |
Total 4 3 |
120 |
ear
No. of Abstractions Hours per
Across all three studies the total estimated annual burden for respondents is 3,693 hours (sum total of estimated annual burden hours for respondents for each study: 497 + 1,381 + 1,815), the total annual number of respondents is 5,280 (974.3 +2,273.5+ 2032), and the total annual response is 23,556 (the sum of the [annual average respondents multiplied by the average annual responses]: 2,924 + 12,504 + 8,128). Exhibit A12.4 includes all of the actual and average numbers from which the total annual average numbers discussed above were calculated.
Exhibit A12.4 Total Number of Respondents and Responses across all three States |
|||
State |
Total No. of Respondents (Average in parentheses) |
Total No. of Responses (Average in parentheses) |
Total No. of Burden Hours (Average in parentheses) |
MA |
2,924 (974.7) |
9 (3) |
1,492 (497.3) |
IL |
4,547 (2,273.5) |
11 (5.5) |
2,762 (1,381) |
WA |
4,064 (2032) |
8 (4) |
3,630 (1,815) |
A13 Estimates of the Cost Burden to Respondents
For all three studies, there are no direct monetary costs to respondents other than their time to participate in the study.
A14 Estimates of the Cost to the Federal Government
The information collection activity and associated forms have been developed in the performance of DHHS contract number 233-01-0012. The period of performance of the project is from September 30, 2001 through September 30, 2008. The costs associated with the data collection activity for which clearance is requested are as follows:
a. Massachusetts
Development of Data Collection Instruments $0
Data Collection $854,000
b. Illinois
Development of Data Collection Instruments $32,000
Data Collection $854,000
c. Washington
Development of Data Collection Instruments $35,000
Data Collection $450,000
A15 Reasons for Any Program Changes or Adjustments
All three studies are new projects.
A16 Plans for Tabulation, Statistical Analysis and Publication
a. Massachusetts
Analysis Plan
There are two primary sources of data to be analyzed in this study: (1) observation measures of providers’ behaviors and interactions with children and (2) measures of children’s language development. Questions about the impact of Learningames on provider behavior and child outcomes will be answered by estimating the mean difference between the treatment and control groups on the outcome measures at each post-assessment. One of the primary benefits of a randomized experiment is that it produces unbiased estimates of program impacts. Because providers have been randomly assigned, on average the treatment and control groups will be the same across all dimensions except for the presence or absence of the Learningames curriculum. As a result, any differences in average outcomes can be attributed to the implementation of Learningames. It should be noted, however, that this estimate could become biased because of poor implementation of the randomized design. Therefore, it is critical to take steps to ensure that the fidelity of the design is preserved, both in the execution of the study and analytically. Four common sources of bias are (a) post-assignment attrition, (b) crossovers of group members from treatment and control groups, (c) contamination of the control group, and (d) treatment group non-participation or under-participation (Shadish, Cook & Campbell, 2002). While the study is designed to minimize the burden on providers, it is possible that homes could drop out of the study after being assigned to a condition, particularly as this is a longitudinal study. This would result in both a loss of statistical power, and, in the case of non-random attrition, would bias estimates. Abt Associates has an impressive record of minimizing such attrition, and has established strategies for retaining participants throughout the course of longitudinal studies.
Crossover and contamination, the second and third potential threats, both result in control providers or children being exposed to the treatment either by switching groups or by being in close proximity to treatment children or their providers. This seems unlikely in this study, both because providers have little contact with each other and because the agency staff assigned to visit treatment providers will not visit any control providers. This will eliminate any temptation to introduce elements of Learningames to control providers. It is possible that a child could switch from a control provider’s home to a treatment home during the study. We will identify the children in each home, as well as any previous homes they attended, so that we can include a measure of crossover in our final analyses.
The final threat to the design is of greater concern. While the goal is for each child in a treatment home to receive the complete Learningames curriculum, for a variety of reasons (e.g., absences, differences in provider practices, timing of the child’s enrollment etc.), it is possible that there will be some children who will not receive the full Learningames treatment. If there are a large number of untreated or under-treated children in our treatment group, we will underestimate the true impact of Learningames. Therefore, we will examine whether variation in program effects across children is related to differences in the implementation of the program itself, including child attendance and provider practices as well as other variables related to the fidelity of the implementation of the curriculum.
Estimating Impacts on Children and Providers. In each home in the sample, there will be a single provider and a small number of children ranging in age at baseline from 2 to 36 months. We expect the number of children to range from one to four, and these children will therefore be clustered within homes.
To estimate the impact of Learningames on children while accounting for the fact that they are clustered within child care homes, we will fit a separate, two-level hierarchical linear model for each child outcome of interest. The model will include baseline covariates at both the child and provider level, including demographic characteristics and other characteristics known to be associated with study outcomes.6
The model is specified as follows:
Level-1 (child):
where:
is the outcome measure (e.g., PLS-4) for child i, in home j,
is the mean of the outcome in home j,
BLij is the measure of the outcome at baseline for child i, in home j,
is the coefficient associated with the baseline measure,
are a set of child-level covariates (1-h) for child i, in
home j,
are the coefficients associated with those covariates, and
is the unique error term associated with child i, in home j.
Level-2 (provider/home):
where:
is
the mean of the outcome (e.g., PLS-4) in control homes,
LGj is a school-level group indicator variable equaling 1 for Learningames homes and 0 for control homes,
is
the mean difference between the Learningames and control homes
on the outcome measure,
are a set of provider level covariates (1-h) for home j,
are the coefficients associated with those covariates, and
is the unique error term associated with home j.
The coefficient associated with the Learningames
indicator (
)
can then be directly interpreted as the impact of Learningames
on the outcome measure. If this coefficient is positive and
significant, we will conclude that Learningames has had a
positive impact on the children in family child care homes.
To estimate the impact of Learningames on providers, we will fit separate OLS regression models for each provider outcome. These models will also control for baseline measures as well as provider demographics and other characteristics known to be related to study outcomes, such as the size of the child care home.
The model is specified by the following equation:
where:
is the outcome measure for provider i,
is the mean of the outcome for control providers,
LGi is an indicator variable equaling 1 for providers in the Learningames group and 0 for those in the control group,
is the difference between the Learningames and control group
on the outcome measure,
BLj is the measure of the outcome at baseline for provider i,
is the coefficient associated with the baseline measure,
are a set of provider-level covariates (1-h) for provider i,
are the coefficients associated with those covariates, and
is the unique error term associated with provider i.
The coefficient
,
can then be directly interpreted as the impact of Learningames on
the outcome measure, controlling for a set of provider
characteristics known to be related to the study outcomes. We will
then conduct a hypothesis test to assess whether this estimate is
statistically different from zero in favor of the Learningames
group (a one-tailed hypothesis test).
Study Schedule
The planned time schedule for the study is as follows:
Expected OMB approval May 2006
Recruitment and random assignment begins May 2005
Random assignment ends July 2007
First data collection: provider observations June 2006
Second data collection provider observations +
Child assessments May/June 2007
Final data collection January 2008
Final Report December 2008
b. Illinois
Analysis Plan
There are two major sets of statistical analyses that will be completed as part of the evaluation: impact analyses and cost-benefit analyses. The analysis plan for each is provided below.
Impact Analysis
The purpose of the impact analysis is to estimate the effect of the treatments on outcomes such as employment, child care quality, and child care satisfaction relative to the control group. Random assignment ensures the internal validity of the analysis based on this comparison: we can assume that the three groups are equivalent, and that the outcomes observed for the control group in each experiment represent on average the outcomes that would have been experienced by members of the treatment group, absent the experiment.
Although simple comparisons of means and proportions would yield consistent impact estimates, more precision can be obtained by use of multivariate methods. Within each group, outcomes will vary because of other underlying factors, such as mother’s employment opportunities, children’s ages, and so on. It is easier to detect differences in central tendencies between two groups if each group is more homogeneous, i.e., if their outcomes are more tightly clustered. Regression-adjusted outcomes are indeed more tightly clustered to the extent that the covariates have explanatory power. For example, a partial R2 of 0.25 due to the covariates implies that the variance of the regression-adjusted mean is only three-quarters as large as the variance of the raw mean, and this reduction correspondingly increases the statistical power of the comparison.
Regression adjustment will therefore be used for each outcome, with a technique appropriate to the form of the outcome. The goal of these models is not to “explain” the outcomes; these will be reduced-form models and the coefficients, other than those on the treatment indicators, will in general be uninterpretable. Baseline data will include information such as the ages and relationships of family members, work and training activities of other parents or stepparents in the household, ethnicity and citizenship of children for whom subsidies are sought, family participation in TANF and other federal cash income programs, and child support received.
The form of the regression equation for a continuous outcome will therefore be:
yi = b0 + (b1 × T6i )+ (b2 × T12i)+ Σj (b3j × Xij) + εi,
where yi = value of outcome for family i,
T6i = indicator that family i is in the treatment group, with a 6-month recertification period,
T12i = indicator that family i is in the treatment group, with a 12-month recertification period,
Xij = value of baseline variable j for family i, and
εi = residual.
The parameters b1 and b2 are estimates of the impact of the treatment with a 6- and 12-month recertification period, respectively. The average of b1 and b2 provides an estimate of the average effect of the two versions of the treatment, while the difference between them is an estimate of the added effect of extending certification to 12 months. Results will be presented in tables as in Exhibit A16.1 (which would show the average effect of the full treatment group compared to the control group) and Table A.16.1a (which would show the effect of the treatment with the two different eligibility periods and the effect of the extended eligibility period as measured by the difference in outcomes between the 12-month and 6-month groups). In addition, key results will be presented graphically, in bar and line charts as in Exhibits A16.2 and A 16.3.
Exhibit A16.1
Sample Table Shell: Regression-Adjusted Estimates of Total Impact of Intervention on Employment and Earnings |
|||||
|
|
|
|||
|
Treatment Group |
Control Group |
|
Estimated Impact (Difference) |
|
Employment (%) |
|
|
|
|
|
Any covered employment, Q1-Q8a |
|
|
|
|
|
Any employment, months 1-24b |
|
|
|
|
|
(etc.) |
|
|
|
|
|
|
|
|
|
|
|
Earnings ($) |
|
|
|
|
|
Total UI earnings, Q1-Q8a |
|
|
|
|
|
Total reported earnings, months 1-24b |
|
|
|
|
|
(etc.) |
|
|
|
|
|
|
|
|
|
|
|
Sample size |
|
|
|
|
|
UI records |
|
|
|
|
|
Follow-up survey |
|
|
|
|
|
Exhibit A16.1a
Sample Table Shell: Regression-Adjusted Estimates of Impacts of Experiment 1 on Employment and Earnings |
|
|||||||||||||||||
|
Control Group Mean |
Treatment
Group: |
|
Treatment
Group: |
|
|
||||||||||||
|
Regression-Adjusted Mean |
Estimated Impact |
|
Regression-Adjusted Mean |
Estimated Impact |
Estimated Impact of Longer Eligibility |
||||||||||||
Employment (%) |
|
|
|
|
|
|
|
|
||||||||||
Any covered employment, Q1-Q8a |
|
|
|
|
|
|
|
|
||||||||||
Any employ-ment, months 1-24b |
|
|
|
|
|
|
|
|
||||||||||
(etc.) |
|
|
|
|
|
|
|
|
||||||||||
|
|
|
|
|
|
|
|
|
||||||||||
Earnings ($) |
|
|
|
|
|
|
|
|
||||||||||
Total UI earnings, Q1-Q8a |
|
|
|
|
|
|
|
|
||||||||||
Total reported earnings, months 1-24b |
|
|
|
|
|
|
|
|
||||||||||
(etc.) |
|
|
|
|
|
|
|
|
||||||||||
|
|
|
|
|
|
|
|
|
||||||||||
Sample size |
|
|
|
|
|
|
|
|
||||||||||
UI records |
|
|
|
|
|
|
|
|
||||||||||
Follow-up survey |
|
|
|
|
|
|
|
|
||||||||||
*** Statistically significant difference from control group, p < 0.01 ** Statistically significant difference from control group, p < 0.05 * Statistically significant difference from control group, p < 0.10 |
|
|||||||||||||||||
Sources: a: UI records b: Follow-up survey |
|
|||||||||||||||||
Exhibit A16.2
Sample Exhibit: Covered Employment, Q1-Q8, by Treatment Group |
|
Exhibit A16.3
Sample Exhibit: Covered Employment, by Quarter and Treatment Group
|
|
Subgroup Analyses
The population participating in this experiment will be low-income families in Cook County. There may be measurable differences in impacts among subgroups. We believe the most important of these will be between new applicants and those at recertification, but there might also be differences by age of youngest child, or by race and ethnicity, which are two important determinants of parental decisions with regard to both child care modal choice and maternal employment. We note that the two standard approaches to subgroup analysis—including interaction terms in the regression model, and estimating separate models for the groups—test different hypotheses. Interaction terms find effects for members of a subgroup holding other factors constant—e.g., the effect of ethnicity on outcomes, conditional on receipt of cash assistance. Estimating separate models finds effects for subgroups given their different characteristics—e.g., the effects of race and ethnicity on outcomes, given that some groups are more likely to receive cash assistance than others. We expect that the latter approach would be of more interest to policymakers.
Causal Pathways
If significant differences in outcomes are found, it will also be of interest to determine the paths through which these impacts occurred. Although the pathways cannot be traced with as much confidence as the assessment of impacts, joint examination of proximal and distal outcomes may yield suggestive results. Suppose, for example, it is hypothesized that the experiment improved a distal outcome (reliability of child care arrangements) by altering a proximal outcome (mode of care). Evidence in support of this hypothesis would be that:
Children’s care was more reliable on average in the treatment group;
Children in the treatment group were more likely to be in more formal modes of care; and
Within the treatment and control groups, children in more formal modes of care had more reliable care.
This sort of evidence would suggest that the treatment had changed the allocation of children among modes of care, from those that were less reliable to those that were more reliable.
Implementation Study
To meet its objectives within the context of the overall evaluation of the demonstration, the Implementation Study includes a variety of analytic approaches. In this section, we summarize our approach to analysis for the Implementation Study of the Illinois demonstration.
Describing the Demonstration and Contextual Factors
These study goals are addressed mainly through two types of descriptive strategies: narrative description and descriptive statistics. The narrative description consists in weaving together a coherent “story of the demonstration project” from the information gathered in interviews, focus groups, and through documentary materials. For this study, the “story” should include the following elements:
Initial implementation of demonstration policies—key actors in implementing demonstration; reasons for participating in the demonstration; startup experiences and problems; solutions to problems; time taken to reach stable operations.
Ongoing administration of demonstration policies for both treatment and control group parents—demonstration operations, including methods of informing demonstration parents and providers; subsidy eligibility determination and recertification processes; random assignment; differences in administration of treatment and control group parents and providers; methods of maintaining research group status.
A first step in building the “story of the program” is synthesizing the varied accounts of the program collected through a select number of interviews with key staff, and through the research staff’s own documentation of the process. Because the demonstration is fairly basic (i.e., providing subsidies to families over the current eligibility income) and its administration largely relies on the existing administrative processes, which are well-documented, we believe that the Implementation Study will be straightforward.
The analysis and presentation of a description of the contextual factors in Cook County will use descriptive statistics, which are simply the measurement of some set of variables across a population or groups within a population. We will be using descriptive statistics to document demonstration participant characteristics and participation patterns, as well as many contextual factors.
Assessing the Faithfulness of the Test
The study will assess the faithfulness of the test in two ways. First, we will describe relevant demonstration operations, including informing families of potential eligibility for the demonstration; administering consent forms; random assignment; application of appropriate policies to treatment and control group families; and maintaining group assignments over the life of the demonstration. Second, we will analyze IDHS administrative data to verify differential treatment of treatment and control group members.
Cost-Benefit Analysis
The cost-benefit analysis will combine results from the impact and implementation analyses with expenditure information, to determine whether the costs of the experimental programs are justified given their impacts. The foundation for the analysis will be the framework shown in Exhibit A16.4 that indicates the components we plan to include in the analysis (the rows) and the perspectives from which they will be assessed (the columns). The pluses and minuses following the name of each component indicate whether that item is expected to increase or decrease; the pluses and minuses in the body of the table indicate whether that change represents a benefit or a cost from each perspective. When the analysis is performed, the table will be filled in with incremental measures, i.e., treatment-control group differences. Separate analyses will be performed for each of the two variants of each experiment.
The families and children perspective will identify gains and losses to families and children in the treatment group relative to the control group. These can occur through changes in:
Parental employment, earnings, and fringe benefits;
Receipt of means-tested benefits such as TANF, food stamps, public housing, and Medicaid;
Amount paid out of pocket for child care; and
Quality of child care, along such dimensions as flexibility, reliability, and stability.
The government and society perspectives identify gains and losses to federal, state, and local government budgets associated with the treatment, and also non-monetary benefits due to taxpayers’ preferences that funds be spent on programs (such as child care subsidies) that support work, rather than on cash welfare.
We discuss measurement of each group of costs and benefits below.
Exhibit A16.4
Cost-Benefit Framework
|
||
|
Anticipated Effects on: |
|
Expected Impacts |
Families and Children |
Government and the Taxpayers |
Employment and Earnings |
|
|
Hours worked (+) / foregone leisure & time for family responsibilities () |
|
+ |
Earnings after taxes (+) |
+ |
|
Fringe benefits (+) |
+ |
|
|
|
|
Means-tested income |
|
|
TANF benefits () |
|
+ |
Public housing subsidies () |
|
+ |
Other programmatic income () |
|
+ |
|
|
|
Taxes/Transfers |
|
|
Income and sales taxes (+) |
|
+ |
EITC payments (+) |
+ |
|
|
|
|
Child care costs |
|
|
Out-of-pocket child care costs () |
+ |
|
Subsidy payments (+) |
|
|
Program administrative costs (+) |
|
|
|
|
|
Quality of child care |
|
|
Flexibility (+) |
+ |
|
Reliability (+) |
+ |
|
Stability (+) |
+ |
|
|
|
|
Net cost/benefit |
+ |
|
Employment and Earnings
Adults in the treatment group families are expected to work more hours. This represents a loss to them in terms of forgone leisure that could have been spent on other activities. Taxpayers are assumed to consider the increased employment of participating families to be a benefit. The increased earnings and fringe benefits associated with the increased employment are benefits to families.
Although earnings will be measured in dollars, the other components in this category will be in natural units: hours per week, likelihood of paid vacation and holidays, etc.
Means-Tested Income
Families are expected to reduce their reliance on TANF and other means-tested programs. These changes represent a loss to them and a gain to taxpayer budgets. TANF effects can be measured in dollars.
Taxes and Transfers
The higher income received by families will lead to greater tax payments by them—a loss to them, a gain to other taxpayers. This will probably be countered by an increase in EITC payments (although the rules are such that if a family achieves more than some level of earnings, its EITC payments will start to decline). These costs and benefits are all measured in dollars.
Child Care Costs
Families’ out-of-pocket expenditures will probably, but not certainly, decline. It is expected that the subsidy will substitute for part of what families pay. It is possible, nonetheless, that the availability of a subsidy will induce some families to move from “free” care provided by household members or other relatives to care that requires some co-payment, so that their out-of-pocket costs will increase. Furthermore, if loss or absence of a subsidy keeps a control group mother from working, she will incur no out-of-pocket expenditures while her treatment group counterpart is making copayments.
The subsidy payments themselves, of course, represent a cost to taxpayers that will be greater for the treatment than the control group. In addition, taxpayers must bear the incremental cost of administering the expanded subsidy program. The costs and benefits in this category are all measured in dollars.
Quality of Child Care
Finally, it is to be hoped that the families in the treatment group obtain better quality child care. Three dimensions that will be measured on the follow-up survey are flexibility, reliability, and stability. These will be measured in natural units (e.g., percent of respondents that rate their child care arrangements as very/somewhat flexible, very/somewhat reliable; percent of focus children that were in the same primary care arrangement for an entire school year).
Net Cost/Benefit
The monetary costs and benefits will be summed for both families and taxpayers, and the non-monetary effects borne in mind. It is anticipated that the net effect of the intervention for families is positive; they are being offered benefits for which they would otherwise not be eligible. Presumably they can therefore be no worse off.
Taxpayers will almost certainly be worse off, in the narrow sense that expanded child care subsidies, with their associated administrative costs, will not pay for themselves. There may be small offsets to these costs in terms of increased tax receipts and lower TANF payments, but even these may be reduced by higher EITC payments. Nonetheless, taxpayers may consider themselves better off in that it is preferable to spend tax dollars on work supports than on TANF.
Study Schedule
The planned time schedule for the study is as follows:
Recruitment and random assignment begins March 2005
First Implementation Study visit March 2006
Random assignment ends March 2006
Expected OMB approval May 2006
Survey at 8 months May 2006 –February 20077
Survey at 16 months July 2006 – June 2007
Second Implementation Study visit August 2006
Survey at 24 months March 2007 – February 2008
Last subsidy payment is made under
enhanced income eligibility ceiling February 2008
Final report August 2008
c. Washington
Analysis Plan
There are two major sets of statistical analyses that will be completed as part of the evaluation: impact analyses and cost-benefit analyses. The analysis plan for each is provided below.
Impact Analysis
The purpose of the impact analysis is to estimate the effects of the alternative co-payment structure on a number of outcomes. The random assignment design enables us to identify the net effect of subsidies on these outcomes by letting us draw comparisons between the treatment and control group.
The types of outcomes we will assess fall into three general categories:
Employment and income outcomes (e.g., months in which employment occurred, average numbers of weeks worked, stability of employment, average monthly earnings, changes in earnings levels, average change in total family income)
Child care outcomes (e.g., reliability, flexibility, and stability of child care arrangements, type of care, hours in care, numbers of child care problems, cost of the child care, average amount parents spend on child care)
Public assistance outcomes (e.g., percent with any use of TANF assistance, average months of use of TANF, percent with any use of food stamps, average months use of food stamps)
Child care subsidy outcomes (e.g., average length of time on subsidy program; average amount of state subsidy payments per child and per family)
Random assignment ensures the internal validity of the analysis based on this comparison: we can assume that the two groups are equivalent, and that the outcomes observed for the control group in each experiment represent on average the outcomes that would have been experienced by members of the treatment group, absent the experiment.
Although simple comparisons of means and proportions would yield consistent impact estimates, more precision can be obtained by use of multivariate methods. Within each group, outcomes will vary because of other underlying factors, such as mother’s employment opportunities, children’s ages, and so on. It is easier to detect differences in central tendencies between two groups if each group is more homogeneous, i.e., if their outcomes are more tightly clustered. Regression-adjusted outcomes are indeed more tightly clustered to the extent that the covariates have explanatory power. For example, a partial R2 of 0.25 due to the covariates implies that the variance of the regression-adjusted mean is only three-quarters as large as the variance of the raw mean, and this reduction correspondingly increases the statistical power of the comparison.
Regression adjustment will therefore be used for each outcome, with a technique appropriate to the form of the outcome: linear regression for continuous outcomes such as quarterly earnings, logistic regression for dichotomous outcomes such as an indicator that the family received any TANF income. The goal of these models is not to “explain” the outcomes; these will be reduced-form models and the coefficients, other than those on the treatment indicators, will in general be uninterpretable. Baseline data will include information on the ages and relationships of family members, work and training activities of other parents or stepparents in the household, ethnicity and citizenship of children for whom subsidies are sought, family participation in TANF and other federal cash income programs, and child support received.
The form of the regression equation for a continuous outcome will therefore be:
yi = b0 + (b1 × Ti ) + Σj (b3j × Xij) + εi,
where yi = value of outcome for family i,
Ti = indicator that family i is in the treatment group,
Xij = value of baseline variable j for family i, and
εi = residual.
The parameter b1 is the effect of the treatment. Results will be presented in tables as in Exhibit A16.5.
Exhibit A16.5
|
|||
|
Control Group Mean |
Treatment Group |
|
|
Regression-Adjusted Mean |
Estimated Impact |
|
Employment (%) |
|
|
|
Any covered employment, Q1-Q8a |
|
|
|
Any employment, months 1-24b |
|
|
|
(etc.) |
|
|
|
|
|
|
|
Earnings ($) |
|
|
|
Total UI earnings, Q1-Q8a |
|
|
|
Total reported earnings, months 1-24b |
|
|
|
(etc.) |
|
|
|
|
|
|
|
Sample size |
|
|
|
UI records |
|
|
|
Follow-up survey |
|
|
|
Subgroup Analyses
We expect that the alternative co-payment scale may result in differential impacts according to subgroups, including the income level of the family as well as whether the family is newly receiving subsidies or is a continuous subsidy user. We note that the two standard approaches to subgroup analysis—including interaction terms in the regression model, and estimating separate models for the groups—test different hypotheses. Interaction terms find effects for members of a subgroup holding other factors constant—e.g., the effect of ethnicity on outcomes, conditional on receipt of cash assistance. Estimating separate models finds effects for subgroups given their different characteristics—e.g., the effects of ethnicity on outcomes, given that some groups are more likely to receive cash assistance than others. We expect that the latter approach would be of more interest to policymakers.
Causal Pathways
If significant differences in outcomes are found, it will also be of interest to determine the paths through which these impacts occurred. Although the pathways cannot be traced with as much confidence as the assessment of impacts, joint examination of proximal and distal outcomes may yield suggestive results. Suppose, for example, it is hypothesized that the experiment improved a distal outcome (stability of child care arrangements) by altering a proximal outcome (mode of care). Evidence in support of this hypothesis would be that:
Children’s child care arrangements were more stable on average in the treatment group;
Children in the treatment group were more likely to be in more formal modes of care; and
Within the treatment and control groups, children in more formal modes of care had more reliable care.
This sort of evidence would suggest that the treatment had changed the allocation of children among modes of care, from those that were less reliable to those that were more reliable.
Implementation Study
To meet its objectives within the context of the overall evaluation of the demonstration, the Implementation Study includes a variety of analytic approaches. In this section, we summarize our approach to analysis for the Implementation Study of the Washington demonstration.
Describing the Demonstration and Contextual Factors
These study goals are addressed mainly through two types of descriptive strategies: narrative description and descriptive statistics. The narrative description consists in weaving together a coherent “story of the demonstration project” from the information gathered in interviews, focus groups, and through documentary materials. For this study, the “story” should include the following elements:
Initial implementation of demonstration policies—key actors in implementing demonstration; reasons for participating in the demonstration; startup experiences and problems; solutions to problems; time taken to reach stable operations.
Ongoing administration of demonstration policies for both treatment and control group parents—demonstration operations, including methods of informing demonstration parents and providers; subsidy eligibility determination and recertification processes; random assignment; differences in administration of treatment and control group parents and providers; methods of maintaining research group status.
A first step in building the “story of the program” is synthesizing the varied accounts of the program collected through a select number of interviews with key staff, and through the research staff’s own documentation of the process. Because the demonstration is fairly basic (i.e., using an alternative schedule to calculate co-payments) and its administration largely relies on the existing administrative processes, which are well-documented, we believe that the Implementation Study will be straightforward.
The analysis and presentation of a description of the contextual factors in Washington will use descriptive statistics, which are simply the measurement of some set of variables across a population or groups within a population. We will be using descriptive statistics to document demonstration participant characteristics and participation patterns, as well as many contextual factors.
Assessing the Faithfulness of the Test
The study will assess the faithfulness of the test in two ways. First, we will describe relevant demonstration operations, including informing families of potential eligibility for the demonstration; administering consent forms; random assignment; application of appropriate policies to treatment and control group families; and maintaining group assignments over the life of the demonstration. Second, we will analyze state administrative data to verify differential treatment of treatment and control group members.
Benefit-Cost Analysis
The benefit-cost analysis will combine results from the impact and implementation analysis with expenditure information, to determine whether the costs of the experimental programs or policies are justified, given their positive impacts.
The benefits and costs of each of the experiment will be assessed from several perspectives, including families and children, the government, tax payers, and society.
From the perspective of families and children, we will identify the gains and losses to families that result from co-payment amounts and use or loss of subsidies, in terms of parental employment and child care stability. We also will identify changes in the use of public assistance, families’ child care costs, and total family income. Because the study will provide parents with lower (or the same) co-payment amount and because all parental changes in employment, child care choices, and other behavior will be completely voluntary, the intervention should be beneficial to families overall and should not harm any families. However, the benefit to families might be partially offset by costs such as increased taxes and reduced public assistance and leisure time.
From the government perspective, we will tally benefits and costs to federal, state, and local governments. The benefits of child care subsidies could include increased tax revenue as a result of increased employment of families that receive subsidies and reduced use of public assistance. The primary costs to government will be the actual subsidy payments for families who use subsidies longer than they would with a higher co-payment, and the higher state payment made to compensate for the lower co-payment amount in the alternative co-payment scale.
From the taxpayers’ perspective, we will tally the benefits and costs to the general public. The taxpayer will benefit from increased tax revenue since it offers the possibility of greater services or lower taxes.
The perspective of society as a whole combines the perspectives of families and other taxpayers. A net gain to society occurs when a gain to one taxpayer is not a loss to other taxpayers.
The foundation for the analysis will be the framework shown in Exhibit A16.6 that indicates the components we plan to include in the analysis (the rows) and the perspectives from which they will be assessed (the columns). The pluses and minuses following the name of each component indicate whether that item is expected to increase or decrease; the pluses and minuses in the body of the table indicate whether that change represents a benefit or a cost from each perspective. When the analysis is performed, the table will be filled in with incremental measures, i.e., treatment-control group differences. Separate analyses will be performed for each of the two variants of each experiment.
The families and children perspective will identify gains and losses to families and children in the treatment group relative to the control group. These can occur through changes in:
Parental employment, earnings, and fringe benefits;
Receipt of means-tested benefits such as TANF, food stamps, and public housing;
Amount paid out of pocket for child care; and
Quality of child care, along such dimensions as flexibility, reliability, and stability.
The taxpayer and government perspectives identify gains and losses to federal, state, and local government budgets associated with the treatment, and also non-monetary benefits due to taxpayers’ preferences that funds be spent on programs (such as child care subsidies) that support work, rather than on cash welfare.
We discuss measurement of each group of costs and benefits below.
Exhibit A16.6
Cost-Benefit Framework |
||
|
Anticipated Effects on: |
|
Expected Impacts |
Families and Children |
Taxpayers and Government |
Child care costs |
|
|
Out-of-pocket child care costs () |
+ |
|
Subsidy payments (+) |
|
|
Program administrative costs (+) |
|
|
|
|
|
Child Care Quality |
|
|
Flexibility (+) |
+ |
|
Reliability (+) |
+ |
|
Stability (+) |
+ |
|
|
|
|
Employment and Earnings |
|
|
Hours worked (+) / foregone leisure () |
|
+ |
Earnings after taxes (+) |
+ |
|
Fringe benefits (+) |
+ |
|
|
|
|
Means-tested income |
|
|
TANF benefits () |
|
+ |
Public housing subsidies () |
|
+ |
Other programmatic income () |
|
+ |
|
|
|
Taxes/Transfers |
|
|
Income and sales taxes (+) |
|
+ |
EITC payments (+) |
+ |
|
|
|
|
Net cost/benefit |
+ |
|
Child Care Costs. Families’ out-of-pocket expenditures will decline or stay the same at any given income level. However, if treatment group families work more than their control group counterparts, they might incur greater out-of-pocket costs for child care if the increased work hours result in them moving from part-time to full-time care. However, the alternative co-payment schedule has been designed so that the after-tax income of families in the treatment group will increase if they choose to work more.
If subsidy payments are increased because co-payments are reduced, this represents a cost to taxpayers. If the alternative co-payment schedule encourages parents to work more hours and earn more, this cost might be reduced or reversed. The costs and benefits in this category are all measured in dollars.
Quality of Child Care. Reducing co-payments might encourage families in the treatment group to use better quality child care. Three dimensions that will be measured on the follow-up survey are flexibility, reliability, and stability. These will be measured in natural units (e.g., percent of respondents that rate their child care arrangements as very/somewhat flexible, very/somewhat reliable; percent of focus children that were in the same primary care arrangement for an entire school year).
Employment and Earnings. Reduced co-payments might encourage adults in treatment group families to work more hours. This represents a gain to them in earnings and fringe benefits although the gain may be offset somewhat by a loss of forgone leisure. Taxpayers are assumed to consider the increased employment of participating families to be a benefit.
Although earnings will be measured in dollars, the other components in this category will be in natural units: hours per week, likelihood of paid vacation and holidays, etc.
Means-Tested Income. If the intervention encourages parents to work more, it will also reduce their reliance on TANF and other forms of public assistance. These changes represent a loss to them and a gain to taxpayer budgets. (It is important to note that the alternative co-payment schedule was set so that family’s income will increase as their hours of work increase, even if their public assistance benefits are reduced). Public assistance amounts will be measured in dollars.
Taxes and Transfers. If the intervention encourages parents to work more, they will also pay more in payroll taxes and might pay more in income taxes, although they might receive greater refunds from the federal Earned Income Tax Credit. An increase in tax payments represents a loss to families and a gain to other taxpayers, while an increase in EITC payments represents a gain to participating families and a loss to other taxpayers (assuming that taxpayers do not value the redistribution of income). (It is important to note that the alternative co-payment schedule was set so that families’ income will increase as their hours of work increase, even if they pay more in taxes). These costs and benefits are all measured in dollars.
Net Cost/Benefit. The monetary costs and benefits will be summed for both families and taxpayers, and the non-monetary effects borne in mind. The net effect of the intervention for families is expected to be positive since all parental decisions are completely voluntary. Presumably they can therefore be no worse off.
Taxpayers might be worse off or better off. Expanded child care subsidies, with their associated administrative costs, will represent a cost to taxpayers. These might be offset to some extent by increased tax receipts and lower public assistance benefits. Regardless, taxpayers might consider themselves better off in that it is preferable to spend tax dollars on work supports than on TANF.
Study Schedule
The planned time schedule for the study is as follows:
Recruitment and random assignment period October 2005
Expected OMB approval May 2006
Survey at 8 months June 2006
Survey at 16 months March 2007
Survey at 24 months September 2007
Last co-payment is made under reduced co-payment schedule September 2007
Final Report March 2008
A17 Display of Expiration for OMB Approval
A space for the OMB approval number and expiration date is indicated at the top of the cover page for each instrument submitted.
A18 Exception to the Certification Statement Identified in Item 19.0 of Form OMB 83-I
None.
Part
B
Collection of Information Using Statistical Methods
B
1 Sample
Universe, Sampling Method and Expected Response Rates
a. Massachusetts
Sample Universe
The target population includes family child care providers who are licensed, part of a network, and stable (i.e., in business for at least two years). The study is being conducted with a sample of family child care providers from family child care networks in the state who have indicated interest in and the capacity to implement Learningames. A statewide sample of such family child care homes is desired in order to obtain results that are applicable to the state as a whole.8 Choosing a sample from only part of the state would yield results that are representative of those parts of the state, but substantial differences in economic and personal circumstances of family child care providers and families in different parts of the state would mean the results would be of less use to the State. However, in order to increase efficiency and reduce costs we will try to cluster the sample of providers within a relatively small number of regions of the state. The study will include children in family child care homes enrolled in the home who, at the start of the study, are 36 months of age or younger.
Sampling Method
Within each region included in the sample, we will recruit family child care networks that can contribute at least 10 homes to the study (e.g., they have 10 homes that will volunteer to participate, and each has been in business for two years and cares for two children under 36 months of age). The number of networks participating in the study will vary by region. Randomization will occur within each family child care network so that all participating agencies are guaranteed to have half of their homes be in the Learningames group.
Within these providers’ homes, we will study the development of preschool children. Since the study is longitudinal, following the same providers over two years, we will include in the study children in the homes who are 36 months old or younger who either (a) are in the homes at the outset of the evaluation or (b) who enroll in the homes during the first 18 months of the study. This “rolling” sampling strategy will help increase our chances of having an adequate sample of children for the impact analyses on child outcomes. We will close study enrollment to new children six months before the end of the evaluation period so that all children evaluated at the final assessment point will have been in the home at least six months.
The children will be in the family child care homes for differing amounts of time. Some children will enter the home during the study period and others will leave. At the end of the two years of the evaluation, the analyses of child impacts will first analyze the average-age standardized score on the measure of language development across all children clustered within the home. Second, we will examine the impacts for different age groups of children, assuming the final sample includes sufficient numbers of children in the relevant age categories. We propose to divide the sample into four age groups, based on age of child at the completion of the study or at the last testing point before the child leaves the home:
under 12 months,
12-23 months,
24-35 months, and
36 – 60 months.
Sample size is determined by our desire to measure child outcomes as well as provider behavior. We will net approximately 350 providers, 175 treatment and 175 control. We assume that each provider will have at least two children in the sample. This sample size allows us to detect effects on children and on providers of 0.23 standard deviations.9
b. Illinois
Sample Universe
The sample universe for the experiment includes all families with income between 50 and 65% of SMI who apply for child care subsidies in Cook County, Illinois either as new applicants or at recertification. In addition, families must meet a number of other conditions to be eligible to participate in the study. They must comply with all other requirements for receiving the subsidy, such as using a provider who has been approved by the state to receive subsidies. In addition, they must (1) provide a valid Social Security number (SSN) so that we can retrieve administrative records on other forms of public assistance and earnings reported to the unemployment insurance (UI) system; (2) provide informed consent; (3) live in Cook County; and (4) not work for the subsidy administering agency (Action for Children, AFC). Finally, the applicant must not be a member of a group that is currently certified for subsidies for less than six months: the self-employed, those paid in cash, teachers or students, and employees of temporary agencies.
Sampling Method
Exhibit B1.1 presents the expected samples sizes for the experiments, by subgroup. A total of 2,000 families will be randomly assigned—1,000 to the treatment group and 1,000 to the control group. Within the treatment group, half of the sample will be a 6-month authorization period (at which point they need to reapply or be “redetermined”). The other half will be assigned a 12-month authorization period. The size of the sample will allow us to be able to detect an effect of 0.11 standard deviations for impacts measured using administrative records—for example, a difference in employment from 50 percent in the treatment group to 55 percent of the combined treatment group—with power of 0.80. It will also allow us to detect effects of .125 standard deviations for outcomes measured using the follow-up surveys assuming 80 percent of participants respond to the survey. This assumes no regression adjustment in the impact calculations, and also does not correct for making multiple comparisons.
Exhibit B1.1
Number of Families to Be Included in Each Experiment, by Treatment Status
|
||
|
Families At Application |
Families At Reapplication |
Control group |
500 |
500 |
|
|
|
Treatment group |
|
|
12-month certification period |
250 |
250 |
6-month certification period |
250 |
250 |
Recent data indicate that about 200 applicant families per month are being recruited into the sample. AFC, the local child care resource and referral agency that is administering the subsidies, is recruiting the sample for the experiment.
The sample is being collected in Cook County, Illinois, which includes the Chicago and its suburban areas. We will obtain de-identified subsidy records for the study period for Cook County as well as the entire state. Using the records of families between 45-50% of SMI (i.e., families just below the state’s eligibility ceiling for subsidies) we will compare the participants in the treatment group to those in the rest of the state in terms of family composition, child care selection, and employment characteristics. This will help us to ascertain roughly the similarities and differences between (1) the study group and similar families whose incomes make them eligible in Cook County; and (2) the study group and such Illinois families who do not reside in Cook County.
c. Washington
Sample Universe
The study identified all households with adult parents (i.e., those over 18 years of age) statewide who were approved to receive child care subsidies for a three-week period in October and November 2005. This period was sufficient to build up the desired sample of 2,000 families in the treatment group. (In order to have equal numbers of families in the treatment groups at each of the three tiers of the current eligibility scale, there are 3,142 families in the control group, since families at the highest income tier do not apply for subsidies at the same rate as those at the lowest tier.) The study has excluded the following groups: (1) those whose youngest child is age 11, as they will not be eligible for a subsidy after their child reaches age 13, and therefore will not be eligible for the subsidy; and (2) households headed by non-needy adults, as they will always be assigned the minimal co-payment. A statewide sample was desired in order to obtain results that are applicable to the state as a whole. Choosing a sample from only part of the state would have yielded results that are representative of those parts of the state, but substantial differences in economic and personal circumstances of subsidy recipients in different parts of the state would mean the results would be of less use to the State.
Sampling Method
Exhibit B1.2 describes the following recruitment process and information provided to study participants. In summary:
5,142 families were assigned either to the alternative or standard co-payment.
All families were sent a notice describing the study and providing them with an opportunity to withhold contact information. The State will process and link administrative records for these families and provide them to the research team without identifying information. All identifiers indicated in 45 CFR 164.514 will be removed from the data for these families before they are shared with the Abt research team.
The contact information (name and address) for a group of approximately 2,500 of the remaining families will be sent to Moore and Associates, the survey firm. These families (divided equally between the treatment and control groups) will be sent a letter by researchers reminding them that they will be contacted. The families not contacted will not be asked for consent for release of their administrative records. Administrative data about families not contacted will also be stripped of identifiers and used for analysis.
If consent is given, families will be interviewed at three points in time. The administrative records for those participants that have been contacted and agree to have them released will be linked to the survey information by Abt and used for analysis by Abt and MDRC. The State will not have access to survey information with identifying information. We estimate that 1,000 families from the treatment group and 1,000 families from the control group will be in this category.
It is important to be clear about what study participants can and cannot opt out of.
Study participants cannot opt out of being assigned to the alternative co-payment. Making co-payments is one of the requirements of receiving a child care subsidy and Washington Department of Child Care and Early Learning (DCCEL) determine their level. Study participants can implicitly opt out of the alternative, lower co-payment schedule in one of two ways. First, they could decide not to use the subsidy for which they were approved. Second, they could voluntarily pay the higher amounts of the standard schedule. Although this violates DCCEL regulations, in practice there would be no way to know whether this happened.
Study participants cannot opt out of having DCCEL collect and analyze data on subsidy use and receipt of TANF and food stamps. DCCEL collects and analyzes these data as part of its normal operations. In addition, DCCEL can collect and analyze data from the unemployment insurance system in accord with whatever agreements are reached between DCCEL and the Department of Employment Security.
Study participants who have been contacted and withheld consent can opt out of having these data shared with Abt Associates and MDRC. However, the research design includes analysis by Abt and MDRC of administrative data records only for the 3,000 families who are not contacted and asked for the release of records. These records will be stripped of personal identifying information.
Study participants can opt out of having contact information sent to the study team. They can opt out of the survey and can refuse to answer any questions on the survey. As described above, if they are contacted and ask, they can also opt out of having their administrative records shared with the evaluators.
Exhibit B1.2 |
|
Proposed Sample Size and Power of Planned Statistical Tests
Using random assignment allows us to estimate the effects of lower co-payments by comparing average outcomes for families in the treatment and control groups. With 2,000 families in the treatment group and 3,142 families in the control group, such comparisons will give us an 80 percent chance of detecting impacts of .07 standard deviation using one-tailed t-tests at the 0.05 significance level for outcomes based on administrative records (for which we will have information on all families in the study, stripped of identifiers in those cases where we did not receive approval for the release of records). We might be able to boost the power of these tests by adjusting for the families’ baseline characteristics. An effect of .08 standard deviations is a 3 percentage point change in employment or subsidy receipt if we expect the typical control group family to work or receive subsidies three-quarters of the time during the two years following random assignment.
For measuring the effects of the intervention for families in different income tiers, the sampling strategy would allow us to detect effects of .11 standard deviations for families in income Tier 1, .12 for families in Tier 2, and .14 for families in Tier 3. For example, this would allow us to detect an increase in employment or subsidy use from 75% of the control group to 80-81% of the treatment group for each subgroup.
As described above, the study will also conduct follow-up surveys with 1,000 treatment group families and 1,000 control group families. For outcomes based on these follow-up surveys, the minimum detectable effects will be 41% larger than the minimum detectable effects described above for outcomes based on administrative records. In no case is the minimum detectable sample size larger than .20.
Exhibit B.1.3 shows sample sizes and minimum detectable (mde)effects for each of the three income tiers in Washington as well as the total. Recall that Tier 1 contains families below 82 percent of the federal poverty level (in which all families have a copayment of $15), Tier 2 contains families with income between 82 percent and 137.5 percent of the federal poverty level, and Tier 3 contains families with higher income. The top half of the exhibit shows mdes for the full sample, which would be used in estimating effects using administrative data, and the bottom half of the exhibit shows mdes for the survey sample. For purposes of this exhibit, we assume the survey sample would be drawn equally from the three tiers.
Exhibit B.1.3
Sample Size and Minimum Detectable Effects for Income Tiers in Washington Child Care Study
|
Tier 1 |
Tier 2 |
Tier 3 |
Total |
Full sample |
|
|
|
|
Control group |
1,292 |
1,073 |
777 |
3,142 |
Treatment group |
524 |
710 |
766 |
2,000 |
Minimum detectable effect size (standard deviations) |
0.13 |
0.12 |
0.13 |
0.07 |
Minimum detectable effect on employment (percentage points) |
6.4 |
6.0 |
6.3 |
3.6 |
Survey sample |
|
|
|
|
Control group |
333 |
333 |
334 |
1,000 |
Treatment group |
333 |
333 |
334 |
1,000 |
Minimum detectable effect size (standard deviations) |
0.19 |
0.19 |
0.19 |
0.11 |
Minimum detectable effect on employment (percentage points) |
9.6 |
9.6 |
9.6 |
5.6 |
Notes: Power calculations were based on one-tailed t-tests at the 5 percent significance level and assumes that baseline covariates will not increase the precision of estimated effects. The calculations assumed a 50 percent employment rate for the control group.
B2 Data Collection Strategy
a. Massachusetts
Four kinds of measures will be collected for the evaluation: systematic observations of provider behavior; standardized assessments of children’s development; a provider questionnaire; and a home visitor questionnaire.
Exhibit B2.1 shows the categories of data to be collected, data sources, time-period for collection and analyses in which they will be used.
Exhibit B2.1 Overview of Data Needs and Data Sources |
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Child characteristics Age, gender, home language, length of time in care setting |
|
|
|
|
|
|
|
Provider characteristics Age, ethnicity, education, training, experience, job motivation |
|
|
|
|
|
|
|
Home environment Health and safety, support for cognitive, language, social-emotional development, equipment materials |
|
|
|
|
|
|
|
Provider behaviors and interactions
|
QUEST caregiver rating |
|
|
Child outcomes
|
|
|
|
|
|
|
|
Home visitor characteristics Education, training, experience, caseload size, frequency and duration of home visits, job responsibilities
|
|
|
|
Observations of Providers
Baseline data will be collected by the staff of the family child care networks using the QUEST form and trained by Abt staff. Study staff, hired by Abt Associates, will collect similar observation data using QUEST and FDCRS six months after the intervention begins, and again at 12 and 24 months. At each observation point, providers will be for approximately 2.5-3.0 hours. The observations will use a standardized rating system. All observers trained to reliability by the Abt staff.
Child Assessments
Baseline information on the developmental status of children in the study will be drawn from extant data collected by the participating family child care systems for children who are in the homes at the outset of the study, and for children who enter the homes at a later date and up to six months before the study ends. Similar data will be obtained for children who are under 36 months and enroll in the family child care home after the study begins and up to six months before the end of the study. The evaluation team will collect assessment data at three points over the two years, on the same schedule as for the observation data. These assessments will use a standardized measure, the PLS-4 described in an earlier section. For those children 3 years and older, the test will be administered individually to the children by study staff, at the family child care homes.
Provider Questionnaire
A provider questionnaire will be administered by Abt study staff in June 2006 and January 2008. The initial questionnaire will obtain information on the background and educational and training experience, and motivation of the providers. The second questionnaire will focus specifically on additional education and training obtained over the two years, beyond that offered by Learningames.
Home Visitor Questionnaire
A questionnaire for home visitors will be distributed by Abt study staff in June 2006. The questionnaire will collect data on education and training, caseload size, job responsibilities, frequency and duration of home visits.
b. Illinois
Exhibit B2.2 presents a summary of our data collection strategy. Our main sources of data are extant administrative data and documents, the parent interview as described in sections above, and unstructured interviews with state officials and child care experts.
Exhibit B2.2
Overview of Data Needs and Data Sources
|
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Family and household characteristics (e.g., family size, number of parents, number and ages of children) |
|
|
|
|
|
|
|
Employment and educational characteristics (e.g., number of employers, employment hours and schedules, earnings, school attendance) |
|
|
|
|
|
|
|
Family income (e.g., total household income, child support received, household income from employment) |
|
|
|
|
|
|
|
Public assistance use and costs (e.g., use of TANF cash assistance, use of food stamps, administrative costs of subsidy receipt) |
|
|
|
|
|
|
|
Child care characteristics (e.g., number of children receiving child care, type of subsidized arrangements, schedule of arrangement, child care subsidy costs, administrative costs, family costs) |
|
|
|
|
|
|
|
Planning and start up (e.g., demonstration design, rationale, target groups, intended impacts, planning Implementation, start-up experiences, etc.) |
|
|
|
|
|
|
|
Demonstration operations (e.g., client flow through random assignment, levels and patterns of participation) |
|
|
|
|
|
|
|
Site-related contextual factors (e.g., local child care market conditions, local economic conditions, expectations about subsidy use among low-income families) |
|
|
|
Data Collection for the Implementation Study
The Implementation Study will rely on information from the baseline, administrative, and survey data (Exhibit B2.2). In addition, this part of the subsidy evaluation will rely on a range of open-ended interviews and document reviews. These are described briefly below.
On-Site Data Collection
Open-ended interviews, as well as the collection of various documents, will take place on-site through two field visits over the course of the demonstration. During the visits, researchers will conduct individual and small-group interviews with State DCCEL and DSHS management and staff, and local DSHS management and staff. Researchers will also use both visits to observe demonstration operations.
Exhibit B2.4 Data Collection Strategies for the Implementation Study |
|
Data Source |
Collection Strategy |
Demonstration providers Parents |
Small group open-ended interviews Follow-up surveys (as part of the impact analysis) |
State DSHS staff |
Individual and small group open-ended interviews |
Local DSHS staff |
Individual and small group open-ended interviews |
DSHS administrative data |
Periodic files provided by DCCEL (as part of impact analysis) |
DSHS statistical reports |
Periodic requests to DCCEL |
Demonstration plans and design |
Requests for DCCEL planning documents; MOAs between Abt Associates and DCCEL |
Subsidy system policy manuals and eligibility forms |
Requests to DCCEL |
Census information |
U.S. Census |
BLS labor market data |
BLS publications (hard copy and online) |
Individual and Small Group Open-Ended Interviews
Much of the descriptive information about intervention design, planning, and implementation, as well as about the context in which the demonstration will operate, will come from individual and small-group interviews with key informants during the first site visit to Washington. The Implementation Study will include open-ended interviews with the following informant state and local subsidy and public assistance agency staff.
Researchers will use interview guides that will be developed for each type of informant. The open-ended interviews will be conducted individually or in small groups of up to three informants. An advantage of small-group interviews (where possible) is that although one respondent may forget one or more details, or may answer incorrectly, informants in small groups usually correct one another and can fill in details others may leave out. Because we are primarily interested in "getting the story right," we will try to organize small-group interviews, where possible.
The interview guides will be organized by topic area for each type of informant. Within each topic area, the guides will include basic questions and probes designed to stimulate discussion and more complete information for each topic area. The use of detailed interview guides ensures some level of uniformity across researchers and informants. Also, the guides as annotated by interview notes provide a structure to data collection that readily organizes field notes for analysis and reporting.
Another useful practice in conducting open-ended interviews is to ask respondents the reasons and/or evidence for their judgmental answers. First, this may force informants to think more carefully about their responses and qualify them in the light of their grounds for holding their opinions. Second, it allows the researcher to weigh the informant's opinion against the strength of the evidence used to support it.
Subsidy Agency Statistical Reports
Extant subsidy agency statistical reports will be used to help characterize the child care subsidy market in the demonstration sites. We expect such reports to provide basic information about: subsidy use, including numbers of families, children, and providers; mean subsidy amounts; types of care used.
Subsidy System Policy Manuals and Eligibility Forms
We will collect demonstration site subsidy system manuals and eligibility forms as our primary source of information about subsidy eligibility criteria, subsidy levels, and co-payment amounts and collection processes. The manuals and eligibility forms will also allow some insight into the initial eligibility and recertification processes, although information about those operations will also be collected in the open-ended interviews at demonstration sites.
Census Information
Census Bureau information will be used as a primary data source for information about site demographic and socio-economic characteristics. Using data from census tracts that most closely overlap with the demonstration sites, the Implementation Study will summarize information about demonstration site ethnicity, household number and composition, number of families with children, distribution of children by age, and other relevant contextual factors.
c. Washington
Exhibit B2.3 aligns the categories of data with our data sources and provides the time period during which they will be collected.
Exhibit B2.3 Overview of Data Needs and Data Sources |
|||
Data Needs |
Sources of Data |
Time Collected |
Analyses for Which Data Are Used |
Family and household characteristics (e.g., family size, number of parents, number and ages of children) |
|
|
|
|
|
|
|
Employment and educational characteristics (e.g., number of employers, employment hours and schedules, earnings, school attendance) |
|
|
|
|
|
|
|
Family income (e.g., total household income, child support received, household income from employment) |
|
|
|
|
|
|
|
Public assistance use and costs (e.g., use of TANF cash assistance, use of food stamps, administrative costs of subsidy receipt) |
|
|
|
|
|
|
|
Child care characteristics (e.g., number of children receiving child care, type of subsidized arrangements, schedule of arrangement, child care subsidy costs, administrative costs, family costs) |
|
|
|
Data Collection Strategies
The major sources of data include the parent interview, administrative data and other extant information, and interviews with staff at Washington Department of Social and Human Services (DSHS) and the State’s regional offices. Each of these is discussed briefly below.
Parent Interviews
As stated earlier, parent interviews will be conducted by telephone at 8, 16, and 24 months after random assignment. We do not plan to interview people in their homes. We will attempt to interview about 2,500 families and expect to interview 2,000 families (for an 80 percent response rate). Interviews will be divided about equally between the treatment and control groups. The interview will be a vital source of information for the impact and benefit-cost analyses and parts of it may also be used for the Implementation Study. The interview will provide us with more detailed information about family characteristics than is available in the baseline data, as well as changes that have occurred in some of these characteristics (e.g., the birth of a child, an additional adult moving into the household) since random assignment began. The survey will also be the study’s primary source of information about child care and employment characteristics over the course of the two-year period.
We believe that we can capture changes in employment and child care with sufficient accuracy through a telephone interview. If, at the completion of the first interview at 8 months, it becomes clear that a higher percentage of in-person interviews will be necessary, we would be able to adjust our data collection plan.
Administrative Data
Records from various public assistance programs will be used for the impact and benefit-cost analyses. In general, these records will be used to quantify participants’ use of various forms of public assistance. In addition, information on employment and earnings from Unemployment Insurance records will augment data from parent surveys.
For each automated system, data will be provided one year (understanding data system limitations) prior to and two years following random assignment. If additional funding is obtained, we may seek additional follow-up data. In that case, we would ask for consent for release of identifying information when families are surveyed at the 24-month point. The automated systems include the following:
Child care subsidy amounts and provider information;
TANF authorized grant amounts and dates;
Food Stamps authorized amounts and dates; and
Unemployment Insurance (UI), quarterly wages (earnings), and employer ID numbers.
Data Collection for the Implementation Study
The Implementation Study will rely on information from the baseline, administrative, and survey data (Exhibit B2.4). In addition, this part of the subsidy evaluation will rely on a range of open-ended interviews and document reviews. These are described briefly below.
On-Site Data Collection
Open-ended interviews, as well as the collection of various documents, will take place on-site through two field visits over the course of the demonstration. During the visits, researchers will conduct individual and small-group interviews with State DCCEL and DSHS management and staff, and local DSHS management and staff. Researchers will also use both visits to observe demonstration operations.
Exhibit B2.4 Data Collection Strategies for the Implementation Study |
|
Data Source |
Collection Strategy |
Demonstration providers Parents |
Small group open-ended interviews Follow-up surveys (as part of the impact analysis) |
State DSHS staff |
Individual and small group open-ended interviews |
Local DSHS staff |
Individual and small group open-ended interviews |
DSHS administrative data |
Periodic files provided by DCCEL (as part of impact analysis) |
DSHS statistical reports |
Periodic requests to DCCEL |
Demonstration plans and design |
Requests for DCCEL planning documents; MOAs between Abt Associates and DCCEL |
Subsidy system policy manuals and eligibility forms |
Requests to DCCEL |
Census information |
U.S. Census |
BLS labor market data |
BLS publications (hard copy and online) |
Individual and Small Group Open-Ended Interviews
Much of the descriptive information about intervention design, planning, and implementation, as well as about the context in which the demonstration will operate, will come from individual and small-group interviews with key informants during the first site visit to Washington. The Implementation Study will include open-ended interviews with the following informant state and local subsidy and public assistance agency staff.
Researchers will use interview guides that will be developed for each type of informant. The open-ended interviews will be conducted individually or in small groups of up to three informants. An advantage of small-group interviews (where possible) is that although one respondent may forget one or more details, or may answer incorrectly, informants in small groups usually correct one another and can fill in details others may leave out. Because we are primarily interested in "getting the story right," we will try to organize small-group interviews, where possible.
The interview guides will be organized by topic area for each type of informant. Within each topic area, the guides will include basic questions and probes designed to stimulate discussion and more complete information for each topic area. The use of detailed interview guides insures some level of uniformity across researchers and informants. Also, the guides as annotated by interview notes provide a structure to data collection that readily organizes field notes for analysis and reporting.
Another useful practice in conducting open-ended interviews is to ask respondents the reasons and/or evidence for their judgmental answers. First, this may force informants to think more carefully about their responses and qualify them in the light of their grounds for holding their opinions. Second, it allows the researcher to weigh the informant's opinion against the strength of the evidence used to support it.
Subsidy Agency Statistical Reports
Extant subsidy agency statistical reports will be used to help characterize the child care subsidy market in the demonstration sites. We expect such reports to provide basic information about: subsidy use, including numbers of families, children, and providers; mean subsidy amounts; types of care used.
Subsidy System Policy Manuals and Eligibility Forms
We will collect demonstration site subsidy system manuals and eligibility forms as our primary source of information about subsidy eligibility criteria, subsidy levels, and co-payment amounts and collection processes. The manuals and eligibility forms will also allow some insight into the initial eligibility and recertification processes, although information about those operations will also be collected in the open-ended interviews at demonstration sites.
Census Information
Census Bureau information will be used as a primary data source for information about site demographic and socio-economic characteristics. Using data from census tracts that most closely overlap with the demonstration sites, the Implementation Study will summarize information about demonstration site ethnicity, household number and composition, number of families with children, distribution of children by age, and other relevant contextual factors.
Bureau of Labor Statistics Labor Market Data
The BLS is an important source of data about local area labor markets, wage rates, industrial mix, employment/population ratios, unemployment data, and other labor market factors. The BLS data are organized by major metropolitan areas and the larger standard statistical metropolitan areas (SMSA). The BLS data will be important in characterizing the low-income labor market facing many subsidy families.
B3 Methods to Maximize Response Rates
a. Massachusetts
The data collection strategies planned for the study involve observations in the family child care home and direct assessment of children. Early in the study, providers will be asked to complete a brief questionnaire about their educational background, experience and motivation. Since the response burden for providers is very little (7-10 minutes), and since home visitors will assist Abt staff in collecting any missing questionnaires, we expect a response rate for the questionnaire of better than 90%. There is, however, a burden imposed by the presence of observers and assessors; if not addressed with sensitivity, this could, over time, affect provider willingness to allow data collection in their home.
Using past experience as a guide, we propose several strategies to address this issue. First, in scheduling visits to the home, we will emphasize that the visit will occur on a morning that is convenient for the provider, and that their schedule and preferences will be decisive in scheduling a visit. The date and length of the visit will be confirmed in a letter, which will also set out expectations for what will happen during the visit. Data collection staff will telephone providers the day before the visit to confirm the schedule since, in any child care setting, unscheduled events can throw off the provider’s schedule. If this occurs, we will reschedule the visit at a time that is convenient for the provider.
Second, at the end of each visit, we will give each provider a $20 gift certificate to compensate her for the disruption in her schedule occasioned by the data collection.
Finally, as part of our validation efforts, we will telephone a sample of providers visited by each data collector to ensure that the visit went as planned, that the data collector explained what she was doing, answered questions, and was respectful and unobtrusive. For all other providers, we will send a thank you card with a toll-free number they can call if they have any concerns about the data collection.
In addition to these strategies, early in the study, each provider will receive a library of 12 children’s books. We will maintain contact with providers through holiday cards and newsletters.
We expect that these efforts will be successful in maintaining providers’ cooperation. However, there are many reasons why we might experience attrition from the study that have to do with providers’ own lives. Providers may leave the study because they have decided not to continue providing care, because of a family or personal emergency or for reasons beyond our control. If their reasons for leaving the study have to do with the demands of the study, we will work with home visitors and system staff to negotiate a solution. We have planned for approximately 15% attrition. If attrition increases beyond this rate, we plan to refresh the sample by adding new providers. We would randomly assign these providers within systems to either Learningames or the control group, following the same procedures as those initially used.
We expect children to leave the child care home in the course of the study and will replace these children with new entrants under three years of age. While we hope to obtain two assessments on each child, the design does not call for a longitudinal study of specific children. We will continue to recruit age-eligible children into the study until six months before the study ends. Our plan is to have essentially continuous data collection and to have providers notify us if a child is leaving the home. This will allow us maximum flexibility in assessing children and reduce non-response because of brief absence or permanent attrition. At the same time, we expect to have no more than three measurements of each child, for the purposes of calculating burden.
b. Illinois
Survey data will be collected at three points in time. All families in the treatment and control group (a total of 2,000) families will be contacted to be interviewed. Our goal is to achieve an 80% response rate at the first survey wave, conducted approximately 8 months after random assignment (1,600 respondents), 75% of the sample at Wave 2 at 18 months (1,500 respondents) and 70% at Wave 3 at 24 months (1,400 respondents). For each wave, we will attempt to reach the entire study sample, excluding those who ask not to be contacted further. For example, for Wave 2, we will not exclusively attempt to contact the 1,600 respondents who participated in the Wave 1 interview but will use the total study group of 2,000, with the exception of those who refused to be contacted further. While we are estimating a response rate for each wave of the study, we estimate that the overall response rate will be close to 80%; that is, 80% of the sample will respond to at least one of the three survey waves.
We believe, based on our past experience, that an 80% response rate is an achievable goal. Exhibit B.3.1 provides information on response rates from similar data collection efforts.
Exhibit B.3.1 Response rates for similar data collection efforts.
Study |
Mode |
Sample Size/Sites |
Response Rate and Follow-up Period |
Moving to Opportunity |
Mail, telephone, in-field surveys tracking household members over six years |
4,608 household heads in 5 cities; 3 waves |
92% (1-2 years) 83% (2-6 years) 81% (2-6 years) |
New York’s State Child Assistance Program |
Longitudinal telephone, with AFDC recipients |
4,300 welfare recipients in 3 counties of New York |
90% (1 year) 87% (2 years) 85% (5 years) |
Summer Career Exploration |
Telephone interviews: two follow-up waves of economically disadvantaged high school students |
1,708 high school students in three cities. |
93% (8 weeks) 89% (8 months) |
In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact information from study participants is accurate and of high quality. The contact information provided by the study participants will include their own address and telephone numbers as well as similar contact information of relatives and friends who are likely to know the participant's whereabouts and do not cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over the course of the study period. Contact information will be entered into a centralized sample database that will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a number of interim tracking methods to ensure that we continue to have up-to-date information. The evaluation team will provide the study participants with a toll-free number to call should they move or get a new phone number. To ensure that the number is on hand we will print it on both a refrigerator magnet and a coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they may send with any updated address or telephone information. Finally, sample members will be mailed "tracking" letters at points prior to their interviews times. Families who return these postcards will receive a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of efforts to locate the proper address and telephone number. All respondents who complete an interview will receive a $20 voucher or gift certificate.
Using data from UI wage records and other public records, we will be able to gather basic information about the non-respondents. If necessary, we will be able to construct weights to address non-response. We do not expect that there will be differential response rates between the treatment and control groups.
c. Washington
Survey data will be collected at three points in time. All 2,500 families who are selected to be in the interview sample (drawn evenly from the treatment and control groups) will be contacted to be interviewed. Our goal is to achieve an 80% response rate at the first survey wave, conducted approximately 8 months after random assignment (2,000 respondents), 75% of the sample at Wave 2 at 18 months (1,875 respondents) and 70% at Wave 3 at 24 months (1,750 respondents). For each wave, we will attempt to reach the entire interview sample, excluding those who ask not to be contacted further. For example, for Wave 2, we will not exclusively attempt to contact the 2,000 respondents who participated in the Wave 1 interview but will use the total interview group of 2,500, with the exception of those who refused to be contacted further. While we are estimating a response rate for each wave of the study, we estimate that the overall response rate will be close to 80%; that is, 80% of the sample will respond to at least one of the three survey waves. Again, based in prior experience, we believe that this response rate is reasonable. (See Exhibit 2.3 for Abt’s relevant data collections.)
In order to increase the likelihood of obtaining this rate, the evaluation team will ensure that the contact information from study participants is accurate and of high quality. The contact information provided by the study participants will include their own address and telephone numbers as well as similar contact information of relatives and friends who are likely to know the participant's whereabouts and do not cohabitate with the respondent. In addition, where it is pertinent, the team will use contact information that it can obtain from public assistance records for those who use TANF, food stamps, or Medicaid over the course of the study period. Contact information will be entered into a centralized sample database that will be used for data tracking and management purposes.
In addition to ensuring that we have high-quality contact information, the evaluation team will use a number of interim tracking methods to ensure that we continue to have up-to-date information. The evaluation team will provide the study participants with a toll-free number to call should they move or get a new phone number. To ensure that the number is on hand we will print it on a refrigerator magnet and a coffee mug. We will also give sample members a pre-addressed, postage-paid postcard that they may send with any updated address or telephone information. Finally, sample members will be mailed "tracking" letters at points prior to their interviews times. Families who return these postcards will receive a $5 voucher or gift certificate. If these letters are undeliverable, the team will engage in a number of efforts to locate the proper address and telephone number. All respondents who complete the first interview will receive a $10 voucher or gift certificate; $15 for completion of the second interview; and $20 for completion of the third interview. The incentive will not affect participants’ potential benefits for public benefits.
Using data from UI wage records and other public records, we will be able to gather basic information about the non-respondents. If necessary, we will be able to construct weights to address non-response. We do not expect that there will be differential response rates between the treatment and control groups.
B4 Tests of Procedures
a. Massachusetts
The observation measures and provider questionnaire have all been tested and used in other large-scale studies with similar populations and so do not require pretesting. The same is true for the standardized child assessments. However, to ensure that our plan for collecting the data is realistic and does not impose undue burden on the provider, we will pretest the data collection procedures in nine family child care homes early in 2006. The results of the pretest will be sent to OMB, with a description of any recommended changes in procedures.
For the children in the study, we want to obtain permission from the maximum number of parents to allow their child to participate in the standardized assessments. We will work closely with the providers to have them help us contact and convince parents of the importance of the study and the low risk of negative consequences for their child. We have a hotline that parents and providers will be able to use to call with questions or concerns at any time during the study.
b. Illinois
We will pre-test the parent telephone interview survey with up to 30 respondents. The results of the pretest will be sent to OMB, with a description of any recommended changes in wording or administration of the survey. (See burden estimate in Exhibit A.12.2a).
c. Washington
The parent telephone interview survey used in the Illinois study will also be used in the Washington study. The results of the pre-test conducted in the Illinois study apply to the Washington study as well.
B5 Individuals Consulted on the Statistical Aspects of the Design
The information for all three studies is being collected by Abt Associates Inc. and its subcontractor, Moore & Associates, on behalf of the Administration for Children and Families (ACF), U.S. Department of Health and Human Services. With ACF oversight, Abt Associates is responsible for study design, data collection, analysis, and report preparation.
a. Massachusetts
The project staff responsible for the design include the project director (Jean Layzer) the deputy project director (Ann Collins), and the director of analysis (Barbara Goodson).
b. Illinois
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the deputy project director, Ann Collins (Abt Associates), and the co-leads for analysis, Nancy Burstein (Abt Associates) and Charles Michalopoulos (MDRC).
c. Washington
The project staff responsible for the design include the project director, Jean Layzer (Abt Associates); the deputy project director, Ann Collins (Abt Associates) and the director of analysis, Charles Michalopoulos (MDRC).
For all three studies, outside consultants reviewed the statistical aspects of the design. These include:
Robinson Hollister
Professor: Econometrics, Labor and Social Economics, Health Economics
Swarthmore College
500 College Avenue
Swarthmore, Pennsylvania 19081
Marcia Meyers
Associate Professor
University of Washington
School of Social Work and
Daniel J. Evans School of Public Affairs
4101 15th Ave NE
Seattle, WA 98105
206-616-4409
Ann D. Witte
Professor of Economics
Wellesley College
Wellesley, MA
References
Anderson, S., Ramsburg, D., & Rothbaum, B. (2003). Illinois study of license-exempt child care: Interim report. Report to the Child Care Bureau, U.S. Department of Health and Human Services. Urbana, IL: School of Social Work, University of Illinois at Urbana-Champaign.
Blau, D. M., “The Economics of Means-Tested Child Care Subsidies,” paper prepared for an
NBER volume on Means-Tested Transfers, February, 2000.
Burstein, N., Layzer, J., and Cahill, K. (2001). National Study of Child Care for Low-Income
Families. Patterns of Child Care Use Among Low-Income Families. Cambridge, MA: Abt
Associates, Inc.
Capizzano, J., Adams, G., and Sonenstein, F. (2000). Child Care Arrangements for Children Under
Five: Variations Across States. Washington DC: Urban Institute.
Casper, L. What Does It Cost to Mind Our Preschoolers? Census Bureau, Current Population
Reports, Household Economic Studies, P70-52.
Chaplin, D., Robins, P.K., Hofferth, S., Wissoker, D., and Fronstin P., “The Price Sensitivity of Child Care Demand: A Sensitivity Analysis,” unpublished manuscript, January 1999.
Collins, A., Layzer, J., Kreader, J., Werner, A., and Glantz, F. (2000). National Study of Child Care for Low Income Families: State and Community Substudy Interim Report. Cambridge, MA: Abt Associates Inc.
Council of Economic Advisors. (December, 1997). The Economics of Child Care. White Paper. Washington, DC: Author.
Hart, B. & Risley, T.R. (1995) Meaningful differences in the everyday experience of young American children. Baltimore, MD: Paul H. Brookes Publishing Co.
Marshall, N. L., Creps, C.L., Burstein, N. R., Glantz, F. B., Robeson, W. W., and Barnett S. (2001). The Cost and Quality of Full Day, Year-round Early Care and Education in Massachusetts: Preschool Classrooms. Wellesley Center for Women and Abt Associates Inc.
Marshall, N. L., Creps, C.L., Burstein, N. R., Cahill, K., Robeson, W. W., Wang, S. Y., Schimmenti, J., and. Glantz, F. B (2003). Family Child Care Today. Wellesley Center for Women and Abt Associates Inc.
Meyers, M., Peck, L., Davis, E., Collins, A., Kreader, J.L., George, A., Schexnayder, D., Schroeder, D., Olson, J. (2002). The Dynamics of Child Care Subsidy Use. A Collaborative Study of Five States. New York, NY: The National Center for Children in Poverty, Mailman School of Public Health, Columbia University.
Shadish, W.R., Cook, T.D., & Campbell, D.T. (2002). Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston: Houghton-Mifflin.
Siegel, G.L., and Loman, L.A. (1991). Child Care and AFDC Recipients in Illinois: Patterns,
Problems, and Needs. St. Louis, MO: Institute of Applied Research.
Appendix A: First Federal Register Notice
[Federal Register: September 21, 2005 (Volume 70, Number 182)]
[Notices]
[Page 55402-55403]
From the Federal Register Online via GPO Access [wais.access.gpo.gov]
[DOCID:fr21se05-89]
-----------------------------------------------------------------------
DEPARTMENT OF HEALTH AND HUMAN SERVICES
Administration for Children and Families
Proposed Information Collection Activity; Comment Request
Proposed Projects:
Title: Evaluation of Child Care Subsidy Strategies.
OMB No.: New Collection.
Description: To conduct four experiments to test aspects of the
child care subsidy system. Two simultaneous experiments will occur in
Cook County, Illinois; one will occur in Washington State; and one will
occur in Massachusetts.
Illinois. The State of Illinois has agreed to conduct two
simultaneous experiments, which will occur in Cook County. The first
will test the impact of receiving a child care subsidy on parental
employment and income, and on the stability of child care arrangements;
the second experiment will test the impact of losing a subsidy on the
same set of outcomes. For the first experiment, families with incomes
above the current income eligibility ceiling who apply for subsidies
will be approved to receive subsidies. In the second experiment,
families in the treatment group with incomes above the eligibility
ceiling who apply to be recertified to continue using subsidies will
remain eligible. In addition, each experiment will test the effects of
a longer certification period by certifying eligibility for some
families for six months and other families for one year. Families in
the two treatment groups will retain eligibility for subsidies over the
two-year study period, provided their income remains below the
experimental limit and they comply with other requirements (e.g.,
continue to work). Outcomes will be measured through administrative
records and periodic interviews with parents.
Washington. In Washington State, the study will test a co-payment
schedule that smoothes out what are currently abrupt increases in co-
payments that occur when a family moves from one income category to the
next and reduces the co-payment burden for many
[[Page 55403]]
families. Families that apply (or reapply) for subsidies and are
determined to be eligible under current rules will be randomly assigned
to the experimental co-payment schedule or the existing schedule.
(Families with co-payments from the experimental schedule will either
pay the same amount, or less, than families whose co-payments are
calculated using the existing schedule.) Families will retain the same
co-payment schedule for two years, provided they continue to be
eligible for subsidies. Outcomes will be measured through analysis of
administrative data and periodic interviews with parents.
Massachusetts. In Massachusetts, the study is an experimental test
of the effectiveness of a developmental curriculum implemented in
family child care homes. Family child care providers who serve
subsidized and other low-income children and are linked to family child
care networks will be randomly assigned to a treatment or control
group. Providers in the treatment group will use the developmental
curriculum and be trained through regular visits to the home by
specially trained mentors. These providers will receive materials to
use with children from 0 to 5 years of age. Providers in the control
group will receive the more general technical assistance and support
visits that they currently receive. Impacts on provider behavior and
the home environment will be measured through direct observations in
the homes. Child assessments will be conducted through provider reports
for the younger children and through standardized tests for children 30
months and older.
Respondents: Illinois. Parents who apply (or reapply) for subsidies
and are eligible and agree to be in the study will be interviewed by
telephone up to three times in the 24 months after they enter the
study.
Washington State. Parents who apply (or reapply) for subsidies and
are eligible and agree to be in the study will be interviewed by
telephone up to three times over the 24 months of the study.
Approximately 30 state employees working at the Department of Health
and Human Services in the Division of Child Care and Early Learning or
the Division of Community Service will be interviewed as part of the
implementation study.
Massachusetts. Children will be assessed 7 months after
implementing the curriculum, after 11 months, and after 23 months.
Providers will be asked to respond to a brief survey 7 and 23 months
after the study begins.
Annual Burden Estimates
----------------------------------------------------------------------------------------------------------------
Number of Average
Instrument Number of responses per burden hours Total burden
respondents respondent per response hours
----------------------------------------------------------------------------------------------------------------
Illinois parent survey......................... 5,000 1.5 .58 4,350
Washington parent survey....................... 2,000 1.5 .58 1,740
Washington process study interview............. 30 .5 .5 8
Massachusetts child assessments................ 700 1.5 .5 525
Massachusetts provider questionnaire........... 350 1 .16 56
----------------------------------------------------------------------------------------------------------------
Estimated Total Annual Burden Hours: 6,679.
In compliance with the requirements of Section 3506(c)(2)(A) of the
Paperwork Reduction Act of 1995, the Administration for Children and
Families is soliciting public comment on the specific aspects of the
information colleciton described above. Copies of the proposed
collection of information can be obtained and comments may be forwarded
by writing to the Administration for Children and Families, Office of
Administration, Office of Information Services, 370 L'Enfant Promenade,
SW., Washington, DC 20447, Attn: ACF Reports Clearance Officer. E-mail
address: grjohnson@acf.hhs.gov. All requests should be identified by
the title of the information collection.
The Department specifically requests comments on: (a) Whether the
proposed collection of information is necessary for the proper
performance of the functions of the agency, including whether the
information shall have practical utility; (b) the accuracy of the
agency's estimate of the burden of the proposed collection of
information; (c) the quality, utility, and clarity of the information
to be collected; and (d) ways to minimize the burden of the collection
of information on respondents, including through the use of automated
collection techniques or other forms of information technology.
Consideration will be given to comments and suggestions submitted
within 60 days of this publication.
Dated: September 15, 2005.
Robert Sargis,
Reports Clearance, Officer.
[FR Doc. 05-18771 Filed 9-20-05; 8:45 am]
BILLING CODE 4184-01-M
Appendix B: Review Procedures for Releasing Public and Restricted Use Data Files
Research Connections, a cooperative effort between the Child Care Bureau (CCB), the National Center for Children in Poverty (NCCP) and the Inter-University Consortium for Political and Social Research (ICPSR), provides access to and promotes the effective use of child care and early education research data that can be used to make informed policy decisions. Data files available through Research Connections comply with the standards of the organization responsible for their preservation and dissemination, ICPSR.
Disclosure Review at ICPSR
ICPSR encourages and facilitates research in the social sciences and related areas by acquiring, developing, archiving, and disseminating data and documentation relevant to a wide spectrum of disciplines. The sole permitted and intended use of archival data is for statistical analysis of trends, groups, or categories of cases, not for investigations of specific individuals or organizations. Nevertheless, whenever data are made available in convenient and readily accessible form, the possibility of intentional or inadvertent disclosure of confidential or erroneous information on individuals or organizations is present.
In recognition of this possibility, ICPSR has developed policies designed to provide an appropriate balance between individual privacy and the essential need of the research community for data. The issue of confidentiality applies to a variety of data including survey data, personal interviews, observation records, notes and recordings, organizational and institutional data, location or geographic coordinate data, and public records. The term "confidential" refers to data that directly identify individuals or organizations or which can be used indirectly or in combination to identify individuals or organizations.
ICPSR processing staff examine the contents of each data collection for the purposes of identifying information that presents problems of preserving confidentiality. This is done in recognition of explicit or implicit pledges of confidentiality given to respondents or research subjects and in accordance with ICPSR policy, University of Michigan policy, and in the case of some of our topical archives, with federal laws and regulations. This process, called a disclosure risk assessment, identifies variables and variable content that must be modified to preserve confidentiality. All such changes in the data are noted in the data documentation. This modified version of the data is made publicly available.
Common methods used to protect respondent confidentiality include dropping cases (eliminating outliers from the sample or providing only a sub-sample of the cases in the total sample), dropping variables that may directly identify respondents or pose a re-identification risk, and coarsening categories by collapsing original values into groupings or capping the tails of a distribution through top and bottom-coding.
In some circumstances the use of these methods alone may not be adequate to fully protect the data in a manner that allows for both its public dissemination and optimal analytic utility. Additional methods can be used to decrease disclosure risk, usually applied in concert with the techniques enumerated above. These complementary procedures can include record swapping, microaggregation, and blank and impute; each systematically distorts original values while maintaining the statistical properties of the data. The application of distortion disables the certainty of respondent identification and provides deniability when claims of identification are made.
As modifying a dataset for reasons of confidentiality can sometimes compromise the research potential of the data. ICPSR may elect to retain the confidential information but to restrict the use of the data set to individual researchers with bona fide research purposes for a specific time period based on agreements established with the data provider. The decision to retain and disseminate a restricted-use version of the data is made only with the consent of the data producer and in consultation with them.
Accessing Public-use Data Files
While current configurations permit both registered access to public-use datasets, as well anonymous registration through a guest authentication procedure, Research Connections is moving toward requiring data users to register with Research Connections as a requirement of public-use data file access.
Prior to downloading data files, all data users, anonymous or registered, are met with the following warning:
WARNING! Some studies we distribute are collected and archived with the support of the U.S. government and thus are covered by government regulations in addition to standard terms of use. Several Federal laws and regulations forbid using these data for any purpose other than statistical analysis and reporting. The applicable laws and regulations may be found in the United States Code, 42 USC Section 3789g(a), the Code of Federal Regulations, 28 CFR 22, and Section 308(d) of the Public Health Service Act (42 U.S.C. 242m). Failure to abide by these laws may result in criminal prosecution. Your use of these data signifies that you have read and agree to abide by these laws and regulations. For more information, please contact netmail@icpsr.umich.edu.
In addition, they must agree to abide by the Responsible Use Statement detailed below:
In preparing data for public release, ICPSR performs a number of procedures to ensure that the identity of research subjects cannot be disclosed. For example, direct identifiers are omitted from datasets, and some characteristics are recoded or masked if they can be combined with others to identify individuals.
Any intentional identification or disclosure of a person or establishment violates the assurances of confidentiality given to the providers of the information. Therefore, users of data obtained from the ICPSR archive and/or any of its special topic archives agree:
To use these datasets solely for statistical analysis and reporting of aggregated information, and not for investigation of specific individuals or organizations, except when identification is authorized in writing by ICPSR
To make no use of the identity of any person or establishment discovered inadvertently, and to advise ICPSR of any such discovery
To produce no links among ICPSR datasets or among ICPSR data and other datasets that could identify individuals or organizations
To comply with the request that downloaded material not be redistributed or sold to other individuals, institutions, or organizations without the written agreement of ICPSR
ICPSR further asks that any books, articles, conference papers, theses, dissertations, reports, or other publications that employ data or other resources provided by ICPSR reference the bibliographic citation provided in the abstract and codebook for each ICPSR data collection. These citations acknowledge the principal investigators, the data producers, and ICPSR as the data distributor, in accord with recommended citation procedures for computer files in the social sciences. Also, authors of publications based on ICPSR data should send copies of their published works or references to the publications to ICPSR for inclusion in a database of related publications.
In addition, the user acknowledges that the original collector of the data, ICPSR, and the relevant funding agency bear no responsibility for use of the data or for interpretations or inferences based upon such uses.
By continuing past this point to the data retrieval process, you signify your agreement to comply with the above-stated requirements and give your assurance that the use of statistical data obtained from ICPSR and/or its Special Topic Archives will conform to widely-accepted standards of practice and legal restrictions that are intended to protect the confidentiality of research subjects.
Accessing Restricted-use Data Files
ICPSR employs several methods of disseminating restricted-use data files, depending on the level of security necessary for the safeguarding of the data.
These methods range from making the data available to a defined subset of the research community, to members of a specific research team, to researchers who have been closely reviewed following the completion of a formal application process, including the submission of a data protection plan, to researchers who may access the data only in a controlled and secure data enclave environment. Each of these approaches requires an application process to gain access to the data. Some require the submission of a data protection plan and include time limits after which the data must be destroyed or returned to ICPSR.
1 In the fourth site, Miami-Dade County, we are using extant data on children and teachers collected by the Early Learning Coalition of Miami-Dade/Monroe Counties as part of its ongoing Assessment and Improvement Initiative to address research questions about the impact of the experimental treatment. Abt is collecting observation data on each study classroom, but there is no data collection burden and thus no request for clearance.
2 The Labor-HHS appropriations bill states that “$9,920,000 shall be for use by the Secretary for child care research, demonstration, and evaluation activities.”
3 “New” is defined as having a break of one month or longer in the use of subsidies.
4 Baseline data comes from extant administrative records so there is no respondent burden.
5 The interviews will not commence until OMB clearance has been obtained. Therefore, families who were recruited into the study in the period of March through May 2005 will have an initial interview that falls between 12-14 months. This group includes 244 study participants.
6 Although random assignment means that in expectation the characteristics of the children and homes in the treatment and control groups will be the same, in our particular sample of children and homes there may in fact be differences between the two groups on observable characteristics. Therefore, we will include demographic and other child and provider characteristics in our models to control for any observable differences between the two groups.
7 The interviews will not commence until OMB clearance has been obtained, which is expected in March. Therefore, families who were recruited into the study in the period of March through May 2005 will have an initial interview that falls between 10-12 months. This group includes 244 study participants
8 Not all family child care providers in the state are associated with networks. This limits the generalizability of the findings to providers who are linked to networks and receive the support and monitoring provided by network staff. However, this subset of providers, who receive child care subsidies, is of particular policy interest to the state. We will use additional extant data from the networks, and from Abt’s Cost-Quality Study of Family Child Care in Massachusetts, to investigate the differences between the study sample of providers and the wider universe of providers in Massachusetts.
9 We are purposefully looking for whether or not Learningames produces positive changes in provider and child outcomes. Therefore, we will conduct a one-tailed test using α1=0.05. We have also assumed that the analyses will include baseline measures that explain 25% of the variation in study outcomes.
Abt
Associates Inc. OMB Clearance Request
| File Type | application/msword |
| File Title | Abt Single-Sided Body Template |
| Author | stewartg |
| Last Modified By | USER |
| File Modified | 2006-07-10 |
| File Created | 2006-07-10 |
© 2024 OMB.report | Privacy Policy