Supporting Statement Part B

CONTENTS (continued)

Page

Building Strong Families (BSF) Demonstration and Evaluation: Impact Study Second Follow-up

OMB Supporting Statement Part B

March 2009

B. collection of information employing statistical methods

B1. Respondent Universe and Sampling

The BSF programs target low‑income, adult, unwed couples who are expecting a baby (or have had a baby within the past three months), who are romantically involved and volunteer for the program. The sample frame for the evaluation will be all individuals in the seven chosen sites who are interested in participating in a BSF program during the sample intake period, pass the eligibility criteria, complete a baseline information form, and give consent to participate in the study. The sample intake period will vary in each program. However, it will last until the desired sample size has been randomly assigned in each program or two years, whichever is shorter. Table 9 presents the expected number of couples to be enrolled in the study by site.

Table 9. Expected Number of Couples Enrolled in Study, by Site

	Number of Couples
Site	Program	Control	Total
Atlanta, Georgia	300	300	600
Baltimore, Maryland	288	286	575
Baton Rouge, Louisiana	200	200	400
Florida (Orange and Broward Counties)	200	200	400
Indiana (Allen, Marion, Miami, and Lake Counties)	200	200	400
Oklahoma	250	250	500
Texas (San Angelo and Houston)	250	250	500
Total	1,688	1,687	3,375

B2. Procedures for the Collection of Information

a. Statistical Methodology, Estimation, and Degree of Accuracy

The minimum detectable impacts were estimated for three key outcome variables: the percentage of BSF couples who are married, whether the father is present in the life of his biological child, and the score on the Child Behavior Checklist of Aggressive Behavior. Table 10 shows the minimum detectable impacts. The minimum detectable impacts for the first two outcomes (percent married and father presence) are for the 15-month survey. The minimum detectable impact for the Child Behavior Checklist is for the 36-month assessment.

It is important that the sample is large enough to be able to detect impacts of a size that we would expect given the intervention. Because BSF is a somewhat unique an intervention, it is difficult to predict the size of its expected impacts. However, we do know the following:

• The Minnesota Family Investment Program (MFIP)—a welfare reform program not specifically designed to increase marriage rates—increased the marriage rate 36 months after random assignment among unmarried, long-term welfare recipients by 4 percentage points (Miller et al. 2000). We would expect BSF to have a larger impact on marriage rates than MFIP.

• PREP, a relationship-skills program designed to improve couple relationships, increased the likelihood by 24 percentage points that couples were still married three years after the program (Markman et al. 1988). Unlike BSF, the couples were married before the intervention and typically not low-income. It does suggest, however, that large impacts on marriage are possible.

• Early Head Start—a program designed specifically to improve outcomes for children—decreased scores on the Child Behavior Checklist of Aggressive Behavior by 0.7 scale points (Love et al. 2002). An intervention such as BSF that affects the stability of the child’s family structure over a period of three years could have impacts at least as large as Early Head Start.

When pooled across sites, the expected sample with seven sites—3,375 couples—is sufficient to detect impacts within the expected size range. With the full sample, we will be able to detect an impact of 2.7 percentage points or more in the percent married, 3.8 percentage points in father presence, and an impact of 0.5 points in the measure of aggressive behavior in children. Impacts on scales are frequently presented as effect sizes, that is the impact on the scale as a percentage of the standard deviation of the scale. An impact of 0.5 points in the measure of aggressive behavior is equivalent to an effect size of 8 percent.

The pooled sample will also be sufficient to detect expected impacts for important subgroups. We can detect even quite small impacts with a 50 percent subsample. Subgroups

Table 10. Minimum Impacts Detectable by Sample Size, for Key Outcomes at the 36-Month Follow-up

	15-Month Surveys			36-Month Child Assessment
Sample Size (Program/Control)	Percent Married	Percent of Biological Fathers Present in the Life of their Child		Child Behavior Checklist: Aggressive Behavior
Expected Control Group Mean	12.4%a	71.0%a		11.3b
Pooled Sample: 7 Sites 3,375 (1,688/1,688)	2.7	3.8		0.5 (ES = 8)c
50% Subsample 1,688 (844/844)	3.9	5.3		0.8 (ES = 12)
30% Subsample 1,013 (506/506)	5.0	6.9		1.0 (ES = 15)
Site-Specific Analysis 500 (250/250)	7.1	9.8		1.4 (ES = 21)

Note: Calculations assume: (1) an equal number of program and control members; (2) a 95 percent confidence level with an 80 percent level of power; (3) a one-tail test; (4) a reduction in the variance of 20 percent from the use of regression models; (5) the standard deviation of the Child Behavior Checklist of Aggressive Behavior is 6.5; and (6) response rates of 85 percent for the 15-month mothers and fathers surveys and 78 percent to the child assessments.

^a Based on findings from the Fragile Families 12-month follow-up survey.

^b Based on findings from the Early Head Start evaluation when the child was about 36 months old (Love et al. 2002.)

^c ES = Effect Size

defined by the following characteristics are expected to make up 40 percent or more of the sample: whether the mother is African American, whether the mother is receiving public assistance, and whether the couple is cohabiting. If the impacts are large, we can also detect impacts for subgroups of 30 percent of the sample.

The sample is not large enough to detect impacts for an individual site, unless the impacts are large (such as a 7 percentage point increase in marriage). However, because BSF adheres to a set of detailed, specific guidelines in each site (Hershey et al. 2004), it is meaningful to estimate an impact of the program using pooled data from all seven sites.

Questions about the effects of the implementation of the programs across the sites can be addressed by examining the impacts of BSF on groups of sites. The sample size is large enough to examine impacts for groups of two to four sites. These groups of sites could be chosen for the similarities in how BSF is implemented in those sites or for the similarities in the communities in which the program is implemented. A subgroup of couples in sites operating BSF within Healthy Families programs, for example, would comprise about 50 percent of the sample.

b. Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems requiring specialized sampling procedures.

c. Periodic Cycles to Reduce Burden

This OMB submission requests clearance for implementation analysis data collection protocols and the 15‑month follow‑up survey. A subsequent OMB submission will request clearance for the 36‑month follow‑up survey.

B3. Methods to Maximize the Response Rate and to Deal with Nonresponse

For the 15‑month survey, our target response rate is 85 percent. Interviewing will take place at MPR’s centralized telephone interviewing facility. Field locators will do on‑site searching for sample members who could not be reached or located by telephone. They will also attempt refusal conversion of sample members who refuse to complete the interview over the telephone. Once the sample member is located or agrees to complete the interview, the field locator will contact the telephone center on an available land-line using a toll-free number or on an MPR-provided cellular telephone. The sample member will complete the interview with a telephone interviewer. This method is considered the most cost efficient and will give us the greatest flexibility to follow‑up with sample members who are hard to reach.

The following additional approaches will be used to maximize the response rate.

• Telephone interviewers will be selected based on past experience and performance in comparable studies with demonstrated skills in communication and refusal conversion.

• The telephone interviewers will be supplemented by telephone locators with expertise in locating sample members by telephone.

• Experienced on‑site staff will be hired from a pool of field data collectors with expertise in locating sample members and convincing them to participate in the study.

• Given that some sample members will require interviews in Spanish, we will hire qualified bilingual interviewers to complete interviews in Spanish.

Some nonresponse is inevitable. We will conduct an analysis of nonresponse to assess whether the survey sample is representative of the full sample of mothers and fathers. We will conduct statistical tests (chi-squared and t-tests) to gauge whether the program group members who responded to the interviews are representative of all the program group members, whether the control group members who responded to the interviews are representative of all the control group members, and whether there are differences in the baseline characteristics of the program and control group members who responded to the survey.

We will use two approaches to correct for potential nonresponse bias in the estimation of program impacts. First, the regression models described in section B.16 will adjust for any observed differences between the characteristics of program and control group respondents. Second, because this regression procedure will not correct for differences between respondents and nonrespondents in each research group, we will construct sample weights so that the weighted baseline characteristics of respondents in the program and control group in each site are similar to the full sample (respondents and nonrespondents). These weights will be constructed using data from the baseline information form.

B4. Tests of Procedures and Methods to be Undertaken

A pretest of the survey was conducted for two purposes: (1) to identify typical instrumentation problems such as question wording and incomplete or inappropriate response categories, and (2) to measure the response burden. With respect to the latter, our goal was to develop a questionnaire that could be administered to respondents within 55 minutes.

The instrument was tested in two rounds with couples who had or were receiving BSF services and couples who did not receive these services. The interviews were audio taped and/or monitored to determine what questions the respondents had difficulty understanding, which additional response categories might be appropriate, and what wording changes might improve the clarity of the question intent. As a result of the pretest, we made changes to correct minor skip errors and improve the wording of the questions and their sequencing. In general, respondents did not have difficulty answering the questions. There was some respondent fatigue in Section RR on relationships. As a result, the length of the RR section was cut significantly. The average length of the pretest interviews was 66 minutes. However, when the administration is automated using CATI and the instrument shortened, we expect the interview to last no longer than 55 minutes.

B5. Individuals Consulted on Statistical Aspects of the Design

The following persons were consulted on statistical aspects of the study design:

• Irv Garfinkel, Columbia University School of Social Work, 212 854 8489

• John Gottman, Relationship Research Institute, University of Washington, 206 832 0305

• Barbara Devaney, Mathematica Policy Research, 609 275 2389

• Sheena McConnell, Mathematica Policy Research, 202 484 4518

• Robert Wood, Mathematica Policy Research, 609 936 2776

• Peter Schochet, Mathematica Policy Research, 609 936 2783

• David Myers, Mathematica Policy Research, 202 484 4523

• Elizabeth Stuart, Mathematica Policy Research, 202 484 4517

MPR will collect and analyze the data.

References

Abidin, R.R., and J.F. Brunner. “Development of a Parenting Alliance Inventory.” Journal of Clinical Child Psychology, vol. 24, no. 1, 1995.

Abidin, Richard R. Parenting Stress Index. Third Edition. Odessa, FL: Psychological Assessment Resources, 1995.

Amato, Paul. “Studying Marital Interaction and Commitment with Survey Data.” Paper presented at Healthy Marriage Interventions and Evaluation Symposium of the Measurement Issues in Family Demographic Conference. Washington, DC, November
13-14, 2003.

Berlin, Martha et al. “An Experiment in Monetary Incentives.” In Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association, 1992.

Brooks-Gunn, Jeanne, and Greg J. Duncan. “The Effects of Poverty on Children.” Children and Poverty, vol. 7, no. 2, Summer/Fall 1997.

Burghardt, John, and John Homrighausen. “National Job Corps Study: Survey Results.” Princeton, NJ: Mathematica Policy Research, Inc., 2002.

Carlson, Marcia J., and Frank F. Furstenberg, Jr. “The Prevalence and Correlates of Multipartnered Fertility Among Urban U.S. Parents.” Working Paper #03-14-FF. Princeton, NJ: Princeton University, Center for Research on Child Wellbeing, January 2006.

Carlson, Marcia, Sara McLanahan, and Paula England. “Union Formation in Fragile Families.” Demography, vol. 41, May 2004.

Carlson, Marcia. Unpublished tabulations using data from the Fragile Families and Child Well-Being Study, 2002.

Carroll, Jason, and William J. Doherty. “Evaluating the Effectiveness of Premarital Prevention Programs: A Meta-Analytic Review of Outcome Research.” Family Relations, vol. 52, no.2, 2003.

Cherlin A.J., L.M. Burton, T.R. Hurt, and D.M. Purvin. “The Influence of Physical and Sexual Abuse on Marriage and Cohabitation.” American Sociological Review, vol. 69, no. 6, 2004.

Cowan, Carolyn, and Philip Cowan. When Partners Become Parents: The Big Change for Couples. Mahwah, NJ: Lawrence Erlbaum Associates, 2000.

Cummings, E. Mark, M. Ballard, Mona El-Sheikh, and Margaret Lake. “Resolution and Children’s Responses to Interadult Anger.” Development Psychology, vol. 27, no.3, 1991.

Cummings, E. M., and P. Davies. Children and Marital Conflict: The Impact of Family Dispute and Resolution. New York: Guilford Press, 1994.

Downey G., and J.C. Coyne. “Children of Depressed parents: An Integrative Review.” Psychological Bulletin, vol. 108, 1990.

Edin Kathy, and Maria Kefalas. Promises I Can Keep: Why Poor Women Put Motherhood Before Marriage. Berkeley, CA: University of California Press, 2005.

Ehrle, Jennifer, and Kristin A. Moore. 1999. 1997 NSAF Benchmarking Measures of Child and Family Well-Being: Report No. 6. Washington, D.C.: The Urban Institute.

Gelfand, D.M., and D.M. Teti. “The Effects of Maternal Depression on Children.” Clinical Psychology Reviews, vol. 10, 1990.

Guerney, Bernard G. Relationship Enhancement. San Francisco, CA: Jossey-Bass, Inc., 1977.

Harknett, Kristen, and Jean Tansey Knab. “More Kin, Less Support: Multipartnered Fertility and Perceived Support Among Unmarried Parents.” Unpublished working paper. Philadelphia, PA: University of Pennsylvania, 2005.

Hershey, Alan, Barbara Devaney, M. Robin Dion, and Sheena McConnell. “Building Strong Families: Guidelines for Developing Programs.” Washington, DC: Mathematica Policy Research, Inc., 2004.

House, J.S., Umberson, D., Landis, K.R. “Structures and Processes of Social Support.” Annual Review of Sociology, vol. 14, 1988.

Huston, T.L., and A.F. Chorost. “Behavioral Buffers on the Effect of Negativity on Marital Satisfaction: A Longitudinal Study.” Personal Relationships, vol. 1, 1994.

James, J., and R. Bolstein. “The Effect of Monetary Incentives and Follow-Up Mailings on the Response Rate and Response Quality in Mail Surveys.” Public Opinion Quarterly, vol. 54, 1990.

Karney, B.R., and T.N. Bradbury. “The Longitudinal Course of Marital Quality and Stability: A Review of Theory, Method, and Research.” Psychological Bulletin, vol. 118, 1995.

Larzelere, R.E., and T.L. Huston. “The Dyadic Trust Scale: Toward Understanding Interpersonal Trust in Close Relationships.” Journal of Marriage and the Family, vol. 42, 1980.

Love, J.M., E.E. Kisker, C.M. Ross, J. Brooks-Gunn, P. Schochet, K. Boller, J. Constantine, C. Vogel, A. Fuligni, and C. Brady-Smith. “Making a Difference in the Lives of Infants and Toddlers and their Families: The Impacts of Early Head Start.” Washington, DC: Administration for Children and Families, June 2002.

Lundberg, Shelly, and Elaina Rose. “Child Gender and the Transition to Marriage.” Demography, vol. 52, no. 2, May 2003.

Markman, Howard, F. Floyd, S. Stanley, and R. Storaasli. “Prevention of Marital Distress: A Longitudinal Investigation.” Journal of Consulting and Clinical Psychology, vol. 5, no. 2, 1988.

Markman, Howard J., Frank Floyd, Scott Stanley, and Ragnar Storaasli. “Prevention of Marital Distress: A Longitudinal Investigation.” Journal of Consulting and Clinical Psychology, vol. 56, 1988.

Markman, H.J., M.J. Renick, F.J. Floyd, S.M. Stanley, and M. Clements. “Preventing Marital Distress Through Communication and Conflict Management Training: A 4- and 5-Year Follow-Up. Journal of Consulting and Clinical Psychology, vol. 61, 1993.

Mayer, Susan J. What Money Can’t Buy: Family Income and Children’s Life Chances. Cambridge, MA: Harvard University Press, 1997.

McGroder, S. M. Parenting among low-income, African American single mothers with preschool-age children: Patterns, predictors, and developmental correlates. Child Development, vol. 71, no. 3, 2000.

McLanahan, Sara, I. Garfinkel, N. Reichman, J. Teitler, M. Carlson, and C. Audigier. “The National Report: The Fragile Families and Child Well-being Study Baseline Report.” Princeton, NJ: Bendheim-Thoman Center for Research on Child Wellbeing, August 2001.

McLanahan, Sara, and G. Sandefur. Growing Up with a Single Parent: What Hurts, What Helps. Cambridge, MA: Harvard University Press, 1994.

McLanahan, Sarah, and Gary Sandefur. Growing up with a Single Parent: What Hurts, What Helps. Cambridge, MA: Harvard University Press, 1994.

Miller, Cynthia, V. Knox, L. Gennetian, M. Dodoo, J. Hunter, and C. Redcross. “Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program.” Volume I: “Effects on Adults.” New York: MDRC, 2000.

Office of the Assistant Secretary for Planning and Evaluation. Trends in the Well-Being of America’s Children and Youth 1997. Washington: U.S. Department of Health and Human Services, 1997.

Ouelette, Tammy, Nancy Burstein, David Long, and Erik Beecroft. “Measures of Material Hardship: Final Report.” Washington, DC: U.S. Department of Health and Human Services, April 2004.

Radhakrishna, Aruna, Ingrid Bou-Saada, Wanda Hunter, Diane Catellier, and Jonathan Kotch. “Are Father Surrogates a Risk Factor for Child Maltreatment,” Child Maltreatment, Vol. 6. no. 4, November 2001.

Radloff, L.S. “The CES-D Scale: A Self-Report Depression Scale for Research in the General Population.” Applied Psychological Measurement, vol. 1, 1977.

Rangarajan, Anu, and Philip Gleason. “Young Unwed Fathers of AFDC Children: Do They Provide Support?” Demography, vol. 35, no. 2, May 1998.

Singer, E., and R.A. Kulka. “Paying Respondents for Survey Participation.” Ann Arbor, MI: The University of Michigan, Institute for Social Research, Survey Research Center, 2000.

Stanley, S. M., H. J. Markman, and S. Whitton. “Communication, Conflict, and Commitment: Insights on the Foundations of Relationship Success from a National Survey.” Family Process, vol. 41, no. 4, 2002.

Stanley, Scott. “Assessing Couple and Marital Relationships: Beyond Form and Toward a Deeper Knowledge of Function.” Paper presented at Healthy Marriage Interventions and Evaluation Symposium of the Measurement Issues in Family Demographic Conference. Washington, DC, November 13-14, 2003.

Strauss, Murray A., and Emily M. Douglas. “A Short Form of the Revised Conflict Tactics Scales, and Typologies for Severity and Mutuality.” Violence and Victims, vol. 19, no. 5, October 2004.

Strauss, Murray A., Sherry L. Hamby, Sue Boney-McCoy, David B. Sugarman. “The Revised Conflict Tactics Scale.” Journal of Family Issues, vol. 17. no. 3, May 1996.

Turner, R.J. and J.B. Turner. “Social Integration and Support.” In Handbook of the Sociology of Mental Health. Eds. C.S. Aneshensel, J.C. Phelan. New York: Kluwer Academic/Plenum Publishers, 1999.

Wampler, K. S. “An Update on Research on the Couple Communication Program.” Family Science Review, vol. 3, no.1, 1990.

Wechsler, H., G.W. Dowdall, A. Davenport, and E.B. Rimm. “A Gender-Specific Measure of Binge Drinking Among College Students.” American Journal of Public Health, vol. 85, 1995.

Wechsler, H. “Binge Drinking: The Five/Four Measure.” Journal of Studies on Alcohol,
vol. 59, 1998.

Western, Bruce. “Incarceration, Marriage, and Family Life.” Working Paper #05-09-FF. Princeton, NJ: Princeton University, Center for Research on Child Wellbeing, 2004.

Wilson, Chris M., and Andrew J. Oswald. “How Does Marriage Affect Physical and Psychological Health? A Survey of the Longitudinal Evidence.” Discussion Paper No. 1619. Bonn, Germany: Institute for the Study of Labor (IZA), June 2005.

Wood, Robert G., Anu Rangarajan, and John Deke. “WFNJ Clients Under Welfare Reform: How Is an Early Group Faring Over Time?” Princeton, NJ: Mathematica Policy Research, Inc., September 2003.

2