Response3

Date: April 14, 2008

To: Brenda Aguilar

Office of Management and Budget

From: Naomi Goldstein, Director

Office of Planning, Research and Evaluation

Administration for Children and Families

Subject: Response to Additional Follow-up Questions on ACF Healthy Marriage Information Collection Request: 12 Month Follow-Up and Implementation Research Data Collection Instruments for the Supporting Health Marriage Project

Comment/Question 1

We are interested in understanding the specifics of ACF's imputation methodology, especially for missing data for one spouse.  Please describe.

Response

We do not currently have plans to impute missing responses for variables that are used to measure program impacts. 

 

We plan to take one of two approaches to missing outcome data. For impacts that are measured using individual responses to one survey question, individuals who did not respond to that question will typically be dropped from that particular analysis. In these cases, we will conduct a survey non-response analysis that compares baseline characteristics of program and control group members who provide full responses to ensure that they are not systematically different.

 

For impact measures that are constructed using multiple items, or using items from both spouses, we will create decision rules about the number of items that must be non-missing in order to create the variable. For example, there are some impact variables for which the preferred method of creating the variable is to average the two spouses' responses, but for which we could use one spouse's responses rather than drop the whole couple from the analysis.  In those cases, in addition to analyzing survey non-response as described above, our impact analysis will include a control variable for whether the question was constructed using one or both spouses' responses.

 

 

Comment/Question 2

We wanted to follow up on our concerns regarding the privacy of respondents in cases where there is either domestic violence, child abuse or infidelity in the household.  We appreciate all of the steps ACF has already taken to reduce the risk to such respondents.  However, we would like to explore the possibility of going one step further and randomizing some of the most sensitive questions, such that one spouse wouldn't necessarily know whether the other received exactly the same questions they did.  We understand that there are methodological pros and cons, but would like for ACF to consider seriously this approach.  OMB's statistical policy branch can provide examples of other surveys where this approach was used. 

Response

Research indicates that low-income couples experience more marital instability than higher-income couples. Addressing a full range of relationship issues is very important in this first rigorous evaluation of the effects of marriage education for low-income couples. Although some of the questions address sensitive topics, we believe the experience from prior research on sensitive questions and from the current 15-month follow-up for the Building Strong Families (BSF) evaluation suggests that interviewers are able to follow privacy protocols and that these questions can be asked without creating problems for couples.

The sensitive questions about the quality of the couple relationship (e.g., whether there is any physical violence in the relationship, potential or actual cheating) that are included in the SHM 12-month survey are very similar or the same as the set of questions in the BSF 15-month survey that is currently being administered after OMB approval. The BSF survey team is following the same type of procedures as will be used in SHM – interviews are scheduled when respondents say they will have privacy, interviewers are trained to listen for clues to indicate that the respondent is not alone or does not feel comfortable answering some questions, respondents are reminded they do not have to answers questions that make them uncomfortable, etc. The BSF contractor is on track to achieve an 80 percent response rate and there have been no reports of problems related to sensitive questions being asked of both mothers and fathers in the BSF study. Of all the surveys administered to date, there has been only one report of a man who did not want his partner to answer survey questions. However, this was not related to any specific question or set of questions -- the man had not yet been interviewed so he was not aware of the questions being asked.

The SHM programs are required to screen out from the research sample couples who indicate possible domestic violence at the time of intake; such couples are referred to other appropriate services. This approach is expected to substantially minimize the group for whom the sensitive questions may be most problematic. In addition, specifically related to domestic violence concerns, researchers working with couples involved in domestic violence have found that asking spouses/partners survey questions about violence in their relationship was not associated with negative or violent reactions (Heyman et al., 2006).

With the proposed survey administration approach, a spouse will not know what questions are asked of and answered by the other spouse unless the respondent tells them. Randomly leaving some questions out would not fundamentally change the circumstance for individual couples.

Based on the BSF experience to date, we believe the survey can be administered as proposed without creating problems. The survey researchers will closely monitor this issue and report any problems identified. If problems are identified we will take corrective action utilizing the field experience.

Comment/Question 3

We continue to be concerned about the length of the phone survey.  While we could not think of another time that we approved such a long telephone instrument, we recognize that ACF has a strong desire for comprehensiveness.  It would be helpful to understand better ACF's plans for pre-testing this instrument.  Please provide a thorough description of the pretest methodology and discuss what fall-back plans ACF has in place if the pre-test indicates that the questionnaire is too long.

Response

The pretest sample will be selected to be similar in demographic characteristics to the survey sample. . None of the pretest respondents will be part of the actual full-scale study sample. The design of the pretest is intended to reflect the realistic conditions that are expected during the full-scale fielding of the survey instrument. By doing so, the research team aims to mimic those conditions that are likely to pose difficulties during fielding. The pretest survey will be administered to respondents over the phone.

We estimate that the survey instrument will take about 50 minutes to complete, which is comparable to Building Strong Families 15-month survey OMB #0970-0304 (average 53 minutes) and other surveys conducted by MDRC and Abt Associates for past research projects e.g., Hard-to-Employ Rhode Island 15-month survey OMB # 233-01-0012 (average 45-min), and the Employment Retention and Advancement 12-month survey OMB # 0970-0242 (average 45-min).

Should the survey instrument take longer than an average of 50 minutes to administer, we would consider measuring some constructs with fewer items. While using the proposed items increases the reliability of the information gathered, if necessary, there are some questions for which collecting one or two fewer measures is expected to provide adequate reliability for the constructs of interest.

Comment/Question 4

We are interested in better understanding ACF's incentive structure and justification, specifically the $55 payment for the observational component, given that the interviewer will bring an assistant so the respondent will not be expected to obtain a sitter.  Please flesh out more completely the rationale behind this payment.

Response

First we would like to clarify the proposed approach. We are not proposing a payment of $55 for each member of the couple who participates in the observational component. Rather, each member of a couple who completes the 12-month survey instrument and observational component would receive a total payment of $55, whereas each member of the couple who completes only the 12-month survey instrument would receive a $30 payment.

To secure sufficiently high response rates for the telephone survey and the observational component, we propose making financial payment commensurate with the time commitment required of respondents. Research on financial payments to survey respondents has found that such payments are cost effective, lowering the overall cost of most surveys (Singer and Kulka, 2002). These researchers also found that financial payments reduce differential response rates and the potential for non-response bias. Further, there is evidence that financial payments are effective at increasing response rates for people with lower educational levels (Berlin et al. 1992) and low-income and nonwhite populations (James and Bolstein 1990) – population groups expected to be involved in the SHM programs.

Evidence suggests that the payment cannot be much lower than $25 for adults. An experiment from the 1996 panel of the Survey of Income and Program Participation showed that a $20 payment significantly increased response rates, while a $10 payment had no effect relative to those who received no payment. Burghardt and Homrighausen (2002) found response rates for the third follow-up survey of youth in the National Job Corps Study were low with only a $10 payment. When OMB approval was received to increase the payment to $25, the response rate increased and the cost per completed interview was nearly 20 percent lower than those interviews conducted with the $10 payment.

3