Part B

Evaluation of the Community Healthy Marriage Initiative — Impact Evaluation - Wave 2

OMB No.: 0970-0322

Supporting Statement Part B

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

370 L’Enfant Promenade, SW

Washington, DC 20447

B. Collection of Information Employing Statistical Methods

B.1 Respondent Universe and Sampling Methods

B.1.1 Respondent Universe

The respondent universe for Wave 2 will be the 4,024 respondents to the baseline survey and, in the treatment sites, a supplemental sample of 1,200 persons who participated in a CHMI program. Table 1 shows the sample sizes by site. The purpose of the participant sample is to increase our ability to detect potential impacts of CHMI. Assuming that the impact on marriage and relationship outcomes is more concentrated among participants than non-participants, we plan to oversample participants to ensure sufficient cases for impact analysis. Moreover, by oversampling participants, we will gain other valuable information about participation in marriage- and relationship- education classes that arise out of local programs. Understanding factors related to take up of programming is an important part of understanding community-based initiatives and their potential for success.

The supplemental sample of participants, together with participants occurring naturally in the panel sample, will constitute the overall sample of participants. We will weight all sample members by the inverse of their probability of selection, which will be based on the share of the panel sample that participates in a CHMI-sponsored program or on administrative figures on participants divided by the area population. With appropriate weighting, the calculations of differences between sites will be correctly adjusted to maintain the community representativeness of the panel and participant samples combined. In other words, participants will be weighted downward to reflect their incidence in the community when treatment-comparison site differences are investigated.

Table 1. Round 2 Sample Sizes by Site

Treatment Sites		Comparison Sites		Total
Site	Sample Size	Site	Sample Size
Dallas		Fort Worth
Panel	726	Panel	606	1,332
Participant	400			400
Milwaukee		Cleveland
Panel	766	Panel	577	1,343
Participant	400			400
St. Louis		Kansas City
Panel	746	Panel	603	1,349
Participant	400			400
Total	3,438		1,786	5,224

Because the CHMI and comparison communities were purposively selected, i.e., not selected at random, the estimates from the survey data will apply only to these specific communities. Estimates from the survey data will not be generalized to any other population.

B.1.2 Sampling Methods

As mentioned, all of the respondents to the baseline survey will be included in the Wave 2 data collection. Additionally, from lists provided by the CHMI programs, we will randomly select 400 participants per treatment site. The sampling frame will be stratified by site, grantee, marital status, and gender. We will select 400 participants using systematic sampling within grantee (a site may have more than one grantee), with the sample allocated proportionally to the grantees within treatment site. We expect to complete interviews with 300 of these participants in each site, for a response rate of 75 percent. Table 2 summarizes the approach to the participant sample.

Table 2. Supplemental Sample of Participants Selecting Participants from Grantee Lists

Sampling Frame	List of CHMI participants from grantees in treatment sites
Stratification	Explicit—site, grantee, marital status, and gender
Type of Selection	Systematic sampling
Sample Sizes	400 per treatment site with proportional allocation by grantee; 1,200 total

Table 3 summarizes the sample design and sizes for the two rounds of data collection.

Table 3. CHMI Sample Design in Waves 1/Baseline and 2

	Wave 1/Baseline Sep 07—Feb 08	Wave 2 Oct 09—Apr 10
In Each Treatment Site:	Sample 1 = Persons 18-49 Living in CHMI Core Area Sample size = 2,416 Interviews = 746 Response rate = 79% Combined screening/ interview rate = 63%	Sample 1 = Respondents to Baseline Survey Sample size = 746 Interviews = 597 Response rate = 80% ____________________ Sample 2 = Supplemental Sample of Participants Sample size = 400 Interviews = 300 Response rate = 75%

Continued

Table 3. CHMI Sample Design in Waves 1/Baseline and 2 (Continued)

	Wave 1/Baseline Sep 07—Feb 08	Wave 2 Oct 09—Apr 10
In Each Comparison Site:	Sample 1 = Persons 18-49 Living in Matched Comparison Areas HH Sample size = 1,961 Interviews = 595 Response rate = 79% Combined screening/ interview rate = 63%	Sample 1 = Respondents to Baseline Survey Sample size = 595 Interviews = 476 Response rate = 80%
Total Sample Size	13,134	5,224
Total Interviews All Sites	4,024	4,120

B.2 Information Collection Procedures

There is no change in the information collection procedures. Procedures used for Wave 1/baseline will be followed for Wave 2 data collection. A summary of the procedures is presented here. As was the practice in Wave 1, the Wave 2 data will be collected using computer-assisted in-person interviewing (CAPI). The survey instrument will be computer-assisted to maximize data quality, minimize missing data, ease administration, and reduce time for data preparation and processing. Although in-person interviewing will be the dominate approach, telephone interviews will be conducted with Wave 1 respondents who have relocated to an area that is not proximal to a CHMI interviewer. We estimate this proportion to be less than 5 percent of the sample.

Prior to making contact attempts, each sample member will be mailed a lead letter. For the panel cohort, the letter will remind them of their earlier participation and alert them of the opportunity to participate in the follow-up interview. For the participant cohort, the letter will be a first introduction to the study. The lead letters will explain how the study data will be used, offer assurances of privacy, and include a telephone number that recipients can call for more information (see Attachment C).

About two weeks after the lead letters are mailed, field interviewers will begin making in-person contact attempts. If the respondent is currently available, the interviewer will immediately work with the respondent to identify a private setting within the home where informed consent procedures and the interview can take place. Once in a private setting, the interviewer will read the informed consent statement aloud, obtain verbal consent, and give the respondent a blank copy of the consent form to keep.

The interviewer will administer the interview in a prescribed and uniform manner. After the interview is completed, each respondent will be given a $25 payment. Respondents who report any domestic violence or who become distressed during the interview will also be provided with a discreet card that lists toll-free hotline numbers.

If a respondent refuses to be interviewed, the interviewer will be trained to accept the refusal in a positive manner, thereby avoiding the creation of an adversarial relationship and precluding future contact opportunities. Refusal conversion will include a letter followed by an in-person contact. At least 10 percent of the completed interviews will be verified. Telephone interviewers will contact randomly selected respondents to confirm that the interview was actually completed and that proper procedures were followed (i.e., interview conducted in person, a consent form was provided to the respondent, the incentive was paid, etc.).

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

The methods used to maximize response rates are the same as used in Wave 1. We are committed to successfully contacting and interviewing sample members, and achieving the highest possible response rates. As discussed in prior sections, our methods include:

 Panel Maintenance. For Wave 2, beginning approximately three months before data collection and continuing as long as necessary, all baseline respondents will be submitted to a series of locating vendors for batch address updates. We will send a panel maintenance letter to the address that is determined to be the “most likely address” for each respondent. We will request mail forwarding from the U.S. Postal Service. Any forms returned as undeliverable will be subject to additional centralized tracing efforts by specialists in RTI’s in-house tracing operations unit.

 In-Person Interviewing. Our experience has shown that in-person interviewing yields higher response rates than telephone interviewing with a hard-to-reach population. We will conduct in-person interviews using CAPI because it will help us interview sample members with efficiency.

 Respondent Convenience and Multiple Attempts. We will be flexible in scheduling interviews at the respondent’s convenience. We will also make multiple attempts to reach nonrespondents.

 Advance Contact. We will send customized lead letters in advance of fielding to promote respondent cooperation.

 Telephone Follow-Up for Relocated Respondents. We anticipate that a small proportion of our baseline respondents will have moved out of the interviewing area prior to the Round 2 interview. In order to include these respondents in the survey without incurring large travel expenses, we will attempt to interview these respondents over the telephone.

 Bilingual Approach. The questionnaire and other respondent materials have been translated into Spanish, the most prevalent second language in these communities. The bilingual approach makes it more likely that Spanish-speaking respondents will complete interviews because they will have the choice of using the language with which they feel more comfortable.

 Comprehensive Interviewer Training. We will provide a multi-day, comprehensive training to the interviewing staff. They will be trained on the study purpose and procedures, interview administration, and the protection of human subjects.

 Refusal Aversion and Conversion. Part of the interviewer training will address in detail specific techniques to avert and convert a refusal from a respondent. Respondents who initially refuse to participate will be assigned to interviewers who have a proven record of turning refusals into completed interviews.

 Regular Debriefings with Data Collection Staff. The project management staff will regularly meet with data collection staff to discuss issues related to data collection operations. Methods to enhance response rates will be a standard agenda item at these meetings.

B.3.1 Response Rate Calculations

RTI will calculate response rates in accordance with The American Association for Public Opinion Research (AAPOR) document Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.¹ Specifically, the response rates will be calculated using response rate calculation number five (RR5).² The formula for RR5 is

,

where I denotes completed interviews, P denotes partial interviews, R denotes refusals and break-offs, NC denotes non-contacts, and O denotes other eligible but no interview cases. RR5 is a special case of RR3 that assumes there are no cases of unknown eligibility. In Wave 2, all cases can be assumed eligible because we will be recontacting respondents from Wave 1 and sampling from lists of known participants in CHMI programs.

RTI will calculate RR5 for both unweighted counts and, following the recommendation in the section for Some Complex Designs concerning unequal probabilities of selection and multistage designs, weighted counts using the design weight for the sample overall and specified strata at each stage of sample selection.³ RTI will use the final disposition of case codes consistent with the AAPOR definitions for In-Person Household Surveys.⁴ Finally, RTI will consider an interview complete if a core set of 25 questions is answered.

The Wave 1 survey achieved an overall 63% response rate, about 3 percentage points higher than anticipated. In Wave 2, we anticipate achieving an 80 percent response rate among baseline respondents (who are continuing with the study) and a 75 percent response rate among participants (who are new to the study). We are projecting higher response rates in Wave 2 because we are interviewing persons with whom we have established relationships (baseline respondents) and with whom we have programmatic ties (participant respondents who were told at program intake that they might be contacted for follow-up interviews). In contrast, Wave 1 involved screening the general population for eligible sample members who had no prior history with the study or a CHMI program. It is usual for general population surveys to achieve lower response rates than surveys using samples of prior participants or program participants.

B.3.2 Nonresponse Bias Analysis

Virtually all surveys experience some type of nonresponse. Nonresponse can occur when no information on the sampled unit is collected, called unit nonresponse, or only partial information is collected, i.e., some questions are not answered, called item nonresponse. In either case, estimates of population characteristics from the survey data have the potential for bias. In order to investigate if there is any bias in Wave 2, RTI will examine various aspects of the sample. These investigations will include the following three categories: (1) evaluation of nonresponse rates, (2) comparisons between respondents and nonrespondents, and (3) comparisons based on the level of effort (i.e., number of contact attempts made). A brief summary of these different aspects of the nonresponse bias analysis is provided below. For information on the analyses of nonresponse for Round 1, please refer to the Nonresponse Bias Report for Round 1, included as Attachment D.

Information about Response/Nonresponse Rates. If the response rates overall and by specific analytic domain are high, then the potential for biased estimates from the survey data is reduced. Conversely, if the response rates are low, there is a higher potential for biased estimates from the survey data. Also, if there is differential response across categories of variables correlated with important analytic variables, there will be bias. The calculation of response rates is described in the Response Rate Calculation section and will be used to determine if any of the conditions exist that could potentially introduce bias into the estimates.

Information about Respondents and Nonrespondents. We have frame information available for all sampled cases by disposition, i.e., sampled cases, nonrespondents and respondents. We will investigate the relative sizes of the dispositions by site for the following stratification variables: (1) access to CHMI services, (2) age, and (3) if available, race/ethnicity. The investigation will take two forms: visual assessment and statistical testing.

Information from Respondents Only. In addition to the stratification variables, the level of effort to contact the selected households will also be analyzed in relation to key analytic variables, for example, employment status and relationship status. We will observe if proportions change as the number of contacts increases, suggesting that difficult-to-contact individuals may be different from early responders, and that the potential for nonresponse bias exists among residual nonrespondents. We will use the results of this analysis to inform the weighting process, particularly the formation of strata within which we will make nonresponse adjustments.

B.4 Tests of Procedures

The Wave 2 questionnaire is a slightly modified version of the questionnaire used in Wave 1. Questions that were added or deleted in wave 2 are shown in Part A, Table 1. For purposes of the new supplemental sample of participants for Wave 2, we conducted a pretest of the survey questionnaire with eight respondents recruited from participant lists provided by the CHMI programs. The pretest interviews were administered by telephone to replicate the planned data collection mode for respondents who have moved out of the local area.

The pretest pointed to the need for several revisions to the questionnaire logic that routes participants through the instrument. Corrections were made. The pretest also showed that interviewing respondents by telephone is feasible.

B.5 Statistical Consultants

The following individuals were consulted on the statistical aspects of the survey design:

Dr. Paul Biemer RTI International 919-541-6056

Dr. Karol Krotki RTI International 202-728-2485

Mr. Darryl Creel RTI International 301-770-8229

1 The American Association for Public Opinion Research. 2008. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 5^th ed. Lenexa, KS: AAPOR.

2 AAPOR 35.

3 AAPOR 39-41.

4 AAPOR 18-22 and 46.