Ssb_streams_fup2_2-13-19

Strengthening Relationship Education and Marriage Services (STREAMS) Evaluation

OMB Information Collection Request

0970-0481

Supporting Statement

Part B

February 2016

Updated June 2016

Updated March 2017

Updated January 2019

Submitted By:

Office of Planning, Research and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Samantha Illangasekare

The Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services issued grants to organizations to provide healthy marriage and relationship education (HMRE) services. As approved under OMB#0970-0481, the Office of Planning, Research, and Evaluation (OPRE) within ACF is conducting the Strengthening Relationship Education and Marriage Services (STREAMS) evaluation with five Healthy Marriage and Relationship Education (HMRE) grantees. The purpose of STREAMS is to measure the effectiveness and quality of HMRE programs designed to strengthen intimate relationships. In particular, the evaluation will examine HMRE programs for youth in high school, adult couples, and adult individuals. The study will fill knowledge gaps about the effectiveness of HMRE programming for youth and adults and strategies for improving program delivery and participant engagement in services.

The STREAMS evaluation includes two components: (1) an impact study to provide rigorous estimates of the effectiveness of program services and interventions to improve program implementation and (2) a process study to support the interpretation of impact findings and document program operations to support future replication. This data collection request is for (1) an extension of previously approved data collection instruments for the impact study and (2) two additional data collection instruments associated with the impact study. Data collection for the STREAMS process study is complete.

B1. Respondent Universe and Sampling Methods

Target population

The STREAMS evaluation examines HMRE programs in five sites selected from among the 46 HMRE grantees that were awarded funding by ACF in September 2015. The sites that were selected were those best suited to examine research questions that will fill gaps in the literature and field and that address issues of particular interest to ACF. Issues of particular interest included the effects of widely used curricula that have not yet been rigorously studied, effective strategies for combining relationship skills and economic stability services, and implementation factors that could influence program effects. Sites also needed to generate adequate sample sizes to detect meaningful impacts and be able to implement a random assignment research design. STREAMS sites were not selected to be representative of all HMRE grantees. Instead, they were chosen to address specific research questions of particular policy interest. The target population and primary research questions of focus for each site are described in Table B.1.

Table B.1. STREAMS impact study site target populations

Grantee	Target population	Primary research questions
More Than Conquerors, Inc.	1,900 students from two Atlanta, Georgia-area high schools	What is the effect of offering relationship skills education as part of the regular school curriculum? How does abbreviating the curriculum influence program effects?
University of Denver	1,000 low-income pregnant women or new mothers from the Denver Health hospital system in Colorado	What is the effect on adult relationship quality and stability of offering relationship skills education and other support services to low-income pregnant women?
The Parenting Center	900 low-income romantically involved couples in Fort Worth, Texas	How does an integrated approach to relationship skills and economic stability services affect the outcomes of participating couples?
Family and Workforce Centers of America	900 low-income job seekers from an employment center in St. Louis, Missouri	What is the effect on adult relationship quality and stability of supplementing job readiness services with the relationship skills education?
University of Florida	1,500 couples participating in relationship education workshops across five Florida counties	Can text messages informed by behavioral insight theory improve attendance at relationship skills education group sessions for couples?

Sampling frame, sample design, and coverage of target population

In each of the five participating sites, the STREAMS impact study is being conducted with individuals eligible to receive the HMRE services provided by the grantee. In the site serving youth in high schools (More Than Conquerors, Inc.), these individuals are 9^th grade students enrolled in regular health classes in schools and classrooms selected by the grantee to receive HMRE services. All students attending the class were eligible for STREAMS. In sites serving adults, study participants are individuals or couples who have voluntarily applied to receive the HMRE services provided by the grantee through a hospital system (University of Denver), co-operative extension service (University of Florida), or community-based setting (The Parenting Center, and Family and Workforce Centers of America). The targeted sample size for each site is shown in Table B.1 above. In each site, the grantee may continue to provide HMRE services to similar individuals after the end of the STREAMS enrollment period. However, these additional individuals served through the grant program will not be part of the STREAMS evaluation sample.

Precision needed for key impact estimates

For the STREAMS impact study, each site will be analyzed separately, so relatively large samples are needed to answer the study research questions. Given the targeted sample sizes shown in Table B.1, minimum detectable effect sizes range from 0.14 for the More the Conquerors, Inc. site to 0.18 for the Family Workforce Centers of America site and The Parenting Center site (Wood et al. 2018). These minimum detectable effect sizes are in line with the findings from prior evaluations of HMRE programming. For example, a prior evaluation of HMRE programming in high schools found an average effect size of 0.20 across seven indicators of students’ attitudes and communication skills. Similarly, recent pre-post analysis of one HMRE curriculum delivered to low-income adults found improvements on communication and conflict management of 25 to 30 percent of a standard deviation six months after the program ended (Antle et al. 2013).

Expected response rates to participant surveys for the impact study

Data collection for the first STREAMS follow-up survey is currently ongoing. To date, the response rate is approximately 85 percent for the site serving youth in high schools and 80 percent for the sites serving adults. For the proposed second follow-up survey, the evaluation team is expecting an 80 percent response rate for both the site serving youth in high schools and the site serving adults. See Section B3 for details on the proposed methods for maximizing response rates to follow-up surveys.

Expected item nonresponse rate for critical questions

Based on experience asking many of the same, or very similar questions with the same population during the baseline and first follow-up survey data collections, the evaluation team does not anticipate significant item nonresponse on baseline or follow-up surveys. The baseline and follow-up surveys are available through computer-assisted web interviewing (CAWI), as well as computer-assisted telephone interviewing (CATI), and CAWI and CATI will also be used for the second follow-up surveys. In addition, the youth survey uses audio computer-assisted self-interviewing (ACASI). Through the ACASI programming, respondents listen to a recording of questions being read aloud, improving comprehension. The use of these three modes improves data quality and completeness because skip logic, appropriate wording variations, and consistency checks are programmed into the instrument. CATI interviewers are trained to use appropriate probes and prompts to encourage responses to all questions.

B2. Procedures for Collection of Information

Survey data collection for the impact study follows different procedures in sites serving adults and the site serving youth in high schools. These procedures are described below.

Sample intake in sites serving adults. In three of the sites serving adults (University of Denver, the Parenting Center and Family and Workforce Centers of America (FWCA), program staff meet with eligible and interested potential participants to enroll them in the study sample. They use an introductory script (Instrument 6) to describe the study to potential enrollees. This script highlights the purpose of the study, the baseline and follow-up measures, and random assignment. If the participant wishes to continue with the sample enrollment process, the intake worker calls Mathematica’s Survey Operations Center to connect the applicant with a trained interviewer who will administer the consent form (Attachment G) and conduct the baseline survey (Instrument 10 and Attachment C). Once the baseline survey is complete, the interviewer instructs the applicant to hand the phone back to the intake worker, who then uses the add-on component of nFORM (Instrument 7) to first confirm whether the applicant is eligible for random assignment and then to conduct the random assignment process. Once the applicant has been randomly assigned, the worker informs the applicant of his or her research status. In the fourth site serving adults (University of Florida), the program recruits and enrolls participants following their existing processes, but after enrollment, participants are randomly assigned to different strategies to promote program participation.

Follow-up survey for adults. Follow-up surveys (Instruments 11 and 13) are administered using CATI and CAWI platforms. Prior to calling sample members to complete the survey, the evaluation team sends sample members an advance letter (Attachment P) to notify them that a Mathematica interviewer will be calling to ask that they complete an interview. In addition, those sample members who provided an email address at the time of the baseline survey receive an invitation email with information required to complete the survey on the web. Mathematica’s locating team sends mail and email reminders, as well as text message reminders to encourage response among hard-to-reach sample members (Attachments N and O). If the respondent cannot be reached by phone and cannot be found by in-house locating staff, the evaluation team assigns a field locator in the program’s community to contact the sample member in person. The field locator then provides the sample member a cell phone to call the Mathematica Survey Operations Center to complete the interview.

Baseline and follow-up surveys for youth in high schools. For the site serving youth in high schools, baseline data collection (Instrument 8) began in July 2016 and was completed in February 2018. Follow-up surveys (Instruments 9 and 12) are administered primarily via ACASI in schools. Since a majority of the sample was enrolled in the study during 9th grade, we anticipate that many participants will be in 11th or 12th grade at the time of the follow-up and still enrolled in the same schools in which they were enrolled at baseline, simplifying locating efforts and improving response rates. We expect approximately 50 percent of the student sample will still attend the same school from baseline. Sample members who are not reached in school will receive an advance letter (Attachment M) and invitation email inviting them to complete the survey through CAWI or CATI. These materials will also notify them that a Mathematica interviewer will be calling to ask that they complete an interview. Mathematica’s locating team will send mail and email reminders, as well as text message reminders to encourage response among hard-to-reach sample members. If they cannot be reached by phone (Attachment L) and cannot be found by in-house locating staff, the evaluation team will assign a field locator in the program’s community to contact the sample member in person. The field locator will then provide the sample member a cell phone to call the Mathematica Survey Operations Center to complete the interview.

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

Data collection for the first STREAMS follow-up survey is currently ongoing. To date, the response rate is approximately 85 percent for the site serving youth in high schools and 79 percent for the sites serving adults. For the proposed second follow-up surveys, the evaluation team is expecting an 80 percent response rate for both the site serving youth in high schools and the site serving adults.

Dealing with Nonresponse

Fielding Protocol to Minimize Differential Non-Response

Throughout data collection, nonresponse will be monitored to ensure representativeness across our sample. We will monitor live reports which breakdown response by critical subgroups (such as treatment and control), to confirm that the level of response is similar across all subgroups. If one subgroup appears to have lower levels of response, we will re-focus resources to decrease differential response rates (such as, sending additional reminder mailings, prioritizing during field locating).

Survey Post-Processing

At the conclusion of data collection, the evaluation team will analyze nonresponse to assess whether the sample of follow-up survey respondents is representative of the full study sample. Using the data on participants’ characteristics collected at baseline, the team will conduct statistical tests (chi-square and t-tests) to gauge whether the sample members who completed the follow-up survey are representative of all sample members, and whether there are systematic differences in the treatment and control group members who responded to the survey.

If necessary, the evaluation team will use two approaches to correct for potential nonresponse bias in the estimation of program impacts. First, the regression models described in A16 will adjust for observed differences between the characteristics of treatment and control group respondents. Second, because this regression procedure will not correct for differences between respondents and nonrespondents in each research group, sample weights will be constructed so that the weighted baseline characteristics of respondents in the treatment and control group in each program are similar to those of the full sample (respondents and nonrespondents). These weights will be constructed using data from the baseline surveys.

Maximizing Response Rates

In sites serving adults, follow-up surveys are conducted using CATI and CAWI platforms, and the proposed second follow-up surveys will be conducted using the same platforms. Prior to calling sample members to complete the survey via CATI, the study team will send sample members an advance letter to notify them of the upcoming data collection (Attachment P). This letter will provide details on their prior participation in the study and reminders about the study purpose and confidentiality. To facilitate locating, detailed contact information is gathered from sample members. In addition, all telephone interviewers are thoroughly trained in both the specifics of the project as well as strategies for securing respondents’ cooperation and averting refusals. Further, we will assign sample members who are reluctant to participate to phone and field interviewers who are especially effective with soft refusal conversion. The team includes bilingual interviewers who can administer the instrument in Spanish when needed. Mathematica’s Survey Operations Center is adequately staffed for both daytime and evening interviews in all U.S. time zones. Mathematica uses automated search tools and online locating services to locate hard-to-reach sample members. Mathematica’s locating team will also send post card reminders, as well as text message reminders to encourage response among hard-to-reach sample members (Attachment N). For the hardest-to-reach sample members, Mathematica deploys field locators in the communities where programs in the study operate to locate sample members and provide them with a cell phone to complete the survey if needed.

In the site serving youth in high schools, follow-up surveys are conducted primarily through group administration in schools. The majority of participants in this site were served during their first year of high school, therefore we anticipate that a large number of these sample members will be enrolled in the same school 2-3 years later at the time of the second follow up survey. Sample members not available for the in school administration will receive an advance letter to notify them of the upcoming data collection (Attachment M). This letter will provide details on their prior participation in the study and reminders about the study purpose and confidentiality. Mathematica will use automated search tools and online locating services to locate sample members no longer in their original schools. Mathematica’s locating team will also send post card reminders, as well as text message reminders to encourage response among hard-to-reach sample members. For the hardest-to-reach sample members, Mathematica will deploy field locators in the communities where programs in the study operate to locate sample members and provide them with a cell phone to complete the survey if needed.

B4. Tests of Procedures or Methods to be Undertaken

Many of the items included in the youth and adult surveys are standardized scales or items adapted from existing surveys. In some cases, the study team developed new items for measuring constructs for which existing measures are not currently available. These items were developed drawing phrasing and language used in survey questions with similar populations.

The evaluation team conducted telephone pretests of the follow-up survey for youth, baseline survey for adults, and first and second follow-up surveys for adults. An additional pretest was not completed for the second follow-up survey for youth due to the fact that approximately 90 percent of the instrument remained the same from the first follow-up survey. The team used these pretests to ensure that questions are understandable, that they use language familiar to respondents, and that they are consistent with the concepts they aim to measure. All pretests were conducted with fewer than 10 people. The team has also used these pretests to identify typical instrumentation problems (such as question wording and incomplete or inappropriate response categories), to measure the response burden, and to confirm that there are no unforeseen difficulties in administering the instruments. The pretests confirmed that the response burden falls within the estimates specified in Part A of this information collection request.

B5. Individual(s) Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Mathematica Policy Research is conducting this study under contract number HHSP233201500095G. Mathematica received input and guidance on the plans for the statistical analyses for this study from ACF staff. The Mathematica evaluation team is led by the following individuals:

Dr. Robert G. Wood

Co-Project Director

Mathematica Policy Research

Ms. Diane Paulsell

Co-Project Director

Mathematica Policy Research

Dr. Brian Goesling

Principal Investigator

Mathematica Policy Research

Mr. Shawn Marsh

Survey Director

Mathematica Policy Research

5