Supporting Statement A_1 22 12

Office of Management and Budget

Supporting Statement Part A

for text4baby Evaluation:

Justification

January 27, 2012

This page has been left blank for double-sided copying.

OMB Supporting Statement Part A for text4baby Evaluation

A. Justification

1. Circumstances Making the Collection of Information Necessary

This statement requests Office of Management and Budget (OMB) approval for data collection under the evaluation of the text4baby^SM program. The purpose of the evaluation is to examine the characteristics of women who subscribe to the text4baby mobile phone-based program, assess their experience with the program, and describe their health behaviors and outcomes. Results from the evaluation will help the Department of Health and Human Services (HHS) understand the usefulness of mobile technology for conveying health information to pregnant women and new mothers. In addition, HHS will use information from the evaluation to assess the potential for expanding and/or adapting mobile phone messaging to other health topics or conditions.

The evaluation is being overseen by the Health Resources and Services Administration (HRSA) of HHS. This study is of high interest to HRSA as the primary Federal agency for improving access to health care services for people who are uninsured, isolated, or medically vulnerable. HRSA will use results of the evaluation to improve interventions for pregnant women and new mothers, especially those who are low-income and disadvantaged. This use of results is consistent with HRSA’s mission to improve health and achieve health equity through access to quality services, a skilled health workforce, and innovative programs.

The information collection for this evaluation is new and is authorized by Section 301 of the Public Health Service Act (42 U.S.C. 241); see Attachment A, Authorizing Legislation. Please note this supporting statement (Parts A and B) has been organized by the four instruments for which we are seeking clearance: the Safety Net Consumer Survey, Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews.

Overview of text4baby

With over 1 trillion Short Message Service (SMS) text messages sent in the U.S. last year and text messaging disproportionately high among women of childbearing age and minority populations, text messages represent an enormous and as yet untapped channel for delivering vital health information to those who need it most. Building on the emerging evidence about the potential for using mobile technology as a tool for health communication, text4baby represents an innovative strategy for reaching and engaging underserved populations and promoting healthy behaviors. Women who sign up for text4baby by texting BABY to 511411 (or BEBE for Spanish) receive free SMS text messages each week. These messages focus on a variety of topics essential to maternal and child health: birth defects prevention, immunization, nutrition, seasonal flu, mental health, oral health, and safe sleep, among others. Text4baby messages also provide women with information about early prenatal care and other existing resources available to them. Text4baby messages are brief (160 characters or fewer), evidence-based, tested for health literacy, and timed to reach women when the advice is salient to their pregnancy or postpartum stages. Text4baby messages have been developed as a recognizable “brand” that is designed to associate health messages with a public health program.

Need for Evaluation of text4baby

The goal of text4baby is to promote and reinforce positive health behaviors and use of recommended services. Over the long term, the anticipated increase of healthy behaviors and appropriate health care utilization is intended to lead to a reduction in the estimated 500,000 premature births and 28,000 infant deaths in the United States each year. To understand text4baby’s progress towards reaching its goals and for purposes of program improvement, an evaluation is needed. Much can be learned about the implementation of text4baby from multiple perspectives: implementation partners, health care providers, and consumers. Moreover, the evaluation can shed light on experiences at the national level in designing and implementing the program as well as how communities perceive the text4baby program in a local context. This work has the potential to add significantly to the evidence base that will bolster the development of mobile health (mHealth) and related initiatives in other public health interventions, such as healthy behavior applications focused on weight management, nutrition, smoking cessation, and disease management applications focused on asthma and diabetes.

The evaluation of text4baby is salient for other reasons beyond those stated above and could be used to inform federal decisions regarding mHealth initiatives. As noted in the HHS Text4Health Task Force Recommendations issued in September 2011, research and evaluation is needed to provide further evidence on the effectiveness of health text messaging programs. Moreover, the task force recommended that HHS explore and develop partnerships among federal agencies and non-federal organizations to create, implement, and disseminate health text messaging and mHealth programs. Much can be learned from the text4baby public-private partnership that can inform future decisions to commit resources to similar partnerships, as well as strategies for structuring such partnerships.

Data Collection Activities under the Evaluation of text4baby

To provide the information needed for the evaluation, four types of data collection will be conducted:

• Safety Net Consumer Survey with subscribers and non-subscribers to the text4baby program who receive prenatal care in community health centers in four communities. The health centers will recruit eligible women, obtain their signed consent to participate in the study, and convey their contact information to the study team who will conduct the survey via telephone. Data from the survey will be linked to selected data from electronic health records (EHRs) for respondents who consent to the release of their EHR data. The survey will be conducted in two rounds: Round 1 will include pregnant women and Round 2 will include the same women approximately 9 months later during the postpartum period.

• Consumer Focus Groups with current subscribers in four communities to obtain more in-depth qualitative data regarding the usefulness of the messages and the program. This is a one-time only data collection.

• Key Informant Interviews with a diverse mix of providers in four communities to obtain provider perspectives on the usefulness of the text4baby programs. Providers will include a mix of physicians, midwives, nurses, case managers, outreach workers, and health educators. This is a one-time only data collection.

• Stakeholder Interviews with text4baby partners (public and private) to examine the implementation of text4baby at the national, regional, state, or local level, including outreach, enrollment, coalition building, sustainability, and replication. This is a one-time only data collection.

Together, information collected through these four activities will be used to evaluate the effectiveness of the text4baby program in promoting healthy prenatal and postpartum health practices. The mixed-mode data collection approach will capture both quantitative measures of program effects and qualitative impressions of program implementation and lessons learned. This data collection approach will generate results useful to policymakers and practitioners, informing them about the value of text4baby, specifically, and mHealth applications more generally.

2. Purpose and Use of Information Collection

The approach to the evaluation of text4baby involves several key features designed to produce results that will shed light on the effectiveness of the program and ways it can be improved to make it more effective. First, the approach includes data collection and analysis at both the community and national levels to gain a national perspective on program implementation and a local perspective on program effects. Second, the approach includes responses from multiple stakeholder perspectives—partners, providers, and consumers—to ensure that all stakeholders have a voice in the evaluation. Third, the approach assesses the effects at the individual and system levels, recognizing that the text4baby program not only is likely to affect women and their infants, but also the maternal and child health system designed to serve them. Table 1 highlights the approach across these three dimensions and how the four data collection activities will be used to assess these perspectives.

Table 1. Features of Approach to the text4baby Evaluation

	Stakeholder Perspectives			Level of Effects
Data Collection Component	Partners	Providers	Consumers	Individual	System
Community Level
Safety Net Consumer Survey			√ (S, NS)	√ (S, NS)	√
Consumer Focus Groups			√ (S)	√ (S)	√
Key Informant Interviews		√		√ (S, NS)	√
National Level
Stakeholder Interviews	√				√

S = text4baby subscribers

NS = text4baby nonsubscribers

The community-level components will be conducted in four selected communities served by health center controlled networks (HCCNs) that have a concentration of text4baby users and well-established EHR systems. The method of selecting the four HCCNs and associated health centers is described in Supporting Statement Part B. The purpose of each data collection component is summarized below.

• Safety Net Consumer Survey. To provide more in-depth analysis of subscriber and non-subscriber prenatal and postpartum experiences at the community level, including changes in knowledge, behavior, and outcomes between two points in time; awareness of text4baby; reasons for participating or not participating; and levels of satisfaction among participants. The survey, to be known as the Healthy Pregnancy and Parenting Survey (HPP Survey), will be conducted with consumers receiving prenatal care from health centers associated with four selected Health Center Controlled Networks (HCCNs). To augment self-reported data from survey respondents, consent will be obtained to abstract EHR data to provide additional quantitative data on health care utilization and outcomes. (See Attachment B1 HPP Survey: Round 1, Pregnant Women; Attachment B2, HPP Survey: Round 2, Postpartum Women; and Attachment C for the list of data elements HRSA will request from the 4 HCCNs.)

• Consumer Focus Groups. To gather in-depth information on subscribers’ experiences with text4baby, how it complements other services in the community, and how it affects consumer knowledge and behavior during pregnancy and the baby’s first year of life. Focus groups will be conducted with text4baby subscribers in the same four selected communities as the consumer survey. (See Attachment D, Consumer Focus Group protocols.)

• Key Informant Interviews. To be conducted with health care providers and outreach partners in the four communities selected for the evaluation to obtain observations on how text4baby has affected health care practices among patients and any changes that have occurred among safety net providers at the system level. (See Attachment E, Key Informant Interview protocols).

• Stakeholder Interviews. To provide insights into the implementation of text4baby and identify lessons learned and implications for broader application and sustainability of text4baby specifically and mHealth technologies more generally; to be conducted with text4baby stakeholders at the national level. (See Attachment F, Stakeholder Interview protocols.)

Underlying the evaluation is a conceptual framework (Figure 1). This framework was used to identify short- and long-term outcomes that are hypothesized to be associated with text4baby. The long-term outcomes—such as improvement in maternal and infant health outcomes and reduction of health disparities—are unlikely to be observed during the three-year study period. The framework identifies short-term outcomes that are known to be associated with longer-term outcomes and that can be observed through the evaluation. Evaluation questions and data collection instruments were developed using this conceptual framework.

Figure 1. Conceptual Framework for the text4baby (t4b) Evaluation

3. Use of Improved Information Technology and Burden Reduction

Safety Net Consumer Survey. The survey will comply fully with the Government Paperwork Elimination Act (GPEA), Public Law 105-277, Title XVII by employing technology efficiently in an effort to reduce burden on respondents. HRSA will use a Computer-Assisted Telephone Interview (CATI) methodology to survey all respondents in both rounds (pregnancy and postpartum).

CATI surveys optimize resources and typically guarantee high quality data because the technology incorporates automated range checks and branching, and enforces consistency among critical questions. CATI programming will allow interviewers to collect information that is specific to each respondent, thereby eliminating undue time burden on respondents. The questionnaires solicit only information that corresponds to the specific research items discussed in question A2, above. No superfluous or unnecessary information is being requested of respondents. Finally, interviewers can toggle between English and Spanish versions of the instrument in order to minimize respondent language burden. Use of CATI technology will reduce public burden for the Round 1 survey of pregnant women (960 interviews) and the Round 2 survey of postpartum women (768 interviews).

In addition, by abstracting information on health care utilization and health conditions from electronic health records for those consumer survey participants who give us permission to do so, the burden on individual respondents will be further reduced by relying on existing secondary data. HRSA will work with the four HCCNs and/or their associated health centers to abstract a core set of data elements from their electronic health records. As HRSA will purposively select four HCCNs and associated health centers with mature electronic health records systems, we expect to receive the information from them electronically.

Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews. As these are qualitative data collections, HRSA will not use information technology to collect information from the 30 stakeholders, 80 consumer focus groups participants, and 40 key informants. These collections are qualitative in nature and so few in number it is neither practical nor affordable to build separate electronic instruments to collect the information. All information will be collected orally using paper instruments, supported by digital recordings. Focus group transcripts and key informant interview notes will be analyzed using Atlas.ti, a software system used for the qualitative analysis of large bodies of textual data.

4. Efforts to Identify Duplication and Use of Similar Information

Safety Net Consumer Survey. HRSA sought to avoid duplication of effort in the design of the survey by identifying existing instruments with relevant questions. Since text4baby is a new program, never evaluated previously, many text4baby-specific questions were developed by HRSA or were modified from the few text messaging surveys we identified. Most of the questions related to text4baby awareness, enrollment and disenrollment were new questions. Many questions related to health care access, utilization, knowledge, and behavior were adapted from a survey of Healthy Start participants conducted by HRSA in 2006. The Healthy Start population is very similar to the population targeted by text4baby. Other questions were adapted primarily from the 1989 National Maternal and Infant Health Survey (NMIHS), the Pregnancy Risk Assessment Monitoring System (PRAMS), the Early Childhood Longitudinal Survey 9-month parent interview (ECLS), and the National Survey of Children’s Health (NSCH). The survey instruments presented in Attachment B specify the source of each survey question.

In addition, in designing the survey, HRSA sought to reduce respondent burden by relying on existing electronic health record data for a small set of health information that cannot be reliably self-reported, such as numbers and types of visits and diagnoses. We will obtain consent from study participants before we request this information from HCCNs and/or health centers.

Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews. HRSA developed all qualitative protocols (Consumer Focus Groups, Key Informants, and Stakeholders) specifically for the text4baby program evaluation.

5. Impact on Small Businesses or Other Small Entities

Safety Net Consumer Survey. HRSA’s study design calls for selecting a sample of pregnant women who receive their prenatal care from health centers associated with an HCCN. Four HCCNs and one or two of their health centers will be engaged in the study as a means of identifying a sample frame and recruiting and obtaining consent from the sample of pregnant women for the survey. In addition, the participating HCCNs/health centers will provide a small subset of EHR data for sample members who consent to releasing their data for the study.

HRSA will minimize the burden on the HCCNs and health centers by tailoring the procedures to the organizational structure and work flow of the individual organizations. HRSA will arrange an initial meeting in order to adapt the survey procedures to fit each selected health center’s operational circumstances and requirements. For example, data collection staff will coordinate closely with health centers to develop procedures for obtaining informed consent that meet both their patient security requirements and our survey rigor. In addition, data collection staff will work with each health center to develop procedures for tracking all eligible pregnant women, regardless of whether they consent to participate or not. These tracking procedures will provide the sample frame for the survey. The Round 1 (Pregnant Women) sample frame will consist of “current prenatal patients,” defined as women who have had at least one prenatal visit at the health center during the enrollment period and are between four and seven months pregnant. To minimize burden on the HCCNs and health centers, HRSA will request a minimum dataset of information about the women who are eligible for the survey, including a unique identifier (that can be linked to EHR data if the woman consents), age, due date, and contact information (name, address, telephone number, and email address if available). Contact information will be released to the data collection staff only for women who give consent. Only participants in the Round 1 interview will be eligible to take part in the Round 2 interview approximately nine months later.

Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews. These components of the evaluation have been designed to minimize the amount of burden on respondents. The consumer focus groups will be conducted in person with current text4baby participants. Two focus groups will be offered in each community – generally in the afternoon and evening – to accommodate work and family schedules of participants. All efforts have been made to minimize burden on the participants, with groups estimated to be 90 minutes. Consumers will be compensated $20 for their time.

A small amount of burden will be placed on health centers when a few of their personnel will be invited to participate in key informant interviews. These interviews will be conducted in person. Burden will be minimized by restricting the interviews to 30 to 45 minutes and conducting them at a time and location that is convenient for the respondent.

Stakeholder interviews will be conducted by telephone with text4baby partners in a wide range of public and private organizations, some of which are small nonprofit organizations. To minimize burden and disruption in other business activities, interviews will be restricted to 30 to 45 minutes and will be conducted by telephone at a time that is convenient for the respondent.

6. Consequences of Collecting the Information Less Frequently

Safety Net Consumer Survey. In order to obtain as complete as possible a picture of the effect of text4baby on both pregnant and postpartum women, HRSA deems it necessary to collect information from sample members at two points in time: the first round with women when they are pregnant (960 respondents) and the second round with the same women nine months after the first round (768 respondents). Each survey respondent will respond once as a pregnant woman and once as a new mother with an infant under age one. Most of the questions in the two surveys are not duplicative and those duplicated are needed to assess change over time. Collecting the data less frequently than at these two intervals will not provide HRSA with sufficient information to evaluate the effects of the text4baby program on pregnant and postpartum women.

Consumer Focus Groups, and Key Informant Interviews, and Stakeholder Interviews. HRSA plans one set of focus groups (with up to 80 respondents), one set of key informant interviews (40 respondents), and one set of stakeholder interviews (30 respondents). Each of these informants will respond one time only. There will be no additional qualitative information collections under this OMB request.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

This request fully complies with 5 CFR 1320.5. There are no special circumstances.

8. Comments in Response to the Federal Register Notice/Outside Consultation

A 60-day Federal Register Notice was published in the Federal Register on November 30, 2010 (Vol. 75, No. 229; pp. 74067). There were no public comments.

In an effort to consult with experts both inside and outside HHS, HRSA assembled a Technical Advisory Group (TAG) of Federal staff from each HHS operating division and USDA’s Food and Nutrition Service for representation of the WIC Program. The TAG has been used to consult with HRSA on the text4baby study design and implementation. TAG meetings have been convened on an ongoing basis since March 2010. Attachment G includes a list of TAG members, their affiliations, and areas of expertise.

HRSA pretested Round 1 of the survey with 7 pregnant women and Round 2 of the survey with 8 postpartum women. The results of the pretest and recommendations for finalizing the instruments are presented as Attachment H. The pretest allowed us to validate the length of the instruments (and thus the public burden) and to refine and clarify the survey language. We conducted the pretest with women who received services through a Healthy Start grantee program, a population that is similar to the one in which the surveys will be conducted.

9. Explanation of any Payment/Gift to Respondents

Safety Net Consumer Survey. HRSA recognizes the time burden placed on respondents to the safety net consumer survey. Incentive payments to respondents have been shown to encourage participation and thereby increase response rates, which in turn improve the validity and reliability of the data. Incentive payments can also lower the overall cost of a data collection by reducing the length of the field period and the effort required to reach and interview respondents. “While there is no gold standard on how much incentive to offer a survey respondent, the OMB has approved use of monetary incentives in the range of $20 to $30 with specific target populations similar to those of interest here.”¹ The referenced study population were recipients of Temporary Assistance for Needy Families—a low-income population similar to the population of interest in text4baby. HRSA will provide a post-paid gift card (customized to the site) worth $20.00, upon completion of each round of the survey by CATI. (In other words, respondents to both rounds of the survey will receive $20 at the end of each round, for a total of $40 across the two rounds.) The $20 incentive will be made in the form of a gift card, because those benefits are easier and more convenient to redeem than checks, especially for participants who may not have bank accounts.

Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews. Focus group participants also will be given an incentive of a $20 gift card when they attend the focus group. Key informants and stakeholders will not receive incentive payments because most are participating as part of their professional positions.

10. Assurance of Confidentiality Provided to Respondents

Safety Net Consumer Survey. HRSA has embedded protections for privacy in the study design. The information collection will fully comply with all aspects of the Privacy Act. Individuals and agencies will be assured of the privacy of their replies under Section 934(c) of the Public Health Service Act, 42 USC 299c-3(c). All participants will be told during the consent process and again during the interview that the data they provide will be treated in a confidential manner, unless otherwise compelled by law. They also will be informed that participation is voluntary, that they may refuse to answer any question, and can stop the interview at any time without any risk to their receipt of health care services. Health centers will ask participants to sign an informed consent form to authorize release of contact information for sample members. An additional assurance will be provided at the beginning of the CATI survey. The preferred method for obtaining consent to a CATI survey is to read the specific elements of consent as part of the CATI survey’s procedures for gaining cooperation for participation. The interviewer will read the consent elements and record the sample member’s response in the CATI survey. HRSA will work with the HCCNs/health centers to customize consent procedures so they are acceptable to both HRSA and the HCCNs/health centers.

Consumer Focus Groups, Key Informant Interviews and Stakeholder Interviews. As part of focus group and key informant recruiting, potential participants will be sent information about the study and what is required of participants. The elements of consent will be explained in this document. When participants arrive at the focus group location, they will be given a consent form to read, sign, and return to the moderator. The focus group moderator or key informant interviewer will answer any questions posed by the participants about consent or privacy.

In addition to specific procedures for the various data collection activities, two approaches cut across the entire study. First, all contractor employees will sign a pledge to protect the confidentiality of data and respondent identity, and breaking that pledge is grounds for immediate dismissal and possible legal action. Second, HRSA, through its contractor, is seeking Institutional Review Board (IRB) clearance from Public/Private Ventures (P/PV) in Philadelphia, PA.

11. Justification for Sensitive Questions

The consumer survey is designed to describe the health care experiences, health knowledge, and health behaviors of pregnant and postpartum women and assess how participation in text4baby may be associated with healthy pregnancy and parenting behaviors. A number of items in the questionnaire refer to personal behaviors and circumstances that may be of a sensitive nature for respondents. Examples of potentially sensitive health behavior questions include those related to smoking and alcohol use during pregnancy, breastfeeding, and use of family planning methods. However, information from women on these topics is necessary to collect as research has linked these behaviors to birth outcomes and text4baby provides educational messages to promote such healthy behaviors.

HRSA has minimized the number of sensitive questions to those necessary for the purposes of the evaluation; the survey includes questions on topics that are discussed through text4baby messages or that are directly relevant to assess outcomes and progress towards goals of the program. In addition, interviewer training for the study will stress the importance of asking all questions that involve sensitive issues in a professional and non-judgmental manner. Finally, women will be assured that they do not have to respond to any questions that they do not want to answer.

12. Estimates of Annualized Hour and Cost Burden

Safety Net Consumer Survey. The annualized hour and cost burden for these low-income respondents will be two 20-minute interviews. We estimate the average (median) wage, based on the Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics, All Occupations, to be $16.27 per hour. In addition, for the health center staff that will be involved in recruiting and enrolling eligible pregnant women into the study, the annualized hour and cost burden is estimated at $28.12 per hour.

Consumer Focus Groups, Key Informant Interviews, and Stakeholder Interviews. The annualized hour and cost burden for these respondents, also based on the Department of Labor, Bureau of Labor Statistics, Occupational Employment Statistics, All Occupations, is estimated to be: $16.27 per hour for Focus Group participants, and $78.61 per hour for key informants (HCCN and/or health center providers and outreach partners) and stakeholders (text4baby partners).

Table 2. Estimated Annualized Burden Hours

Instrument	Person Incurring Burden	Number of Respondents	Responses per Respondent	Total Responses	Hours per Response	Total Burden Hours
Consent Training and Coordination	Health Center Study Coordinators	8	12	96	1.00	96
	Health Center Staff	32	1	32	1.00	32
Prenatal Patient Consent	Health Center Staff	1630	1	1630	0.08	130
	Prenatal Patient	1630	1	1630	0.08	130
Parent-of-Minor Consent	Health Center Staff	195	1	195	0.25	49
	Parent of Minor	195	1	195	0.08	16
Safety Net Consumer Survey Round 1	Prenatal Patient	960	1	960	0.33	317
Safety Net Consumer Survey Round 2	Postpartum Patient	768	1	768	0.33	253
Focus Groups	Prenatal/ Postpartum Patient	80	1	80	1.50	120
Key Informant Interviews	Providers	40	1	40	0.75	30
Stakeholder Interviews	Stakeholders	30	1	30	0.75	23
Total						1196

Table 3. Estimated Annualized Cost to Respondents

Instrument	Person Incurring Burden	Total Burden Hours	Hourly Wage Rate	Total Respondent Costs
Consent Training and Coordination	Health Center Study Coordinators	96	$28.12	$2,700
	Health Center Staff	32	$28.12	$900
Prenatal Patient Consent	Health Center Staff	130	$28.12	$3,656
	Prenatal Patient	130	$16.27	$2,115
Parent-of-Minor Consent	Health Center Staff	49	$28.12	$1,378
	Parent of Minor	16	$16.27	$260
Safety Net Consumer Survey Round 1	Prenatal Patient	317	$16.27	$5,158
Safety Net Consumer Survey Round 2	Postpartum Patient	253	$16.27	$4,116
Focus Groups	Prenatal/Postpartum Patient	120	$16.27	$1,952
Key Informant Interviews	Providers	30	$78.61	$2,358
Stakeholder Interviews	Stakeholders	23	$78.61	$1,808
Total		1196		$26,401

13. Estimates of other Total Annual Cost Burden to Respondents or Recordkeepers/Capital Costs

There is no capital and start-up cost to respondents associated with this data collection.

14. Annualized Cost to Federal Government

The evaluation will take place over a 3-year period. The total cost of the evaluation to the government is $1,290,673, which includes the amount awarded via contract to Mathematica ($1,256,480) and HRSA staff time/resources ($34,193). The total evaluation cost was based on the budget developed by Mathematica that calculated wages and hours for all staff, all mailing costs, telephone charges, and overhead costs per contract year along with the Government staff costs. The annualized contract cost has been determined to be $418,827 per year by dividing the total funded amount by three years.

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

Analysis Plan

Although information from the various data collection efforts will be synthesized to answer the evaluation questions, the analyses of data from each collection method will vary. Analyses for the safety net consumer survey, consumer focus groups, key informant interviews, and stakeholder interviews are described below.

Safety Net Consumer Survey. Upon completion of both waves of the survey, a de-identified SAS file will be created for analysis. Weights will be constructed, accounting for the probability of selection, consent rate, and cooperation rate. Overall and HCCN-specific response rates will be developed. All analyses will be performed using weights and accounting for the complex sample design. (Variance estimates will be produced in SUDAAN or STATA to account for design effects.) The analyses will address the evaluation questions specified during the design phase (Table I.1, Attachment I), focusing on variations across sites and population subgroups in the text4baby program’s reach, engagement, education, and connection.

• Reach will be measured according to the characteristics of subscribers and nonsubscribers, including measures of health risk. We will perform both descriptive and multivariate analyses of factors associated with participation in text4baby to identify implications for future outreach efforts. In addition, we will incorporate responses to questions on experiences with the enrollment process (among subscribers) and reasons for nonparticipation (among nonsubscribers) to provide further context on how well the outreach and enrollment processes are working and how they could be improved.

• Engagement will be measured in terms of how subscribers used the text4baby program, including the dimensions shown earlier in the conceptual framework (Figure 1). We will assess which features or content were most useful to participants and identify areas for improvement.

• Education will be measured for both subscribers and nonsubscribers in terms of the proportion that reported receiving health education on specific topics, the sources of that information, and the knowledge or behaviors that correspond to the message content in text4baby. Examples include breastfeeding, interpregnancy intervals, infant sleep position, smoking and alcohol use, usual source of health care, insurance coverage status, and seat belt and car seat use. This analysis will show where gaps remain in knowledge or health behaviors and suggest opportunities for bridging those gaps through additional health messages or complementary services. (Focus groups and provider interviews will provide additional insights.)

• Connection to other services will be measured for both subscribers and nonsubscribers. These indicators may include rates of unmet need, use of dental care during pregnancy, receipt of a flu shot, use of smoking cessation services among smokers, and use of breastfeeding support services, among others.

Complementing the analysis of survey data will be an analysis of EHR data for survey respondents (including subscribers and nonsubscribers). The EHR data will be used to assess compliance with recommended clinical care during pregnancy and the first year of life that can be obtained more reliably from clinical records than self-reports. Examples include timeliness and adequacy of prenatal care and use of primary and specialty care services during and after pregnancy.

Analyses involving comparisons between subscribers and nonsubscribers will involve multivariate methods to control for factors that may affect observed outcomes, such as pre-pregnancy health risk and demographic characteristics. This approach will suggest the association between text4baby participation status and various short-term outcomes, but we will not be able to attribute differences between subscribers and nonsubscribers to the impact of text4baby (that is, to determine what would have happened in the absence of text4baby). Nevertheless, when taken in the context of other evaluation components—especially the consumer focus groups and key informant interviews with providers—the approach will help discern how text4baby fits into the overall system of care for pregnant women and new mothers, including whether text4baby leads to increased consumer engagement that affects the process of care. In addition, the results will have important implications for the ongoing design and enhancement of text4baby in terms of new message content areas that could be developed or existing messages that could be refined, strategies for coordinating care on the local level to improve access and utilization among text4baby participants, and implications for improving outreach to nonparticipants.

Consumer Focus Groups and Key Informant Interviews. Analysis of the focus group and key informant interview data will involve similar steps to ensure a systematic approach to organizing and integrating the evidence across the two qualitative components. Data from the two components will be collected concurrently during two-day site visits by two-person teams to each of the four communities. The analysis of these data will begin with site visit teams developing a brief bulleted list of key themes organized according to a standard template of topics that map to evaluation questions. Debriefing meetings with the site visit teams will be held to discuss the emerging themes, to begin the process of synthesizing results both across sites and between consumers and providers. The focus group transcripts and typed interview notes will be uploaded and coded in Atlas.ti, which allows systematic searching and retrieval of information across all of the visits and according to different types of respondents. A systematic review of data from the focus groups and key informant interviews will be conducted topic by topic to further develop the key themes and to assemble supporting evidence from the Atlas.ti database. Focus group quotes will be identified that help illustrate key themes in the consumers’ own words.

Stakeholder Interviews. Upon completion of the interviews, notes will be typed and uploaded into Atlas.ti using a common coding scheme developed for all qualitative data collection components in the evaluation. The analysis will involve the development of key themes related to implementation successes, challenges, lessons learned, and implications for program improvement and development of future initiatives. The evaluation team will refer to the coded notes in the Atlas.ti database to develop the themes and supporting evidence for the themes. Before writing begins, the team will debrief on the overall takeaway messages from the interviews, to organize the information into a coherent story.

Reports

Results from the evaluation will be summarized at two points in time. The results from the four data collection activities will be presented in two reports. Study briefings will be held with key HRSA staff, TAG members, and other invited guests after submission of each report.

• Interim Synthesis Report. The interim report will summarize the first round of survey results, consumer focus groups, key informant interviews, and stakeholder interviews. Qualitative information assessing program implementation will be combined with descriptive information about text4baby subscribers and nonsubscribers to provide an initial assessment of how the program is performing and how it can be improved.

• Final Synthesis Report. The final report will present quantitative findings from the linked survey and EHR dataset to compare text4baby subscribers and nonsubscribers on health knowledge, behaviors, and outcomes. The two rounds of the survey will provide longitudinal data on use of text4baby, sources of pregnancy and parenting information, and access to and use of health services. The qualitative results on program implementation will provide context for understanding the quantitative results on program effects. The final report will summarize and integrate the themes that emerge across the safety net consumer survey, consumer focus groups, key informant interviews, and stakeholder interviews. It will serve as an overall, accessible summary of key findings and conclusions integrated across the study components.

• Study Briefings. After the completion of each report, a briefing will be conducted to summarize the study’s approach and findings. The audience will be HRSA staff, the text4baby TAG, and other invited guests.

Project Schedule

The project began in October 2010 and is expected to end in September 2013. The schedule for the project is presented in the table below for key data collection, analysis, and reporting tasks relevant to this request for OMB approval.

Table 4. Estimated Time Schedule for Data Collection, Analysis, and Reports

Task	Time Schedule
Develop data collection tools	September 2011
Receive OMB approval	January 2012
Safety Net Consumer Survey
Train health center staff on enrollment process	February 2012
Enroll study sample	February 2012-May 2012
Collect survey data (Round 1)	March 2012-June 2012
Analyze data (Round 1)	July 2012-September 2012
Collect survey data (Round 2)	December 2013-March 2013
Obtain EHR data	March 2013
Analyze linked survey/EHR data (Rounds 1 & 2)	April 2013-June 2013
Consumer Focus Groups and Key Informant Interviews
Conduct site visits	April 2012-May 2012
Analyze data	June 2012-August 2012

Table 4 (continued)

Task	Time Schedule
Stakeholder Interviews
Conduct interviews	February 2012-March 2012
Analyze data	April 2012-May 2012
Reports and Presentations
Interim evaluation report	September 2012
Interim briefing	October 2012
Final synthesis report	August 2013
Final study briefing	September 2013

17. Reason(s) Display of OMB Expiration Date is Inappropriate

There are no exceptions to the certification; the expiration date will be displayed.

1 Jason Markesich and Martha D. Kovac, The Effects of Differential Incentives on Completion Rates: A Telephone Survey Experiment with Low-Income Respondents. Presented at The Annual Conference of the American Association of Public Opinion Research, Nashville, TN, May 16, 2003.