PACT HM Baseline+Impl+Qual+Hisp Impl - Supp Stat A - 2-11-13

U.S. Department of Health and Human Services Administration for Children and Families Office of Planning, Research, and Evaluation Aerospace 7th Floor West 901 D Street, SW Washington DC 20447 Project Officers: Nancye Campbell and Seth Chamberlain

Parents and Children Together (PACT) Evaluation (0970-0403): OMB Supporting Statement for the Healthy Marriage Baseline, Implementation Study, and Qualitative Study Data Collections Part A: Justification February 2013

CONTENTS

1. Circumstances Making the Information Collection Necessary 1

a. Background 1

b. Overview of the Evaluation 2

c. Current Request 3

2. Purpose and Use of the Information Collection 1

3. Use of Improved Information Technology 4

4. Efforts to Identify Duplication and Use of Similar Information 5

5. Impact on Small Businesses or Other Small Entities 5

6. Consequences of Not Collecting Information or Collecting Information Less Frequently 5

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 6

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency 6

9. Explanation of Any Payment or Gift to Respondents 6

10. Assurance of Confidentiality Provided to Respondents 8

11. Justification for Sensitive Questions 9

12. Estimates of Annualized Burden Hours and Costs 12

13. Estimates of Other Total Cost Burden to Respondents and Record Keepers 16

14. Cost to the Federal Government 16

15. Explanation for Program Changes or Adjustments 16

16. Plans for Tabulation and Publication and Project Time Schedule 16

a. Plans for Tabulation 16

b. Time Schedule and Publications 19

17. Reason(s) Display of OMB Expiration Date Is Inappropriate 20

18. Exceptions to Certification for Paperwork Reduction Act Submissions 20

REFERENCES 21

TABLES

Table A.1. PACT Impact Evaluation (Experimental Impact Complemented with Implementation and Qualitative Strategies) Data Collection Instruments 1

Table A.2. PACT Implementation/Qualitative Only Data Collection Efforts 2

Table A.3. Proposed Appreciation for Participants for Data Collection Involvement 7

Table A.4. Justification for Sensitive Questions in (5) HM Baseline Survey and (7) HM Study MIS 9

Table A.5. Justification for Sensitive Questions in the (14) In-Person, In-Depth Interview Guide and (15) Telephone Check-in Guide 10

Table A.6. PREVIOUSLY APPROVED AND ONGOING – Estimates of Burden and Costs for the PACT Evaluation 12

Table A.7. CURRENT REQUEST – Estimates of Additional Burden and Costs for the Impact Portion of the PACT Evaluation 14

Table A.8. CURRENT REQUEST – Estimates of Burden and Costs for the Hispanic RF Implementation/Qualitative Only Sub-study of the PACT Evaluation 15

Table A.9. Estimate of Burden and Cost for the PACT Evaluation – TOTAL Burden Request 15

Table A.10. Schedule for the Evaluation 19

(Data collection instruments already approved and currently requested are numbered (1) through (18): thus, numbers in parentheses refer to the number of the instrument.)

1. Circumstances Making the Information Collection Necessary

This information collection request (ICR) is for clearance to collect information for the Parents and Children Together (PACT) Evaluation, which is evaluating a subset of Responsible Fatherhood (RF) and Healthy Marriage (HM) grants authorized under the Claims Resolution Act of 2010 (Public Law 111-291).

The evaluation is being undertaken by the U.S. Department of Health and Human Services, Administration for Children and Families (ACF), and is being implemented by Mathematica Policy Research and its partner, ICF International.

a. Background

The past several decades have witnessed sweeping changes in family structure. In 1980, 77 percent of children lived with two married parents; by 2010, this figure had fallen to only 66 percent (U.S. Census Bureau 2011). Families have also become more complex: nearly one in five fathers now has children with more than one woman (Guzzo and Furstenberg 2007). While many children do well living with only one parent, research suggests that on average children do better when they have two involved parents (McLanahan 2009). These changes in family structure, their attendant consequences for children, and recent changes in welfare policy set the stage for new investments in programs aimed at strengthening families and in policy research on fatherhood and marriage.

As one response by the federal government, the Deficit Reduction Act of 2005 created the RF and HM grant programs, authorizing ACF to provide up to $50 million for RF grants and $100 million for HM grants each year from 2006 to 2010. This funding represented an “unprecedented financial commitment by the federal government to support marriage and fatherhood programs” (U.S. Government Accountability Office 2008). Under this act, awards were made to 226 grantees to provide three RF services (responsible fatherhood classes, marriage and relationship education, and economic stability services) or one or more of eight HM services, which include parenting classes, marriage and relationship education, and economic stability services.

The Claims Resolution Act of 2010 reauthorized this grant program, evenly allocating the $150 million between RF and HM funding ($75 million for each). New three-year grants were awarded in September 2011 to 55 RF and 60 HM grantees. The PACT Evaluation will provide documentation of the operations of a subset of these grant programs, the characteristics and life experiences of those who apply for services in the selected programs, and, for some study grantees, an assessment of the impact of the programs on a range of outcomes.

Few rigorous studies of RF programs have been conducted to date. Of the 60 impact and implementation studies of programs for low-income fathers included in a recent systematic review of the evidence (Avellar et al. 2011), only 13 used a rigorous evaluation design. These rigorous program evaluations generally showed at least one statistically significant favorable impact, but most did not result in a compelling pattern of positive impacts.

With regard to HM evaluations, beginning in 2002, ACF sponsored two large-scale, multi-site evaluations utilizing random assignment of enrolled couples to evaluate the effects of programs offering healthy relationship and marriage skills and supportive services to unmarried parents having a child together (Building Strong Families [BSF]) and of similar services for low-income married couples with children (Supporting Healthy Marriage [SHM]). ACF also sponsored an evaluation of community-wide healthy marriage programming (Community Healthy Marriage Initiative [CHMI]) utilizing a quasi-experimental design. Interim results from the BSF evaluation showed no significant differences when data from all eight programs were aggregated; however, in site-specific analyses, a positive pattern of impacts in one site and a negative pattern in another were observed (Wood et al. 2010). After three years, across the eight programs, BSF had no effect on the quality and stability of the couples’ relationship, the couples’ co-parenting relationship, family stability, or economic well-being of children. Impact analyses also identified a modest reduction in children’s behavior problems and small negative effects on some aspects of father involvement (Wood et al. 2012). Interim impact findings from the SHM evaluation showed a consistent pattern of small positive effects after twelve months on aspects of the couples’ relationship, including marital happiness, warmth and support, and positive communication; however, SHM did not affect marital stability after twelve months (Hsueh 2012). Final results from SHM, reporting on impacts 30 months after program enrollment, are forthcoming. Results from the CHMI evaluation indicated that 24 months after implementation of healthy marriage services in target communities, there were no significant differences in participation in healthy marriage services or awareness of healthy marriage messaging (Bir et al. 2012).

b. Overview of the Evaluation

The PACT Evaluation is addressing a number of research questions, including:

• What are the net impacts of the interventions on key outcomes (e.g. relationship status; relationship or marital stability; quality of relationships; attitudes and expectations regarding their relationship or marriage; parenting attitudes and behaviors; measures of child well-being and development (e.g., cognitive, social, emotional, health); adult well-being; and economic outcomes for families)? What are the net impacts for different sub-groups within the research sample?

• What are the issues and challenges in designing, implementing and operating interventions to meet stated program goals and objectives?

• What are the characteristics of lead organizations and partners?

• What are the features and characteristics of the interventions and the context within which they are provided? Are they operated as planned?

• What are the characteristics of individuals/couples targeted by interventions?

As indicated in prior submissions, work under PACT will be carried out in stages with different types of information collection in each stage. Thus, clearance is being requested in stages as work progresses. As previously presented, and as currently detailed in Appendix A, the PACT Evaluation uses three interrelated evaluation strategies: (a) experimental impact; (b) implementation; and (c) qualitative evaluations.

These three strategies are combined into two types of multi-component evaluations:

• impact evaluations, complemented with implementation and qualitative evaluations, will be conducted in a subset of grantees to provide rigorous estimates of the effectiveness of the studied programs and information about their operating contexts; and

• implementation and qualitative evaluations (without impact evaluations) will be conducted in a separate subset of grantees which present some particular feature of program design or target population that warrants detailed study, but which would not support an impact evaluation (for example, if power analyses indicate that sample size is inadequate).

For ease of communication, as in the prior ICR these two types of evaluations are called “impact evaluations” and “implementation/qualitative only evaluations,” respectively.

c. Prior Submissions and Current Request

The first submission, approved on April 20, 2012, provided clearance for discussions with grantees that may be considered as sites in the evaluation. OMB Control number 0970-0403 was set for the evaluation. In keeping with our proposal to provide information to OMB about that activity, a summary is included in Appendix B.

The second submission, approved on October 31, 2012, provided clearance for the baseline survey and management information system to be used with RF programs selected to be in the evaluation. The prior submission indicated that additional follow-on ICR submissions would request clearance for additional instruments and protocols, including those for use in HM programs; to collect program implementation and qualitative data; and to collect follow-up data for the impact analysis. This ICR:

• Includes continued use of four existing approved instruments (currently approved through October 2015 under this control number – 0970-0403);

• Requests clearance for eleven new data collection instruments, listed below, for use in the impact evaluations (which includes implementation/qualitative strategies); and

• Requests clearance for three new instruments for use in implementation/qualitative only evaluation.

The following tables list the data collection instruments approved-to-date, as well as those for which we seek approval now. The names and numbers for these instruments are carried throughout the Supporting Statements and Appendices.

Table A.1. PACT Impact Evaluation (Experimental Impact Complemented with Implementation and Qualitative Strategies) Data Collection Instruments

	Site Selection	Experimental Impact	Implementation		Qualitative
			MIS	Additional Implementation Data Collection Instruments1
Responsible Fatherhood Grantee Evaluation	(1) Selecting study grantees (discussion with program and partner organization staff) – APPROVED APRIL 20, 2012	Baseline: (2) Introductory script (for program staff to discuss with program applicants) (3) Baseline survey (for study participants) BOTH APPROVED OCTOBER 31, 2012 Follow-up: To be submitted in subsequent ICR.	(6) RF study MIS – APPROVED OCTOBER 31, 2012	(8) Semi-structured interview topic guide (for program staff) (9) On-line survey (for program staff) (10) Telephone interview guide (for program staff at referral organizations) (11) On-line Working Alliance Inventory (for program staff and participants) (12) Focus group discussion guide (for program participants) (13) Telephone interview guide (for program dropouts) ALL NEWLY SUBMITTED WITH THIS ICR	(14) In-person, in-depth interview guide (for program participants) (15) Telephone check-in guide (for program participants) BOTH NEWLY SUBMITTED WITH THIS ICR
Healthy Marriage Grantee Evaluation		Baseline: (4) Introductory script (for program staff to discuss with program applicants) (5) Baseline survey (for study participants) BOTH NEWLY SUBMITTED WITH THIS ICR Follow-up: To be submitted in subsequent ICR.	(7) HM study MIS – NEWLY SUBMITTED WITH THIS ICR		No evaluation anticipated

¹ Note that instruments (8) through (13) contain variations which are appropriate to RF and HM programs.

Table A.2. PACT Implementation/Qualitative Only Data Collection Efforts

	Site Selection	Implementation/Qualitative Instruments
Responsible Fatherhood Grantee Evaluation	(1) Selecting study grantees – APPROVED APRIL 20, 2012	“Hispanic RF sub-study” (for RF grantees with a focus on Hispanic populations) (16) Semi-structured interview topic guide (for program staff) (17) Focus group discussion guide (for program participants) (18) Questionnaire (for program participants in focus groups) ALL SUBMITTED WITH THIS ICR
Healthy Marriage Grantee Evaluation		To Be Determined

Subsequent follow-on ICR submissions will be submitted for future stages of work under PACT, including a submission for the collection of follow-up outcome data for the RF and HM impact evaluations.

2. Purpose and Use of the Information Collection

The information to be obtained through the PACT Evaluation is critical to understanding the current field of RF and HM programs—the services they provide, the context in which they are provided, the experiences of their participants, and their effectiveness. This information can be used to inform decisions related to future government investments in this kind of programming as well as the design and operation of such services.

The purpose of each approved information collection instrument was discussed in previous ICRs. The purpose of each newly proposed information collection instrument is described below – numbering matches that in Tables A.1 and A.2.

Impact Evaluation (including implementation/qualitative strategies)

Experimental Impact (Healthy Marriage Grantee Evaluation – Responsible Fatherhood Grantee Evaluation discussed in previous ICR)

(4) Introductory Script (for HM program staff to discuss with program applicants). The script will be used in support of the HM impact evaluation and will be read to all program applicants by HM grantee staff to introduce applicants to the program, the study, and the baseline survey. Members of applicant couples who are eligible to participate, will continue on to complete the Baseline Survey with contractor staff. The introductory script to be used by HM grantee staff is very similar to the introductory script for RF grantees that was approved by OMB on October 31, 2012.

(5) Baseline Survey (for study participants). Similar to the purposes of the RF program baseline survey approved by OMB October 31, 2012, data collected through the HM baseline survey will be used within the HM impact evaluation for six purposes: (a) describing the characteristics of participant couples, including demographics and characteristics of the couple relationship; (b) identifying subgroups of interest (e.g., by race/ethnicity, married or unmarried, relationship quality); (c) including these data as covariates in a regression model to increase the precision of the impact estimates; (d) predicting program participation in an analysis of the impacts for those who participate in the program; (e) checking that the characteristics of program and control group members are on average similar at baseline; and (f) tracking and locating members of the program and control groups to support high levels of follow-up data collection. Appendix D contains a question-by-question justification, lists of key outcome domains and subgroups, links between the domains to be assessed at baseline with those to be assessed at follow-up (provided as the “outcome” column in the question-by-question justification), and names of the surveys in which the items have been used previously.

Implementation – MIS (Healthy Marriage Grantee Evaluation – Responsible Fatherhood Grantee Evaluation discussed in previous ICR)

(7) HM study Management Information System (MIS). Like the RF MIS approved on October 31, 2012, the HM MIS will be used to support the HM impact evaluation and has two main purposes. First, the HM MIS will be used to randomly assign couples who have consented to participate in the study. Second, the HM MIS will be used by program staff to input information on the services provided to program participants and will provide documentation on levels of participation by participants (e.g., attend scheduled workshops or meetings).

Implementation – Additional Implementation Data Collection Instruments (Responsible Fatherhood and Healthy Marriage Grantee Evaluation)¹

(8) Semi-structured interview topic guide (for program staff). The purpose of this information collection is to document manager and staff reports and opinions regarding the implementation and operation of RF and HM programs. This guide will be used during on-site visits to collect information from program managers and staff on topics such as the plans and goals for the program, the staffing structure, recruitment and engagement strategies, services offered, enrollment and retention strategies, attainment of goals, and views of the community context in which they work.

(9) On-line survey (for program staff). The purpose of this survey is to obtain more systematic and potentially more candid information than can be gained through interviews about work activities, work experience, interactions with other staff members, opportunity to receive training and supervision, the supportiveness of the organization, and how the program delivers services and makes needed resources available.

(10) Telephone interview guide (for program staff at referral organizations). The purpose of this guide is to obtain information from community organizations identified as organization with which RF and HM programs make and receive referrals. Topics such as referral organizations’ awareness of the RF/HM program, the accuracy of their knowledge about the program and perceptions of the need for the program, and the process for how referrals are made and received, and interagency communication and coordination will be covered.

(11) On-line Working Alliance Inventory (for program staff and participants). This brief inventory measures the quality of the working relationship between a program staff person (e.g., a case manager) and his/her client (participants) as reported by each party and will complement qualitative information obtained through interviews and focus groups. This information collection will cover topics such as agreement on issues the participant will work on during program participation; the intended outcome of program participation; and the extent to which the relationship is based on mutual trust, acceptance, and confidence.

(12) Focus group discussion guide (for program participants). The focus group guide will be used to explore and document program participants’ perspectives on their motivation for enrolling in the program, and the availability, quality, and value of program services. Of particular interest will be participants’ level of satisfaction with the program and their assessment of the knowledge and skills gained as a result of program participation. In HM programs, couples will be invited to the focus groups; in RF programs, fathers will be invited to focus groups.

(13) Telephone interview guide (for program dropouts). The information collected via these telephone interviews will provide information on reasons why people who enroll in these programs do not complete them including the barriers to their participation.

Qualitative (Responsible Fatherhood Grantee Evaluation)

(14) In-person, in-depth interview guide (for program participants). Utilizing ethnographic techniques, three in-person interviews conducted annually with selected fathers in the RF programs will enhance understanding of program participants’ backgrounds and life circumstances and the range of factors that may affect their ability to benefit from the programs. The first interview will focus most heavily on current life circumstances and key aspects of their experiences of becoming a father, views about fatherhood, family and social network, their program and employment experiences, and challenges they face as parents, providers, and partners. The second and third waves of interviews will focus more intensively on changes that have occurred since the last interview, and information about the neighborhood context in which they live and their views and experiences with other organizations, institutions, and systems in their community (e.g., criminal justice, education, social services, etc.).

(15) Telephone check-in guide (for program participants). The guide will be used for telephone calls to fathers which will be made between annual interview schedules. The calls have two purposes: (a) to increase the likelihood the fathers can be reached for subsequent interviews; and (b), to determine if there have been significant changes in his life and if so, to obtain “real time” perceptions of how such changes affected his involvement with his children or their mothers, in work activities, or in the RF program, rather than rely on recall at the time of the next interview. This information will be used by the interviewer to identify topics in the interview guide for specific attention during the next in-person interview.

Implementation/Qualitative Only Evaluation

An implementation study of selected RF grantees that focus on and serve predominantly Hispanic populations (the “Hispanic RF sub-study”) will explore the extent to which cultural factors influence how RF programs are planned, designed, and operated in order to serve Hispanic men, and the role of culture and acculturation in fathers’ views regarding the program and their participation. The following information collection instruments are proposed to be used.

(16) Semi-structured interview topic guide (for program staff). The instrument will be used to facilitate semi-structured interviews with RF program staff, to learn what factors influenced design decisions and operational practices, specifically those influences related to Hispanic culture. This instrument differs from the (8) semi-structured interview topic guide in that it focuses heavily on program features related to Hispanic culture.

(17) Focus group discussion guide (for program participants). The instrument will guide discussion with focus group participants to obtain participants’ view of the RF program and the factors that led them to enroll and participate, specifically the role of Hispanic culture.

(18) Questionnaire (for program participants in focus groups). The questionnaire will provide quantitative data to complement the qualitative data in documenting the role of culture, cultural perspectives and acculturation among fathers participating in the RF programs.

3. Use of Improved Information Technology

The following new information collections will be supported by web-based technology that will facilitate administration and minimize burden as described further below:

(5) Baseline survey (for HM study participants);

(7) HM study MIS;

(9) On-line survey (for RF and HM program staff); and

11. On-line Working Alliance Inventory (for RF and HM program staff and participants).

The (5) baseline survey of HM program applicants will be conducted by computer assisted telephone interviewing (CATI). CATI reduces respondent burden by automating skip logic and question adaptations and by eliminating delays caused when interviewers must determine the next question to ask. CATI is programmed to accept only valid responses based on preprogrammed checks for logical consistency across answers within the survey. Interviewers are thus able to correct errors during the interview, eliminating the need for burdensome and costly call-backs to respondents.

The (7) HM study MIS will be a web-based application providing easy access while maintaining the security of the data and allowing grantees to access the MIS without purchasing or installing additional software or changing the configuration of their computers. The system is designed for multiple users at each organization and will include options for varying levels of system access depending on users’ needs. The system can be accessed from any computer, allowing for ease of entry, while the data are housed on secure servers behind the contractor’s firewall, thereby maintaining data security.

The (9) on-line survey will be administered through a web-based application to enable staff to complete the survey at a location and time of their choice. The web-based survey will reduce burden by automating skip logic.

The 12-item (11) on-line Working Alliance Inventory will be administered through a web-based application that will enable program staff and participants to complete the survey at a time and location of their choice. Staff and program participants will have access to the internet at the program grantee’s location. Participants can complete the inventory during a visit to the program office or at another time via a private computer.

In addition, audio recording, with permission, will be used to facilitate interviewer-participant dialogue and interaction without distraction of extensive note taking and to increase accuracy of documentation of all points raised during the group discussions or one-on-one interviews. Thus, the (12) focus group discussion guide (for program participants), (14) in-person, in-depth interview guide (for RF program participants), and the (17) focus group discussion guide (for RF program participants in the “Hispanic RF sub-study”) will be supported by audio recorded technology.

Technology is not proposed to be used with the other new information collection instruments in this submission.

4. Efforts to Identify Duplication and Use of Similar Information

There are no other evaluations of ACF-funded RF and HM grantees ongoing, thus the PACT Evaluation is not duplicative of other efforts. Likewise, there are no other sources of information that would allow us to answer the specific questions regarding the implementation and effectiveness of ACF-funded RF and HM programs. Within the evaluation, we do propose to use measures (e.g., in the baseline survey for HM program participants) that have successfully been used in prior studies involving similar populations and programs.

At each stage of the evaluation, we will ensure that we do not collect information that is available elsewhere. None of the instruments will ask for information that can be reliably obtained through other sources such as administrative data collection. For example, the baseline survey (neither for RF nor HM program participants) does not ask questions about child support payments because information on these payments is expected to be available from administrative data sources. If a grantee has an existing MIS that tracks information needed for the PACT Evaluation, we will accept data from their existing MIS rather than having them use the RF or HM study MIS.

5. Impact on Small Businesses or Other Small Entities

No small businesses that are not RF/HM grantees or their partners are expected to be involved in data collection. In the case that RF/HM grantees or their partners are small entities, instruments have been tailored to minimize burden and only collect critical evaluation information.

6. Consequences of Not Collecting Information or Collecting Information Less Frequently

The purpose of each information collection instrument included in this submission is described in Item A2, above. Not collecting information using these instruments would limit the government’s ability to document and report on the kinds of activities implemented with federal funds, the characteristics and views and life circumstances of fathers and couples who seek these services, and the effectiveness of the programs. The following newly proposed instruments will collect data more than once. Rationale is provided.

The (7) HM study MIS requires multiple entries by HM program staff over the course of the period of program operations. In addition to data entry required to enroll all couples in the study and conduct random assignment, HM grantee staff will be asked to enter information on services offered or scheduled and service receipt for all program group members over the course of their participation in the program. These multiple entries are necessary to provide documentation on program implementation that is critical to the implementation analysis and to interpreting the findings from the impact analysis.

The (8) semi-structured interview topic guide (for program staff) and (9) on-line survey (for program staff) will be collected twice, during site visits that will be conducted between twelve and eighteen months. These multiple site visits are intended to identify changes due to program maturation.

The (11) on-line Working Alliance Inventory will be completed once by program participants, but may be completed up to 20 times per program staff, since staff will be describing their working alliance with different individual participants. Although staff may complete the inventory multiple times, each completion will be an assessment of a different, separate working alliance.

We are proposing to use the (14) in-person in-depth interview guide (for RF program participants) across three waves of qualitative interviews with a sub-set of fathers. Conducting three waves of interviews will support documentation of the dynamic aspects of the participants’ lives. Limited information beyond demographic characteristics exists about low-income fathers who seek support and services to be engaged and responsible fathers. The more comprehensive information that can be gained over time is intended to provide funders and program administrators with information that may be considered in future program designs or refinements. Three, rather than two, waves are proposed because prior research (Weiss 1994) suggests that a deepening sense of rapport between interviewer and respondent over time may increase respondent comfort with the study so that they are especially willing to share important details of their circumstances in later waves.

The associated (15) telephone check-in guide (for RF program participants) is proposed to be used four times—twice between the first and second waves of in-person interviewing (at the fourth and eighth month following the first interview) and twice between the second and third waves. This frequency has been shown to be useful in retaining respondent engagement in other studies using similar longitudinal interviews (Edin and Young, personal communication, 2012) and as an effective means of learning about events that respondents consider significant or important changes or life events. Less contact could make it more difficult to retain fathers across waves of the qualitative study (i.e., attain a high response rate) and make “starting on the same page” at the next scheduled interview with the respondents more difficult if the opportunity to identify significant life events in between interviews is not provided.

7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances for the proposed data collection.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through the 60-day Federal Register Notice, published on November 21, 2012 (77FR 225, document number 2012-28321, pp. 69861-69863). A copy of this notice is attached as Appendix E. The notice provided 60 days for public comment. No substantial changes in burden are proposed over those proposed in the 60-Day FRN. One comment was received objecting to the entire undertaking; it did not include any specific comments on the proposed information collection.

9. Explanation of Any Gift to Respondents

We propose to provide appreciation to participants for the data collection activities covered by this ICR, as summarized in Table A.3.

Table A.3. Proposed Appreciation for Participants for Data Collection Involvement

Data Collection Activity	Length of Activity (minutes)	Proposed Appreciation (per Participant)
Impact Study
(5) Baseline survey (for HM study participants)	30	$102
(13) Telephone interview guide (for RF and HM program dropouts)	15	$15
(14) In-depth, in-person interview guide (for RF program participants)	120	$60 per wave3
Implementation/Qualitative Only Study
(17) Focus group discussion guide (for RF program participants in Hispanic RF sub-study)	90	$25

We make this proposal for three reasons:

a. To increase response rates. When participants know that their time will be appreciated, the likelihood increases that those respondents will spend time completing the data activity. Research has shown that such tokens of appreciation are effective at increasing response rates for populations similar to participants in RF/HM programs—people with lower educational level (Berlin et al. 1992) and low-income and nonwhite populations (James and Bolstein 1990). Singer and Kulka (2002) similarly showed reduced differential response rates and hence the potential for nonresponse bias.

b. To reduce attrition for follow-up data collection. In longitudinal studies, providing a small symbol of appreciation for earlier surveys may contribute to higher response rates for subsequent surveys/interviews (Singer et al. 1998). Therefore, providing such appreciation for the baseline survey and the first waves of the qualitative interviews may reduce attrition for follow-up data collection.

c. To garner grantee staff cooperation. In general, grantee staff find evaluation very challenging—this is true in implementation and qualitative studies, and especially true in impact studies where participants are randomly assigned. Directors of RF programs⁴ have told us that providing a modest token to all applicants for response to the baseline survey—including those who are ultimately assigned to the control group—as well as a modest token for participation in other data collection efforts demonstrates respect for the applicants’ time and makes it easier for the grantee staff to support the evaluation and the requirement to assign some applicants to control group status.

10. Assurance of Privacy Provided to Respondents

Respondents will be informed that the identifying information they provide will be kept private as provided by the Confidentiality Certificate issued by the Eunice Kennedy Shriver National Institute of Child Health and Human Development on January 3, 2013 (a copy is provided in Appendix F). All consent forms that are given to study participants will include assurances that the research team will protect their privacy to the fullest extent possible under the law (the three consent forms are contained in Appendices G, H, and I).

Specifically with regard to administration of the baseline survey, similar to the procedures used with the (3) RF baseline survey described in the prior submission, before the (5) HM baseline survey is administered, the interviewer will read the consent statement, which includes the assurance of privacy. The respondent will also be given a hard-copy of the consent statement for their reference (Appendix G contains the consent statement for RF program participants; Appendix H contains the consent statement for HM program participants). This consent statement will cover all program participant-focused data collection activities in the impact study—the (3) RF baseline survey or (5)HM baseline survey, the (11) on-line Working Alliance Inventory, (12) participant focus groups, (13) telephone interviews with program dropouts, and the (14) in-person in-depth qualitative interviews and (15) qualitative interview check-in calls (with RF program participants).

In addition to information provided at the time of the baseline data collection, before other types of subsequent information collection (e.g., web-based or telephone surveys, focus groups, interviews, or questionnaires) the data collectors will state that the information provided by the respondent will be kept private and that the results of the study will be presented in aggregate form only. Likewise, data collectors will obtain consent and provide the same assurance of privacy to fathers in the Hispanic RF sub-study implementation/qualitative only study (see Appendix I).

The contractor will take the following specific measures to protect respondents’ privacy:

• Training interviewers in privacy procedures. The oral consent process and (3) RF/(5) HM baseline survey will be administered by telephone interviewers at the evaluator’s Survey Operations Center (SOC). Interviewers will be seated in a common supervised area. As part of the telephone interviewers’ introductory comments, study participants will be told that their responses will be protected and that they will have the opportunity to have their questions concerning the study answered by the interviewer. Interviewing staff will receive training that includes general SOC security and privacy procedures as well as project-specific training that includes explanation of the highly private nature of this information, instructions to not share it or any personally identifiable information (PII) with anyone not on the project team, and warnings about the consequences of any violations. After receiving training, these staff members sign privacy and nondisclosure agreements.

• Using CATI and web-based surveys. Administering consent and the (3) RF/(5) HM baseline survey via CATI and the (9) on-line survey (for program staff) and (11) Working Alliance Inventory (for program staff and participants) via web eliminates security risks related to shipping hard-copy forms containing PII to the evaluator.

• Restricting and logging access to the sample management system (SMS). Some data elements from the baseline survey data will be entered into an SMS to locate sample members for the follow-up survey. This is a sequel server database housed on an encrypted server. A hierarchical architecture will be used to assign user rights to specific individuals who will be able to access the system and enter information only at their own location. All activity in the system will be logged. Unless otherwise required by ACF, the information stored in the SMS will be destroyed when no longer needed in the performance of the project.

• Restricting access to the study MIS. Data collected through the study MIS will be housed on secure servers behind the evaluator’s firewall. Access to the study MIS will be restricted by assigning a password to each relevant staff member.

In addition to these study-specific procedures, the evaluator has extensive corporate administrative and security systems to prevent the unauthorized release of personal records, including state-of-the-art hardware and software for encryption that meets federal standards and other methods of data protection (e.g., requirements for regular password updating), as well as physical security that includes limited key card access and locked data storage areas.

11. Justification for Sensitive Questions

Impact study

Some sensitive questions are necessary in a study of programs designed to affect personal relationships and employment.

The procedures to be used in administering the (5) HM baseline survey are the same as those described for the (3) RF baseline survey in the prior submission (approved Oct 31, 2012). Prior to starting the HM baseline survey, all respondents will be informed that their identities will be kept private and that they do not have to answer questions that make them uncomfortable. Table A.4 describes the justification for the sensitive questions included in the baseline survey. Although these questions are sensitive, they have commonly, and successfully, been asked of respondents similar to those who will be in this study (for example, in the Fragile Families and Child Wellbeing Study, the Building Strong Families Study, and the Early Head Start Research Evaluation Project).

Table A.4. Justification for Sensitive Questions in (5) HM Baseline Survey

Question Topic	Justification
Respondent Social Security Number (SSN)	The respondent’s SSN is essential for this evaluation for three reasons. First, it will be used to collect administrative data on the respondents. The SSN will allow us to obtain important outcome data on the respondent from child support agencies and the National Directory of New Hires. Second, the SSN will also be used to collect information on the location of the study participant for the follow-up data collection. Third, these numbers will be used as an identifier to link the information collected in the study MIS with the survey data and will allow the study MIS to check whether the person has already been randomly assigned.
Earnings	A key goal of HM programs is to improve couples’ economic stability. The outcomes of an individual employed when he/she enters the program may be very different than those of an individual who enters without employment. The survey asks whether the respondent worked in the past month and, if so, the amount he or she earned in the last month from formal and informal jobs. This question has been asked successfully in many surveys including the Building Strong Families survey (Wood et al. 2010). In this survey, only 0.4 percent of mothers and 0.1 percent of fathers did not respond to the earnings questions.
Involvement with the criminal justice system	Recent research suggests that a history of incarceration and involvement with the criminal justice system may be fairly common among individuals, particularly men, in the PACT target population (Pearson et al. 2011). Incarceration has major negative effects on child and family well-being, including reducing the financial support and other types of support adults can provide to their partners, children, and families, thus documenting the incidence is important. Similar questions have been included in other large national studies, such as the Fragile Families and Child Wellbeing Study, the National Job Corps Study, and the Building Strong Families Study. In the Building Strong Families survey, nonresponse was less than 1 percent for these items.
Intimate partner violence	HM programs aim to improve the health of relationships. Documenting the baseline prevalence within the study sample is important for describing the characteristics of couples who seek these services and for use in sub-group analyses. The questions on intimate partner violence are drawn from the Conflict Tactic Scale—a widely used tool for measuring domestic violence in research studies (Straus and Douglas 2004). In Building Strong Families, these questions were used with a nonresponse of less than 1 percent.

The (14) in-depth, in-person interview guides (for program participants) may generate discussion of topics that may be considered sensitive such as criminal history, substance use, and attitudes as part of the course of conversation, though this information will not always be directly requested. Similarly, participants may also volunteer information of a sensitive nature during check-in calls (using the (15) telephone check-in guide) that ask about changes in their lives though those calls will not probe specifically for details about sensitive topics. Prior to starting each wave of the qualitative interviews and each qualitative interview check-in call, all respondents will be informed that their identities will be kept private and that they do not have to answer questions that make them uncomfortable. Furthermore, all respondents will be informed that they can decline to answer any question that they do not wish to answer, with no negative consequences for not responding. Table A.5 presents the justification for the sensitive topics included in the (14) in-person, in-depth interview guide and (15) telephone check-in guide.

Table A.5. Justification for Sensitive Questions in the (14) In-Person, In-Depth Interview Guide and (15) Telephone Check-in Guide

Question Topic	Justification
Respondent’s experience in family of origin, and relationship to his own father	This question will not be of a sensitive nature in most cases. However, respondents may choose to reveal incidences of childhood abuse or neglect during this part of the conversation. Interviewers will not probe specifically for information of that nature, but whatever a respondent does choose to reveal will be valuable to the evaluation as it helps frame the respondent’s perspective on family and parenting. These questions were asked successfully in other studies using qualitative in-depth interviews, such as the Fragile Families and Child Wellbeing Study, the Partnership for Fragile Families, and the Parents Fair Share Study.
Respondent’s pathway to becoming a parent and forming a family	While the pathway to family formation will not reveal sensitive information for most respondents, some may choose to discuss their sexual experiences or attitudes that led to family formation and parenting. This information is necessary to understand a father’s perspective on his role and how it influences his current partnering and parenting relationships, as well as his attitudes towards employment and program participation. These questions were asked successfully in qualitative in-depth interviews conducted as part of the Fragile Families and Child Wellbeing Study and the Evaluation of the Partnership for Fragile Families
Respondent’s challenges in the areas of physical and mental health, substance abuse, and involvement with the criminal justice system; Influence of challenges in the areas of physical and mental health, substance abuse, and involvement with the criminal justice system on the respondent’s parenting, employment, and program participation.	These related conversation topics are the most likely to generate sensitive information for most respondents. Recent research suggests that a history of incarceration and involvement with the criminal justice system may be fairly common among fathers in the PACT target population (Pearson et al. 2011). Parental incarceration has major negative effects on child and family well-being, reducing the financial support and other types of support the parents can provide to their children and families. Similar questions have been included in other large national studies, such as the Fragile Families and Child Wellbeing Study, the National Job Corps Study, and the Building Strong Families Study. In the Building Strong Families survey, nonresponse was less than 1 percent for these items. Similarly, we expect that health and substance use challenges are prevalent among the PACT target population. These factors are likely to influence partnering, parenting, employment prospects and histories, and program participation. Respondents may choose not to share this information, but what they do share will be valuable to understanding the circumstances and dynamics of their lives. These types of questions were asked successfully in qualitative in-depth interviews in the Fragile Families and Child Wellbeing Study, the Evaluation of the Partnership for Fragile Families, and Roy and Dyson’s (2005) study of incarcerated fathers.
Respondent’s relationships (including romantic relationships) over the last 12 months and the influence of changes in relationships (or romantic relationships) on day-to-day life	In general, this question will not be sensitive. However, the discussion may prompt a respondent to raise a discussion of sexual behavior. The decision to reveal this information will be strictly left to the respondent, but if it is revealed this information will be important to understanding the respondent’s circumstances (for example, the circumstance of a male respondent having a baby with a woman who is not his current partner) and his view of parenting and partnering relationships. These questions were asked successfully in other studies using qualitative in-depth interviews, such as the Fragile Families and Child Wellbeing Study, the Evaluation of the Partnership for Fragile Families, and Palkovitz’s study (2002) of fatherhood on men’s adult development.

In the Hispanic RF sub-study, the (18) questionnaires (for program participants in focus groups) contain questions about religious affiliation that may be considered sensitive.

Table A.6. Justification for Sensitive Questions in (18) Father Questionnaire in Hispanic RF Sub-study

Question Topic

Justification

Religious affiliation (Hispanic substudy)

Prior research indicates higher levels of religiosity among Hispanic groups, although differences may exist depending on individuals’ extent of acculturation. This information is needed to explore how culturally-related factors may affect Hispanic fathers’ engagement and participation in fatherhood programs. Similar questions about religious affiliation have been asked in many other surveys, including the General Social Survey.

The additional newly-proposed instruments included in this submission – that is, those associated with the implementation strategy within the impact study and the Hispanic RF implementation/qualitative only sub-study – focus on the experiences of program and community organization staff with their jobs, as well as participants’ impressions of and experiences with the RF/HM program, and do not include sensitive questions.

12. Estimates of Annualized Burden Hours and Costs

The estimated reporting burden for the data collection instruments included in this ICR is presented in Tables A.6 through A.9.

For cost calculations in all of the following tables, we estimate the average hourly wage for staff at the grantee organizations is the average hourly wage of “social and community service managers” taken from the U.S. Bureau of Labor Statistics, National Compensation Survey, 2010 ($27.86). The average hourly wage of program applicants is estimated from the average hourly earnings ($4.92) of study participants in the Building Strong Families Study (Wood et al. 2010). These average hourly earnings are lower than minimum wage because many study participants were not working. We expect that to also be the case for PACT study participants.

Previously Approved Burden: (1) Selecting Study Grantees and (2) RF Introductory Script, (3) RF Baseline Survey, and (6) RF Study MIS

Table A.7 summarizes burden and costs for previously approved instruments. The burden and costs for all these instruments are annualized over three years, meaning the total number of respondents over the three year period has been divided by three to determine the annual number of respondents and calculate annual burden estimates.

Table A.7. PREVIOUSLY APPROVED AND ONGOING – Estimates of Burden and Costs for the PACT Evaluation

Activity/Respondent	Annual Number of Respondentsa	Number of Responses per Respondent	Average Burden per Response (hours)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
SITE SELECTION
(1) Selecting study grantees1 Discussions/ program and partner organization staff	50	1	1	50	$29.34	$1,467
IMPACT
Responsible Fatherhood Grantee Impact Evaluation
(2) Introductory script2
a) Program staff	30	70.2	0.167	351	$27.86	$9,779
b) Program applicants	2,105	1	0.167	351	$4.92	$1,726
(3) Baseline survey2
Study participants	2,000b	1	0.5	1,000	$4.92	$4,920
Responsible Fatherhood Grantee Implementation Evaluation
(6) RF study MIS2
Program staff	30	2,533	0.03	2,533	$27.86	$70,569
Total				4,285		$88,461

^a Burden estimates are annualized over three years.

^b Note that the 2,000 “study participants” are part of the 2,105 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study and complete the baseline survey, thus there are 5 percent more program applicants than study participants.) The 2,000 study participants do not represent 2,000 individuals in addition to the 2,105 program applicants.

¹ Approved April 20, 2012.

² Approved October 31, 2012.

Burden for Data Collection Efforts Covered by this ICR

Impact Evaluation

Table A.8 summarizes burden and costs for newly requested instruments associated with the impact study (which, as stated in A1, contains experimental impact evaluations complemented with implementation and qualitative strategies). The burden and costs for all these instruments are annualized over three years, meaning the total number of respondents over the three year period has been divided by three to determine the annual number of respondents and calculate annual burden estimates. Descriptions of calculations for burden are provided in Appendix J.

Table A.8. CURRENT REQUEST – Estimates of Additional Burden and Costs for the Impact Portion of the PACT Evaluation

Instrument Respondent	Annual Number of Respondentsa	Number of Responses per Respondent	Average Burden per Response (hours)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
IMPACT
Healthy Marriage Grantee Impact Evaluation
(4) Introductory script
Program staff	30	70.2	0.167	351	$27.86	$9,779
Program applicants	4,210	1	0.167	703	$4.92	$3,454
(5) Baseline survey
Study participants	4,000 b	1	0.5	2,000	$4.92	$9,840
Healthy Marriage Grantee Implementation Evaluation – MIS
(7) HM Study MIS
Program staff	30	3,400	0.03333	3,400	$27.86	$94,724
Responsible Fatherhood and Healthy Marriage Grantee Implementation Evaluation – Additional Implementation Data Collection Instruments
(8) Semi-structured interview topic guide
Program staff	250	2	1.033	517	$27.86		$14,404
(9) On-line survey
Program staff	250	2	0.5	250	$27.86		$6,965
(10) Telephone interviews (with staff at referral organizations)
Program staff at referral organizations	50	1	0.5	25	$27.86		$697
(11) On-line Working Alliance Inventory
1) Program staff	50	20	0.167	167	$27.86		$4,653
2) Program Participants	1,000	1	0.167	167	$4.92		$822
(12) Focus group guide
Program participants	600	1	1.5	900	$4.92		$4,428
(13) Telephone interviews
Program participants (program dropouts)	150	1	0.25	38	$4.92		$187
Responsible Fatherhood Grantee Qualitative Evaluation
(14) Guide for in-person, in-depth interviews
Study participants	32	3	2	192	$4.92		$945
(15) Check-in call guide
Study participants	32	4	0.167	21	$4.92		$103
Total				8,731			$151,001

^a All burden estimates are annualized over three years.

^b Note that the 4,000 “study participants” are part of the 4,210 “program applicants,” as the study participants will all begin as program applicants. (Five percent of program applicants are not expected to agree to participate in the study, thus there are 5% more program applicants than study participants.) The 4,000 study participants do not represent 4,000 individuals in addition to the 4,210 program applicants.

Implementation/Qualitative Only Evaluation: Hispanic RF Sub-study

Table A.9 summarizes burden and costs for newly requested instruments associated with the Hispanic RF implementation/qualitative only sub-study. The burden and costs for all these instruments are annualized over three years, meaning the total number of respondents over the three year period has been divided by three to determine the annual number of respondents and calculate annual burden estimates. Calculations for burden are provided in Appendix J.

Table A.9. CURRENT REQUEST – Estimates of Burden and Costs for the Hispanic RF Implementation/Qualitative Only Sub-study of the PACT Evaluation

Instrument Respondent	Annual Number of Respondentsa	Number of Responses per Respondent	Average Burden per Response (hours)	Total Annual Burden Hoursa	Average Hourly Wage	Total Annualized Cost
IMPLEMENTATION/QUALITATIVE ONLY
Responsible Fatherhood Grantee Implementation Evaluation – Grantees with a Focus on Hispanic Populations
(16) Semi-structured interview topic guide
Program staff	42	1	1.5	63	$27.86		$1,755
(17) Focus group guide
Program participants	20	1	1.5	30	$4.92		$248
(18) Questionnaires
Program participants in focus groups	20	1	0.333	7	$4.92		$34
Total				100			$2,037

^a All burden estimates are annualized over three years.

Combined Total Burden

Table A.10 summarizes the total estimated reporting burden and costs for the previously approved and currently requested burden. If the current request is approved, 13,115 hours and $241,149 would be approved for the PACT study.

Table A.10. Estimate of Burden and Cost for the PACT Evaluation – TOTAL Burden Request

Data Collection	Total Annual Burden Hoursa	Total Annualized Cost
Previously Approved and Ongoing (from Table A.6)	4,285	$88,461
Current Request – Impact Evaluation (from Table A.7)	8,731	$151,001
Current Request – Hispanic RF Implementation/Qualitative Only Sub-study (from Table A.8)	100	$2,037
Total	13,116	$241,149

^a Burden estimates are annualized over three years.

13. Estimates of Other Total Cost Burden to Respondents and Record Keepers

These information collection activities do not place any additional costs on respondents or record keepers.

14. Cost to the Federal Government

As reported in PACT’s initial ICR for field data collection, the total cost of the PACT study to the federal government is estimated to be $19,225,787. Since the study will last five years, the total cost over this three year request is $11,535,471 and the annualized cost to the federal government is $3,845,157.

15. Explanation for Program Changes or Adjustments

This submission is for additional data collection under the Parents and Children Together evaluation and therefore increases total burden under OMB Control number 0970-0403.

16. Plans for Tabulation and Publication and Project Time Schedule

a. Plans for Tabulation

New Impact Evaluation (I.e. Experimental Impact Complemented with Implementation and Qualitative Strategies) Data Collection Instruments

The (5) HM baseline survey data will be used to describe the characteristics of HM program participants across all study grantees. For each grantee, tables will present frequencies and means for key participant characteristics, including demographic and family information.

Baseline survey data will also be used in the impact analysis to test for baseline equivalence, define subgroups, improve the precision of impact estimates, and estimate factors that predict participation in the program. The goal of the impact analysis is to provide statistically valid and reliable estimates of the effects of HM programs on the outcomes of participants. To do so, observed outcomes for members of the program group—couples eligible for program services—will be compared with outcomes for members of a control group that was not offered program services. The experience of the control group will be used as a measure of what would have happened to the program group members in the absence of the HM program. Random assignment of couples to a program and a control group ensures that the two groups of couples do not initially differ in any systematic way on any characteristic, observed or unobserved. Any observed differences in outcomes between the program and control group fathers can therefore be attributed to the program with a known degree of precision.

Though random assignment ensures that couples in the program and control groups do not initially differ in any systematic way, there might still be chance differences between groups. To confirm that there were no differences between the program and control groups before random assignment, key characteristics of the program and control groups at baseline will be compared. In particular, t-tests and F-tests will be conducted to test for differences between the two groups both overall and separately by grantee.

Baseline survey data will also be analyzed jointly with the follow-up survey data to estimate impacts. Using baseline data in the impact analysis will improve the statistical precision of impact estimates. Differences of means or proportions in outcomes between the program and control group would provide unbiased estimates of the impacts of being offered participation in the HM program (referred to as the intent-to-treat effect, or ITT effect). However, the impact analysis will use regression models to estimate the ITT effect, allowing us to control for random differences in the baseline characteristics of program and control group members.

Finally, data from the baseline survey will be used to estimate the impact of receiving program services, the effect of treatment on the treated (or the TOT effect). In many settings, the TOT effect can be calculated by adjusting the ITT effect for the difference between the program and control groups in program participation rates. However, in this context, HM programs offer a range of services, and as a result participants may take up only some of those services. For example, program participants might only attend some group meetings or might choose to participate in only some components of the program. Because we are interested in understanding how the impact of the programs varies with the type and intensity of services received, the TOT effect must be calculated using quasi-experimental methods—techniques that do not rely solely on the study’s random assignment design (see Wood et al. 2011 for an application of these methods). To estimate the TOT effect, we will use data from the baseline survey and from the study MIS to predict program participation; possible predictors include motivation to change, barriers to participation, grantee staff predictions of participation, and information on referral source. If participant self-reports and grantee staff assessments are predictive of participation among program group members, we will be able to estimate the TOT effect in addition to the ITT effect.

The (7) HM Study MIS information will also be used for the implementation analyses. The implementation analyses will provide summary statistics for key program features:

• Enrollment patterns. For example, the average number of new applicant couples each month.

• Services provided by grantees. For example, the average number of group workshops offered each month, the average number of service contacts each month, or the percentage of service contacts provided in couples’ homes or in the office.

• Participation patterns. For example, the number of couples that engage in a group activity within two months of enrollment and the average number of hours of group workshops received by program couples.

We expect that data from the study MIS for each grantee will be analyzed at three points in time which correspond to the three planned implementation reports. Each report will describe enrollment patterns, services provided, and participation patterns over the previous 12 months. Analyses will use basic descriptive statistics, such as means, proportions, and standard deviations. Later analyses may describe how patterns changed over time, such as from the early to late implementation period.

The instruments included in this submission for implementation data collection within the RF and HM Impact Evaluations will be analyzed using qualitative and quantitative methods to describe program implementation, assess its overall quality, analyze the factors that appear to be linked to quality, and identify lessons for future practice. A thorough understanding of program implementation will provide context for interpreting program impacts, while a greater understanding of how programs can be implemented with high quality is expected to inform the next generation of RF/HM programming.

The contractor will use standard qualitative procedures to analyze and summarize information from interviews conducted using the (8) semi-structured interview topic guide (for program staff), the (10) telephone interview guide (for program staff at referral organizations), the (12) focus group discussion guide, and the (13) telephone interview guide (for program dropouts). Analysis will involve organization, coding, triangulation, and theme identification. For each qualitative data collection activity, standardized templates will be used to organize and document the information and then code this documentation. Coded text will be searched to gauge consistency and triangulate across respondents and data sources. This process will reduce large volumes of qualitative data to a manageable number of topics/themes/categories (Yin 1994; Coffey and Atkinson 1996) which can then be analyzed to address the study’s research questions.

(Outside of the (3)RF/(5)HM baseline surveys, which will be used in the rigorous analysis described above) quantitative data for remaining instruments will be summarized using basic descriptive methods. Sources of quantitative data include the (9) on-line survey (for program staff), the (10) on-line Working Alliance Inventory (for program staff and participants). Analysis of data from each source will follow a common set of steps involving data cleaning, variable construction, and computing descriptive statistics. To facilitate analysis of each data source we will create variables to address the study’s research questions. Construction of these analytic variables will vary depending on a variable’s purpose and the data source being used. Variables may combine several survey responses into a scale, aggregate attendance data from a set time period, or compare responses to identify a level of agreement.

With regard to the qualitative data collected through the (14) in-person, in-depth interview guide and (15) telephone check-in guide (both for program participants), the study team will organize the data obtained by topic or theme, developing a thematic codebook based on both deductive and inductive (or “open”) thematic coding after reviewing a random set of five transcripts. Deductive codes are predetermined by the research questions and the interview protocol. Inductive codes “bubble up” up from the data. For example, coders might find that fathers repeatedly use the phrase “being there” to capture what they believe is a key dimension of fatherhood—the determination to stand by one’s child no matter what troubles arise. Once a draft thematic codebook is developed, the entire coding team will be trained to reliably code the remaining transcripts, as well as interview transcripts and case notes. An electronic database will store the data and codes for use with qualitative data analysis software.

Once data are sorted into thematic codes, the contractor will conduct analytic subcoding. This stage does not just merely sort data by topic or theme, but rather allows the study team to analyze what has been gleaned from a particular thematic code. Just as in the thematic coding stage, analytic subcodes will be both inductively and deductively derived. But, while the first stage of coding focused on deductive coding, in this second analytic stage, the inductive approach will dominate. The analysis will initially focus on identifying commonalities across all fathers in the study, such as how participants navigate complex family structures, and how that process may shape patterns of employment, program participation, and father involvement. The study team will describe the typical respondent’s notions of what a good father ought to do for his child, as well as how these notions relate to how they interact with the program, the messages they receive from program staff, and their views of their own fathers and how those shape their current relationship with their children.

This subcoding process may also reveal categories that would yield fruitful sources of comparison. It may turn out, for example, that fathers for whom a relationship with a social child (a non-biological child with an active presence in his life) is highly salient feel responsible for the wellbeing of that child and may have more difficulty responding positively to a set of program messages about providing for one’s biological children. By transforming “salience of social child” into a categorical variable, we can address the question of whether this is in fact the case, at least in our small sample. The process is particularly suited for analyzing the network tree data gleaned from the interview. The study team will assess how the complexity in fathers’ relationships, and the strategies men devise to accommodate it, are related to their behaviors as fathers, partners, providers, and workers.

Following this initial analysis, the study team will systematically compare similarities and differences across fathers with various characteristics. Several obvious dimensions for comparison are program type, race/ethnicity, age of father, child gender, residing (or not) with their child, but many more are possible. Racial and ethnic variation will be a special focus in the subgroup analyses. Due to small sample size, we cannot rigorously explore “within-program” questions (that is, “in program X, how do African American fathers’ preexisting views about the father role ‘fit’ with the program message as compared to Hispanic fathers’ preexisting views?”). However, programs may be grouped by type in ways that allow us to address these kinds of questions more reliably.

Implementation/Qualitative Only Evaluation: Hispanic RF Sub-study

The contractor will use standard qualitative procedures to analyze and summarize information from the (16) semi-structured interview topic guide (for program staff) and the (17) focus group discussion guide (for program participants). Analysis will involve organization, coding, triangulation, and theme identification. For each qualitative data collection activity, standardized templates will be used to organize and document the information and then code this documentation. We will search the coded text to gauge consistency and triangulate across respondents and data sources. This process will reduce the qualitative data to a manageable number of topics/themes/categories (Yin 1994; Coffey and Atkinson 1996) which can then be analyzed to address the study’s research questions.

Quantitative data from the Hispanic (18) questionnaires (for focus group participants) will be summarized using basic descriptive methods. Analysis of data will follow a set of steps involving data cleaning, variable construction, and computing descriptive statistics. To facilitate analysis, variables will be created to address the study’s research questions; for example, variables may combine several survey responses into a scale.

b. Time Schedule and Publications

The PACT study is expected to be conducted over a five-year period beginning on September 30, 2011. Discussions with sites began after April 20, 2012, when first ICR approval was received (for such discussions to begin). This ICR is for three years. Table A.11 provide the expected schedule for the study.

Table A.11. Schedule for the Evaluation

Activity

Date

Selection of grantees for inclusion in evaluation Intake period for impact evaluation Report on early findings on implementation Implementation mid-term report Data collection for Hispanic RF Sub-study Report on Hispanic RF Sub-study First report on qualitative interview analysis Second report on qualitative interview analysis Final implementation report Final impact report Final report on qualitative interview analysis

Spring 2012 to spring 2013 December 2012–August 2014 Summer 2013 Spring 2014 Summer-fall 2013 Winter 2013/2014 Winter 2014/2015 Fall 2015 Winter 2016 Summer–fall 2016 Summer–fall 2016

In addition to the planned reports described above, PACT provides opportunities for analyzing and disseminating additional information through special topics reports and research or issue briefs on an as requested basis. We will also provide a restricted use data file for others to replicate and extend our analysis.

17. Reason(s) Display of OMB Expiration Date Is Inappropriate

All instruments will display the expiration date for OMB approval.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

REFERENCES

Avellar, Sarah M., Robin Dion, Andrew Clarkwest, Heather Zaveri, Subuhi Asheer, Kelley Borradaile, Megan Hague Angus, Timothy Novak, Julie Redline, and Marykate Zukiewicz. “Catalog of Research: Programs for Low-Income Fathers.” Washington, DC: The evaluator Policy Research, June 2011.

Axinn, William G., and Lisa D. Pearce. “Innovations in Life History Calendar Applications.” Social Science Research, vol. 28, 1999, pp. 243–264.

Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto 1992 An experiment in monetary incentives. Pp. 393-398 in Proceedings of the Section on Survey Research Methods. Alexandria, VA: American Statistical Association.

Bir, Anupa, Robert Lerman, Elise Corwin, Brian MacIlvain, Allison Beard, Kelly Richburg, and Kevin Smith (2012). Impacts of a Community Healthy Marriage Initiative, OPRE Report # 2012–34A, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

Coffey, A., B. Holbrook and P. Atkinson (1996) 'Qualitative Data Analysis: Technologies and Representations', Sociological Research Online, vol. 1, no. 1, <http://www.socresonline.org.uk/ 1/1/4.html>.

Dion, Robin M., Sarah A. Avellar, and Elizabeth J. Clary. “The Building Strong Families Project: Implementation of Eight Programs to Strengthen Unmarried Parent Families.” Final report submitted to the U.S. Department of Health and Human Services, Office of Planning, Research and Evaluation, Administration for Children and Families. Washington, DC: The evaluator Policy Research, May 2010.Glasner, Tina, and Wander van der Vaart. “Applications of Calendar Instruments in Social Surveys: A Review.” Quality & Quantity, vol. 43, 2009, pp. 333–349.

Edin, Kathryn, and Al Young. September 27, 2012. Personal communication.

England, Paula, and Kathryn Edin (eds). Unmarried Couples with Children. The Unfolding Lives of New Unmarried Urban Parents. New York, NY: Russell Sage Foundation, 2007.

Fagan, J, Palkovitz, R, Roy, K, and Farrie, D (2009).
Pathways to Paternal Engagement: Longitudinal Effects of Risk and Resilience on Nonresident Fathers
Developmental Psychology, 45:1389-1405.

Fitzgerald, M, Roy, K, Anderson, E, and Letiecq, B (forthcoming). The Effect of Depressive Symptoms on Low-Income Men in Responsible Fathering Programs.


Guzzo, Karen Benjamin, and Frank F. Furstenberg, Jr. “Multipartnered Fertility Among American Men.” Demography, vol. 44, 2007, pp. 583–601.

Hsueh, JoAnn, Desiree Principe Alderson, Erika Lundquist, Charles Michalopoulos, Daniel Gubits, David Fein, and Virginia Knox (2012). The Supporting Healthy Marriage Evaluation: Early Impacts on Low-Income Families. OOPRE Report 2012-11. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services.

James, Jeannine M., and Richard Bolstein.1990 The effect of monetary incentives and follow-up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly 54:346-361.

Johnson, Earl, Ann Levine, and Fred Doolittle. Fathers’ Fair Share: Helping Poor Men Manage Child Support and Fatherhood. New York, NY: Russell Sage Foundation, 1999.

Marsiglio, William, Sally Hutchinson and David Nibert. Sex, Man, and Babies: Stories of Awareness and Responsibility. New York: New York University Press, 2002.

McLanahan, Sara. “Fragile Families and the Reproduction of Poverty.” Annals of the American Academy of Political and Social Science. vol. 621, no. 1, 2009, pp. 111–131.

Palkovitz, Rob. Involved Fathering and Men’s Adult Development: Provisional Balances. Mayway, NJ: Lawrence Erlbaum Associates, Inc., 2002.

Pearson, Jessica, Lanae Davis, and Jane Venohr. “Parents to Work!” Denver: Center for Policy Research, February 2011.

Roy, Kevin and Omari Dyson. “Making Daddies into Fathers: Community-Based Fatherhood Programs and the Construction of Masculinity for Low-Income African-America Men.” American Journal of Community Psychology, vol. 45, no. 3–4, 2010, pp. 139–154.

Roy, Kevin and Omari Dyson. Gatekeeping in context: Babymama drama and the involvement of incarcerated fathers. Fathering: A Journal of Theory, Research, and Practice about Men as Fathers, 3, 289-310, 2005.

Roy, Kevin (2006) Father Stories: A Life Course Examination of Paternal Identify Among Low-Income African American Men. Journal of Family Issues, 27:31-54,

Roy, K and Vesely, C (2009). Caring for “The Family’s Child”: Social Capital and Kin Networks of Young Low-Income African American Fathers in The Myth of the Missing Black Father, R. Coles and C. Green (eds), New York: Columbia University Press.

Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. “Does the Payment of Incentives Create Expectation Effects?” Public Opinion Quarterly. vol. 62, 1998, pp. 152–164.

U.S. Census Bureau. “Family Structure and Children’s Living Arrangements.” Available at [http://www.childstats.gov/americaschildren/famsoc1.asp.] Accessed December 24, 2011.

Tach, Laura, and Kathryn Edin. “The Relationship Context of Young Disadvantaged Men.” The ANNALS of the American Academy of Political and Social Science, vol. 635, 2011, pp. 76–94.

Tach, Laura, Ronald Mincy, and Kathryn Edin. “Parenting as a ‘Package Deal’: Relationships, Fertility, and Nonresident Father Involvement Among Unmarried Parents.” Demography, vol. 47, 2010, pp. 181–204.

U.S. Government Accountability Office. “Healthy Marriage and Responsible Fatherhood Initiative: Further Progress Is Needed in Developing Risk-Base Monitoring Approach to Help HHS Improve Program Oversight.” Washington, DC: U.S. Government Accountability Office, September 2008.

Weiss, R. S. (1994). Learning from strangers: The art and method of qualitative interview studies. New York: The Free Press.

Wood, Robert G., Sheena McConnell, Quinn Moore, Andrew Clarkwest, and JoAnn Hsueh. “Strengthening Unmarried Parents’ Relationships: The Early Impacts of Building Strong Families.” Princeton, NJ: Mathematica Policy Research, May 2010.

Wood, Robert G., Quinn Moore, and Andrew Clarkwest. “BSF’s Effects on Couples Who Attended Group Relationship Skills Sessions: A Special Analysis of 15-Month Data.” Princeton, NJ: Mathematica Policy Research, May 2011.

Wood, Robert G., Quinn Moore, Andrew Clarkwest, Alexandra Killewald, and Shannon Monahan (2012). The Long-Term Effects of Building Strong Families: A Relationship Skills Education Program for Unmarried Parents, OPRE Report # 2012-28A, Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services (OPRE).

Yin, R. (1994). Case study research: Design and methods (2nd ed.). Thousand Oaks, CA: Sage Publishing.

1 Variations will be made that are appropriate to RF or HM programs and participants.

2 This is the same level that was proposed and approved by OMB Oct. 31, 2012 for fathers completing the RF Baseline Survey.

3 In a study for the U.S. Housing and Urban Development (HUD) entitled Moving to Opportunity, $50 was provided as a gift for a 60-minute interview with the household head (OMB control number 2528-0161). In PACT, the $60 gift would be in appreciation of the respondent’s participation in a two-hour in-depth interview and interim check-in call.

4 For example, February 21, 2012, conversation with Joseph Jones, Jr., President and CEO of Center for Urban Families, a fatherhood program in Baltimore.