Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Strengthening the Implementation of Responsible Fatherhood Programs (SIRF)
Formative Data Collections for ACF Research
0970 - 0356
Supporting Statement
Part B
October 2020
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Katie Pahigiannis
Kriti Jain
Part B
B1. Objectives
Study Objectives
The Strengthening the Implementation of Responsible Fatherhood Programs (SIRF) Study will use iterative learning methods to identify and test promising practices for addressing critical implementation challenges in Responsible Fatherhood programs. To achieve this goal, the SIRF team will identify common implementation challenges and potential solutions, select programs to undertake iterative learning activities, and work with sites on iterative learning activities to strengthen implementation and increase capacity for participating in summative evaluations. The results from this project are intended to inform future large-scale impact evaluations of programs that adopt them.
The current information data collection request is to hold discussions with relevant programs to gather in-depth information about their current practices as they relate to the SIRF study priorities and inform the development of iterative learning methods and tests, which will be documented in a later information request package.
Generalizability of Results
This study is intended to present an internally valid description of fatherhood programs and the implementation challenges they face, and innovative practices to address those challenges in chosen sites, not to promote statistical generalization to other sites or service populations.
Appropriateness of Study Design and Methods for Planned Uses
Study activities outlined in this information collection request seek to answer the following research questions through semi-structured discussions and guiding staff to complete the customer journey mapping activity with program staff.
How do the implementation challenges and potential areas of improvement of select fatherhood programs align with SIRF priorities?
What solutions might be feasible to test in a learning cycle approach with fatherhood programs?
The information to answer these questions will be collected through semi-structured phone discussions program staff, semi-structured virtual discussions with select program leaders, staff, and program participants, and a guided activity and workshop with program staff to conduct a process known as Customer Journey Mapping. Because responses can vary, semi-structured discussions allow for flexibility during the discussion, such as restricting the discussion length to the minimum required or holding discussions in groups whenever possible. Each protocol is designed to collect the minimum information necessary to allow the study team to understand the variation of programming in the field, the range of perspectives on SIRF and assess what particular study design options will be feasible given the structure of a range of fatherhood programs.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
The target population for this data collection are staff and program participants at up to 25 programs serving fathers. During the first round of phone calls, the team will speak with up to two staff at each program via phone or video conference. After the initial round of phone calls, the study team will conduct follow-up with each program and expects to talk with up to two staff and one participant at each program.
Finally, the study team will hold the Customer Journey Mapping workshop and guided activity with four staff at up to 15 of the initial 25 fatherhood programs through virtual conference calls.
Sampling
To ensure that SIRF findings reflect the full breadth of fatherhood programs, individuals will be purposefully invited to SIRF discussions to vary perspectives by affiliation, role, geography, demographics, service approaches, etc. For example, stakeholders will be selected for their knowledge of fathers, their service needs, and the services provided to them. Likewise, the selection of programs will be based on the target population they serve (i.e. work with fathers or a specific subgroups of fathers), program size, services offered, program structure, geography, past success engaging fathers in services, along with recommendations from the project’s experts, other stakeholders, and consultation with the federal offices overseeing SIRF and Responsible Fatherhood programming.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
Each data collection instrument was designed to streamline efforts by only asking the necessary questions to achieve the objectives of the data collection and to allow flexibility in our approach given that the availability of individuals may vary. The data collection instruments will not be pre-tested as previous large-scale evaluations have successfully used similar discussion topics, such as the Building Bridges and Bonds Evaluation (OMB Control No.: 0970-0485), during information gathering processes. However, these instruments do build off the successfully executed data collection instruments - as approved under approved under the Formative Data Collections for ACF Research (OMB #0970-0356), particularly the initial phone outreach protocols.
B4. Collection of Data and Quality Control
Contractors to ACF, led by MDRC and including researchers from MEF Associates and Insight Policy Research, will gather information. In consultation with OPRE, the study team will identify up to 25 programs to conduct discussions with program staff and participants through virtual conference calls and activities.
The study team will send fatherhood program staff a request for a one-hour telephone or video conference call via email (see Appendix A: SIRF Phone Meeting Email Template). It will describe the study and its goals, introduce the team that is doing the study on ACF’s behalf, offer suggested times for a phone meeting, and will state that the phone meeting is voluntary. When reaching out to sites, the study team will attach the project description (see Appendix B: SIRF Project Description) and a summary of the topics we hope to cover during the phone call (see Appendix C: SIRF Phone Meeting Topics for Program Staff). Each semi-structured discussion will be guided by a protocol for quality assurance and consistency of the data collected.
As a follow-up to the initial phone call, the study team may request an additional telephone or video conference call with program staff using a semi-structured protocol (see Instrument 2: SIRF Follow-up Meeting with Program Staff Protocol). The study team will also speak to fathers who are currently participating or who have previously participated in the program if possible, either one-on-one or in a small group (up to two fathers per program). These discussions will be guided by a semi-structured protocol (see Instrument 3: SIRF Follow-up Meeting with Program Participant Protocol).
After exploring program challenges with staff and participants, the study team will invite staff at a select group of up to 15 programs to participate in the Customer Journey Mapping activity (see Appendix E for the email template for this invitation). In this activity, program staff will be asked to use a set of tools and techniques to map the steps in their service flow in order to identify areas where the process of enrolling and engaging fathers in program activities can be improved. Following an informational webinar (see Appendix F for the webinar agenda), the study team will ask program staff to complete short assignments and attend a virtual workshop to complete the Customer Journey Mapping exercise. The pre-work assignments involve written activities for staff to reflect on their program from the imagined viewpoint of a participant (see Instrument 4: Customer Journey Mapping Pre-Work). These assignments will culminate in a workshop will be guided by a protocol (see Instrument 5: SIRF Customer Journey Mapping Workshop) to complete the mapping exercise and allow the study team and program staff to uncover detailed insights about their program.
All discussions will be completed or overseen by senior researchers with extensive experience conducting discussions. All team members will receive background about fatherhood programming and a training on the semi-structured protocols to ensure that programs are engaged in a consistent manner. After the phone calls or video conferences, if there is missing information, the study team does not intend to follow-up with stakeholders or staff, unless the information is essential.
B5. Response Rates and Potential Nonresponse Bias
Site and Respondent Selection
The data collection activities in this request are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
Because there is minimal burden involved with the initial phone call, we expect nearly all individuals contacted will participate. The purpose of this study is to learn about program services and operations in the field, and we expect that most program directors and fathers will be eager to share this information with the study team. The programs selected for follow-up discussions, conducted virtually, will be based in part on their willingness to have an additional discussion and their availability. For this reason, we expect that most programs selected will participate.
To maximize participation, the study team will be accommodating of respondents’ schedules, including flexibility about the meeting times and date. The program will be contacted by their assigned study team liaisons, which will include at a minimum one senior and one junior staff member from the study team. The senior member has had significant experience in working closely with programs and their stakeholders on previous evaluations and has had a high level of success in engaging programs in preliminary discussions such as these.
Non-Response
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated.
Responses to the first phase of outreach under the previous package was strong (20 of 28 total contacts to programs). Similarly, we expect that most program staff and participants whom we contact will agree to participate. A small number may decline. When members of the project team reached out to programs and stakeholders earlier on the project, nonparticipation was mostly due to scheduling conflicts. The team is conducting these activities virtually to eliminate barriers to participation for programs that may be operating entirely or partly virtually due to the COVID-19 pandemic. The team offer flexible scheduling options to accommodate individuals’ availability. The SIRF team will emphasize the potential benefit to programs of participating in SIRF to increase motivation to participate. For programs that have been newly awarded a federal Responsible Fatherhood grant, the team will also communicate with federal staff and programs to be sensitive to any constraints the program may be facing in terms of scheduling since the interactions will fall in the grant planning period.
If we encounter no response to the initial email request for a meeting, we may send a second request by e-mail or follow up with a phone call, during which a member of the research team will use the e-mail template in Appendix D as a script. After these attempts, we will not continue to pursue a response. Should anyone decline our invitation, we will ask if there are any other suitable staff at their organization who might be able to participate instead.
B6. Production of Estimates and Projections
Estimates will not be produced for this data collection.
B7. Data Handling and Analysis
Data Handling
Key points of each discussions with stakeholders, program staff, and participants will be documented in Microsoft Word by a note-taker who will be present during each discussion. The information gathered will be kept private and protected securely. The study team is required to use encrypted laptops, secure storage locations, secure transfer mechanisms (when necessary), and access to secure locations will only be granted to those on a need-to-know basis.
We are not producing a data set for research purposes. The information gathered will refine study priorities and inform learning cycle designs specific to each program context and its specific challenges related to recruitment, enrollment or retention, priorities.
Data Analysis
Notes from the semi-structured discussion protocols will be organized into categories for review by the SIRF team and summarized in an internal memo.
Data Use
The information gathered will inform the learning cycle design for each program, which will be explained in a future information collection request submission. Findings from the current information collection request will not be used for the primary purpose of publication, but may be incorporated into documents or presentations that are made public, such as research design documents or reports; research or technical assistance plans; background materials for technical workgroups; or contextualization of research findings from a follow-up data collection that has full PRA approval.
B8. Contact Person(s)
Primary Project Contacts
Charles Michalopoulos, MDRC
SIRF Project Director
(510) 844 – 2235
Charles.Michalopoulos@mdrc.org
Dina Israel, MDRC
SIRF Project Deputy Director
(212) 340 – 8606
Michelle Manno, MDRC
SIRF Information Gathering Task Lead
(212) 340 – 8873
Data Collection and Analysis Staffing
Research activities with program staff, participants, and stakeholders will be conducted by trained researchers at MDRC, MEF Associates, and Insight Policy Research. Study teams will be composed of at least one senior member and one junior member for every telephone contact. These staff members are experienced in the process of gathering information for purposes of designing research projects, such as SIRF. In addition, all team members will receive a training to ensure that programs are engaged in a consistent manner. A smaller number of team members will engage in analysis; junior staff will be supervised by senior staff with previous experience in analyzing qualitative information.
Attachments
Appendix A_SIRF Phone Meeting Email Template
Appendix B_SIRF Project Description
Appendix C_SIRF Phone Meeting Topics for Program Staff
Appendix D_SIRF Phone Meeting Topics for Follow-Up Call
Appendix E_SIRF Customer Journey Mapping Email Template
Appendix F_SIRF Customer Journey Mapping Workshop Agenda
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Pahigiannis, Katie (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-13 |