Examining Equity of the Administration for Children and Families’ Healthy Marriage and Responsible Fatherhood (HMRF) Client Surveys
Administration for Children and Families Generic for Engagement Efforts
0970 – 0630
Supporting Statement Part B
August 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Overview of Study Objectives
This study will examine how clients who participated in Healthy Marriage (HM) and Responsible Fatherhood (RF) programs perceive questions on existing client surveys that the Administration for Children and Families (ACF) uses to assess program performance. This information collection will use an equity lens to examine the questions in the client surveys (that is, the applicant characteristics, entrance, and exit surveys) for three types of HMRF grant recipients: HM programs for youth, HM programs for adults, and RF programs. In particular, the study is exploring whether the survey questions are fair and the extent to which survey questions respectfully represent a variety of cultural values, beliefs, and judgments.
This work will be conducted by staff from Mathematica, the contractor to ACF for the Building Usage, Improvement, and Learning with Data in HMRF Programs (BUILD-HMRF) project. The Mathematica study team will conduct focus groups and, if necessary, interviews with clients of different races and ethnicities who participated in ACF-funded HMRF services during the current grant recipient cohort (2020-2025). ACF will use findings from the focus groups and interviews to inform revision of the surveys for the potential next cohort of grant recipients.
Respondent Universe and Sampling Methods
Target Population
The respondent universe is clients who have completed HMRF programs that ACF has funded in the 2020 – 2025 cohort. To avoid influencing the clients’ responses to the exit survey, the universe is limited to clients who have completed the workshops and either completed the exit survey or declined to take it. Clients complete their last (exit) survey at the end of the workshop, and ACF and grant recipients use client entrance and exit surveys to measure change over time. Participating in a focus group or interview about the survey questions before completing the exit survey could lead to clients answering the exit survey questions differently, thereby impacting program monitoring and the pre-post analysis conducted by ACF and grant recipients.
The study will focus on clients in the following HMRF populations:
HM youth: Youth ages 13 to 24
HM adults: Adult men and women aged 18 and older
RF community fathers: Adult men aged 18 and older who are a father or father figure and live in the community.
Sampling and Site Selection
The study will use a convenience sample. In the first stage, the Mathematica study team, with ACF’s approval, will reach out to selected grant recipients. Selected grant recipients will serve clients of different races and ethnicities and have a workshop that will end within the data collection window. If more grant recipients are interested than are needed, ACF and Mathematica will select grant recipients to maximize the racial and ethnic diversity of the sample (based on who the grant recipients serve). ACF will also consider contacting grant recipients that express interest in participating after learning about the study plans.
In the second stage, the Mathematica study team will work with the grant recipients to identify a workshop or workshops of interest. Staff from the grant recipient will share information about the study with the clients of the selected workshops, assisting the Mathematica study team with recruitment of their clients to the study.
Appropriateness of Study Design and Methods for Planned Uses
This information collection is for focus groups and interviews with clients who participate in ACF-funded HMRF services in the current grant recipient cohort (2020-2025).
The focus groups and interviews will incorporate cognitive interviewing techniques. Facilitators of the focus groups will use cognitive interviewing techniques, a common approach for identifying issues with survey questions, to prompt discussion and draw out clients’ nuanced reactions to the survey questions. The facilitators will use group activities, think-aloud interviewing, and verbal probing to encourage clients to describe their interpretations of a question, relevance or appropriateness of the questions, ease or difficulty in answering questions, and completeness or gaps in the response categories.1 Focus groups facilitate group interactions and exchange that can uncover shared experiences and cultural patterns in responses. To allow flexibility in data collection and maximize the sample size, without increasing burden, the study team may conduct interviews when it is not feasible to hold a focus group.
To gather feedback on a range of questions, each focus group and interview will cover surveys developed for the relevant HMRF client population. During the focus group or interview, Mathematica staff will ask for clients’ feedback on the experience of taking the surveys and their opinions on specific survey questions. The focus group and interview will mainly emphasize questions from the exit survey that are also asked on the entrance survey, which will enable ACF and grant recipients to assess clients’ attitudes and behaviors when they begin and end the workshops. The focus group and interview may also explore additional exit survey questions that address clients’ experiences during the program, and a few questions on the Applicant Characteristics survey (such as gender identity). All client surveys have prior OMB approval (OMB No.: 0970-0566, Expiration date: 03/31/2027) and are currently used by grant recipients in the 2020 – 2025 HMRF cohort.
This study uses a convenience sample and the findings will not be representative of all HMRF clients or grant recipients. This limitation will be noted in all verbal and written presentations of the findings.
As noted in Supporting Statement Part A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. The Mathematica study team will recommend revisions to survey items based on multiple analyses (see Supporting Statement Part A, section A.2).
Procedures for Collection of Information
Data Collection Processes
Mathematica study team members will review project-specific procedures for obtaining parental consent, such as those required by Mathematica’s Institutional Review Board, with the grant recipient staff who will be collecting consent. While recruitment and data collection are ongoing, Mathematica study team members will be available to answer questions about the consent or study from grant recipient staff and HMRF clients.
Mathematica staff will facilitate the focus groups and interviews and collect data. All staff will have experience collecting qualitative data and familiarity with HMRF program components and goals. Mathematica facilitators will receive training on the goals of the study and the protocols before the start of the data collection.
During the data collection, Mathematica study team members will review focus group and interview notes to monitor adherence to the protocols and appropriateness of probes. The Mathematica study team will meet regularly as a group during data collection to discuss challenges and remediate any issues with data collection.
Data Handling
The Mathematica study team will use the detailed notes and transcriptions for data analysis. Reliance on detailed notes rather than summaries will allow the Mathematica study team to include respondents’ voices and direct quotes in the final memo. However, the study team will not attribute any responses to individuals by name in the notes, transcriptions, or report. For completeness and accuracy of the notes, the data collectors will review the video or audio recordings to fill in any words, phrases, or portions of text that are unclear in the notes. After using the recording to verify the notes, the Mathematica study team will destroy it to protect client privacy. For more information about protecting the privacy of clients who participate in the study, see A10 in Supporting Statement Part A of this package.
Data Analysis
For each focus group and interview, the Mathematica study team will record the notes in a spreadsheet organized by question. After each focus group and interview, the facilitator and note taker will review the notes and discuss any discrepancies in their interpretations or omissions. If needed, they will refer to the recordings to address the issue. The Mathematica study team will use these finalized notes to identify issues with the existing questions across the range of feedback and to propose possible solutions for them.
Use of Information Collected
The Mathematica study team will summarize findings in an internal ACF memo and briefing to ACF, which will also note any limitations of the data. High-level feedback might be incorporated into a public report summarizing ACF’s proposed revisions to the surveys and data collection guidance. The primary purpose of the data collection is to inform revisions to performance measures for the HMRF grant program.
Methods to Maximize Response Rates and Deal with Nonresponse
Response Rates
The focus groups and interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
NonResponse
Because focus group and interview respondents will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.
Test of Procedures or Methods to be Taken
Development of Data Collection Instruments
The study team developed five data collection instruments for this study. These include a grant recipient recruitment protocol that Mathematica staff will use during a phone call with grant recipients to describe the data collection and answer any questions the grant recipient staff have (Instrument 1), a demographic survey to collect information from the clients participating in focus groups and interviews (Instrument 2) and three protocols that correspond to the different HMRF client types that will participate in the focus group or interview (Instrument 3: HM youth focus group and interview protocol; Instrument 4: HM adult focus group and interview protocol; and Instrument 5: RF community fathers focus group and interview protocol). Each protocol can be used for either a focus group or an individual interview.
Questions in the demographic survey (Instrument 2) are based on the demographic items asked on the existing HMRF program client surveys (OMB No.: 0970-0566, Exp. 03/31/2027), with a new question on race and ethnic identity, consistent with OMB’s new guidance for collecting this information.2
The focus group and interview protocols (Instruments 3 – 5) are structured around the questions asked on the existing HMRF program client surveys (OMB No.: 0970-0566, Exp.: 03/31/2027). Before the focus group or interview begins, clients will complete the program client exit survey (or decline to complete it) as part of standard data collection for grant recipients. For the focus groups or interviews, the study team will provide a packet with the survey questions for reference. Participating clients will be asked to discuss their feedback on the surveys, either as a group (focus group) or individually (interview). The questions and probes the facilitators will use to gather feedback are based on the study team’s experience conducting other focus groups and interviews about survey measures.
The focus group and interview protocols were developed by content experts at Mathematica and informed by reviewing instruments used in similar data collection efforts, including from the Youth Empowerment Information, Data Collection, and Exploration of Avoidance of Sex (IDEAS) Pretest (OMB No. 0970-0355), and research on best practices for cognitive interviewing3 and cross-cultural survey guidelines.4
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The following individuals at ACF and Mathematica are leading the study team:
Rebecca Hjelm
Social Science Research Analyst
Office of Planning, Research, and Evaluation
Harmanpreet Bhatti
Social Science Research Analyst
Office of Planning, Research, and Evaluation
harmanpreet.bhatti@acf.hhs.gov
Pooja Gupta Curtin
Social Science Research Analyst
Office of Planning, Research, and Evaluation
Pooja.Curtin@acf.hhs.gov
Sarah Avellar
Principal Researcher
Mathematica
Instruments
Instrument 1: Grant recipient recruitment protocol
Instrument 2: Client demographic survey
Instrument 3: HM youth focus group and interview protocol
Instrument 4: HM adult focus group and interview protocol
Instrument 5: RF community fathers focus group and interview protocol
Appendices
Appendix A: Outreach to grant recipients
Appendix B: Client consent
1 Beatty, P.C. and G.B. Willis. “Research Synthesis: The Practice of Cognitive Interviewing.” Public Opinion Quarterly, vol. 71, no. 2, Summer 2007, pp. 287–311.
2 Office of Management and Budget. “Revisions to OMB’s Statistical Policy Directive No. 15: Standards for Maintaining, Collecting, and Presenting Federal Data on Race and Ethnicity.” 2024. Accessed at 2024-06469.pdf (federalregister.gov).
3 Beatty, P.C. and G.B. Willis. “Research Synthesis: The Practice of Cognitive Interviewing.” Public Opinion Quarterly, vol. 71, no. 2, Summer 2007, pp. 287–311.
4 Cibelli Hibben, K. and J. de Jong. “Cognitive Interviewing.” In Guidelines for Best Practice in Cross-Cultural Surveys. Ann Arbor, MI: Survey Research Center, Institute for Social Research, University of Michigan. http://ccsg.isr.umich.edu/.https://ccsg.isr.umich.edu/chapters/pretesting/cognitive-interviewing/, 2016.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Pooja |
| File Modified | 0000-00-00 |
| File Created | 2025-02-17 |