SSB - DDDRP Assessment Prog Supp GenIC

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Diaper Distribution Demonstration and Research Pilot (DDDRP) Assessment

Formative Data Collections for Program Support

0970 - 0531

Supporting Statement

Part B

DECEMBER 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Erin Cannon

Shirley Adelstein

Part B

B1. Objectives

Study Objectives

The Administration for Children and Families (ACF) designed the Diaper Distribution Demonstration and Research Pilot (DDDRP) to test the feasibility and impact of federally funded expansions of existing diaper provision through community action agencies (CAAs) and their partners. In addition to diaper provision, ACF’s goal is to connect families with diaper need to existing wraparound services to help families secure additional supports, such as housing, access to early childhood education, and employment and training services. The objective of this study is to inform program administration and the design of a future impact evaluation of the DDDRP through two study components:

1. Process assessment to study grant recipient approaches, structures, activities, reach, and experience with ACF and technical assistance (TA) providers

2. Participant experience and outcome assessment to document caregiver characteristics, experiences as DDDRP participants, and changes in intended outcomes

This project will focus on three cohorts of seven grant recipients distributing diapers, covering 21 grant recipients in total. We will address the study objective using multiple methods. We will conduct in-person site visits to collect data through grant recipient and subrecipient interviews and participant focus groups (see Instrument 1, Grant recipient staff interview protocol; Instrument 2, Grant subrecipient staff interview protocol; and Instrument 3, DDDRP participant focus group protocol). We will also incorporate grant-collected data using a combination of administrative data (e.g., demographics and service delivery) and a previously approved survey (the Beneficiary Enrollment Survey [BES]; OMB control number 0970-0531¹).

Generalizability of Results

This study is intended to present an internally valid description of the implementation of the DDDRP program in chosen sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

The primary objective of this study is to inform program administration and the design of a future impact evaluation of the DDDRP. To meet these goals, we will collect data from grant recipient and subrecipient staff and program participants. The study uses a combination of secondary data, program administrative data, and primary data collection to evaluate the DDDRP implementation. This broad approach to data collection will be key in understanding how the program is implemented, what expected and unexpected program challenges and facilitators the program encounters, and whether the program achieves its intended short-term outcomes. We designed the study to also provide preliminary data to inform a potential impact analysis.

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. Project data are not intended to be representative and are being collected to assess the implementation of the DDDRP. We are analyzing some participant outcome data as a formative, initial assessment to understand what data a future impact study would need to collect and what the challenges of that collection could be. Given the variation in data accuracy, quality, and thoroughness across the DDDRP grant recipients’ data collection activities, we characterize these analyses as exploratory. The findings will not be considered representative of all DDDRP participants. The study team will explain these limitations and any other limitations of the data collection procedures (e.g., explain if one site could not identify enough participants for the focus group) in all written products associated with the study.

B2. Methods and Design

Target Population

ACF provided grants to 21 community agencies and diaper banks across three cohorts to expand diaper distribution nationally (see table 1 for the locations of grant recipients). For each of the 21 grant recipients, the study team will collect grant-collected information. We will also interview grant recipient and subrecipient staff and conduct focus groups with DDDRP program participants.

Table 1. List of DDDRP Grant Recipients

Cohort Number	Grant Recipient
Cohort 1	Arizona Community Action Association (Wildfire)
	Iowa Community Action Association
	North Carolina Community Action Association
	South Carolina Association of Community Action Partnerships
	South Puget Intertribal Planning Agency
	Virginia Community Action Partnership
	Washington State Community Action Partnership
Cohort 2	California Community Action Partnership Association
	Community Action Association of Alabama
	Maryland Community Action Partnership
	Massachusetts Association of Community Action Programs
	Ohio Community Action Training Organization
	Sisseton-Wahpeton Oyate
	Utah Community Action Partnership of Utah
Cohort 3	Community Action Association of Pennsylvania
	Community Action Partnership of Oregon
	Connecticut Association for Community Action
	Maine Community Action Association
	New Hampshire Community Action Partnership (NHCAP)
	New York State Community Action Association
	Ponca Tribe of Nebraska

Sampling/Respondent Recruitment

The specific approach to respondent recruitment varies by study component, although all use nonprobability sampling methods. Because the project objective includes informing program administration, all grant recipients will be included in the data collection. We will identify a primary point of contact at each grant recipient who will identify other staff at the grant recipient and subrecipient organizations who are most familiar with the program logistics, challenges, and successes. Grant recipients will also recruit a convenience sample of program participants for focus groups. Sampling and recruitment for each data collection are described below.

Grant recipient staff. For each of the 21 grant recipients, we plan to interview the project director. We will also work with the grant project director to identify up to two other staff members most involved in the planning, execution, and/or tracking of grant activities. The staff members will be selected based on their ability to provide key data on the project implementation. This nonprobability purposive sampling approach means that the perspectives of the grant recipients will not represent all staff working at the agency.

Grant subrecipient staff. Up to eight subrecipients will be interviewed at each of the 21 grant recipients, for an estimated total of 168 subrecipient interviewees. The number and type of subrecipients vary by site. Every site includes at least one diaper bank as a subrecipient, and most also include local CAAs. Less common types of subrecipients include food banks and local community programs. In selecting the subrecipients for the interviews, study staff will interview at least one diaper bank partner and at least one CAA partner. In most cases, the study plans to interview at least two of the most involved subrecipients (one diaper bank and one CAA). Because the participants will be selected based on their knowledge of and involvement in the DDDRP, this purposive sampling approach will yield a sample of respondents that is not representative of all grant subrecipient staff.

Program participants/caregivers. DDDRP individual-level participant data will come from two sources across all grant recipients: grant-collected data and focus groups. Grant-collected data may vary by grant recipient, although all grant recipients were asked to field the BES. We expect grant-collected data to represent a census of all DDDRP participants, but the final data may be limited by the information previously tracked by the grant recipients. We will also request service delivery measures from grant recipients that are collecting these data.

We plan to conduct participant focus groups at each grant recipient location. We will ask each of the 21 grant recipients to recruit up to 12 current DDDRP participants who represent the range of characteristics observed in their service population (e.g., different gender identities; caregiver roles, such as parent or grandparent; racial and ethnic identities; length of diaper receipt). We plan to recruit 12 participants with the goal of having between eight and 10 participants in each focus group. The sample will be a convenience sample and will not be representative of all DDDRP participants; however, we will recruit with the intention of capturing a range of participant characteristics to obtain diversity of experiences.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Data collection instruments were designed in collaboration with ACF, the contractor (Westat), and its subcontractors, Dr. Jennifer Randles and Ms. Justine Wolitzer. The project’s technical working group and caregiver panel also reviewed the instruments. Items were informed by the prior evaluation of the Help a Mother Out diaper distribution program led by Ms. Wolitzer². We designed the instruments to meet the objectives of the study and minimize burden on participants. Where possible, we use data already collected as part of program operations to reduce the number of questions for participants.

The grant recipient and subrecipient interview guides (Instruments 1 and 2) were reviewed by a diaper distribution operations manager. The DDDRP participant focus group guide (Instrument 3) was pretested with members of the study’s caregiver panel. The caregiver panel includes parents and other caregivers who are helping with the overall study.

Through these efforts, the same questions were not asked of more than 9 individuals and the pretesting activities were not subject to the Paperwork Reduction Act.

Table 2 describes each data collection instrument, key sources used to develop the instrument, and the project component(s) related to each instrument. We have included all instruments as appendices.

Table 2. Data Collection Instruments

Data Collection Instrument	Sources Consulted	Project components covered
Instrument 1. Grant recipient staff interview protocol	Project staff; technical working group; diaper bank operations manager	Process assessment
Instrument 2. Grant subrecipient staff interview protocol	Project staff; technical working group; diaper bank operations manager	Process assessment
Instrument 3. DDDRP participant focus group protocol	Help A Mother Out focus group protocol; project staff; technical working croup; caregiver panel	Participant experience and outcome assessment
Instrument 4. Site visit scheduling template	Project staff	Process assessment Participant experience and outcome assessment
Instrument 5. DDDRP focus group participant recruitment tool	Project staff	Participant experience and outcome assessment
Instrument 6. DDDRP participant photo release and instructions	Project staff	Participant experience and outcome assessment

B4. Collection of Data and Quality Control

Interview and focus group data will be collected during in-person site visits made by ACF’s contractor, Westat, to each grant recipient. The data collection protocols vary by data collection component. For each component, we identify data collectors and describe the recruitment protocol and data collection mode. We will provide consistent and comprehensive training to all data collectors (see quality assurance procedures section below).

Grant recipient and subrecipient interviews. Grant recipient and subrecipient interviews will be conducted in person by study team staff from Westat and its subcontractors. Prior to the site visit, the study team staff will contact the project director by email or telephone to request an interview, identify grant subrecipient sites for focus groups and site visits, and identify a site visit point of contact. The study team will have a second call with the site visit point of contact and ask them to identify up to two additional staff members who could serve as key informants about DDDRP implementation and key informants at the subrecipient sites for the grant subrecipient interviews. For each grant recipient, we will attempt to interview a diaper bank affiliate, a local CAA, and one additional local service provision organization.

The study team will send informed consent forms to each potential interviewee prior to the site visit. If we receive consent, we will audiorecord the interviews. The interviewer will take detailed notes during any interview where the interviewee does not agree to be recorded. We expect the grant recipient interviews to last 60–90 minutes and the subrecipient interviews to last about 60 minutes.

Grant-collected de-identified individual-level participant data. All grant recipients and service-providing subrecipients are collecting individual-level demographic, service delivery, and preliminary outcome data from DDDRP participants. The approach varies based on grant cohort and individual grant recipient interest and capacity. Outcome data will be used to assess the feasibility of fielding an impact evaluation. The grant recipient data will come from administrative data and the BES (OMB control number 0970-0531), an online instrument OCS asks each DDDRP grant to administer to participants as they enroll in program services. To meet project timelines, some grant recipients administered the intake survey prior to the development of the BES. Other grant recipients administered the BES prior to being assigned a study identification number; grant recipients without these IDs would not be able to link baseline data to any follow-up administrations of the survey. Baseline and follow-up data collection will vary by cohort as follows:

• Cohort 1 grant recipients collected baseline data prior to the introduction of the BES. Cohort 1 will be asked to provide administrative data covering BES data elements at an individual level if possible. Grant recipients that cannot provide individual-level data will be asked to provide aggregate data. Grant recipients with high-quality baseline data will also be asked whether they are interested in collecting outcome data. Interested grant recipients will repeat the same instrument they used to collect baseline data.

• Cohort 2 grant recipients administered the BES at program enrollment. Grant recipients that enrolled participants early will not be asked to repeat the BES because participants were not assigned a unique study identification number. Cohort 2 grant recipients that enrolled participants later (i.e., after the study identification number was in use) will be asked to repeat the BES to collect outcome data.

• Cohort 3 grant recipients will administer the BES at baseline and repeat the survey to collect outcome data. The two BES administrations will be linked by the study unique identification number.

We will also instruct grant recipients to remove information about any minor caregivers from their datasets before sharing the data with Westat. We will retain a variable indicating participant age (either date of birth or age conversion). We will screenshare with grant recipients before they transmit data to view the data elements and confirm the set is clean. Grant recipients will then transmit their data via a secure file transfer. We will also ask grant recipients to provide data documentation, such as dictionaries or coding guidance, for all grant-collected data.

Caregiver focus groups. We will conduct in-person focus groups with current DDDRP participants. We will conduct at least two focus groups of eight to 10 participants per grant. We will provide participants with a $50 gift card for their participation. We will secure participant consent to participate in the focus groups. We will provide participants with consent language prior to the focus group, review the language at the beginning of the focus group, and ask for participants’ oral consent to participate in the focus group discussion (which will be recorded). If participants do not consent to be recorded, we will ask that they leave the focus group and not participate. We will ask each grant recipient to recruit current DDDRP participants who represent their service population (e.g., different gender identities; caregiver roles, such as parent or grandparent; racial and ethnic identities; length of diaper receipt). We will provide each grant recipient with a template flyer to use for participant recruitment (see Attachment A. Template flyer for focus group recruitment). Focus group participants will also be invited to submit photos representing their experiences with diaper need and the DDDRP.

Quality assurance procedures. The study leadership will provide training to staff on all data collection procedures to ensure consistent, high-quality procedures across data collectors. The trainings will include existing materials and start with an overview of the study and its purpose. The training will describe each data collection instrument, its purpose, and procedures for administering the tool. We will role-play using the instruments. The training will highlight participant privacy, data security, and staff and participant safety and include resources to support participants who identify any challenges during data collection (e.g., 988 Suicide and Crisis Lifeline). We will also explain staff responsibilities as mandatory reporters of suspected child abuse. We will describe best practices for site visits (e.g., printing maps in advance, planning extra time for transportation) and strategies to connect with and show respect to study participants. Lastly, the training will include cultural humility training to help data collectors understand how to approach engaging in cultures and communities of which they are not members. Once in the field, the data collectors will have regular check-ins to share challenges and brainstorm solutions. The study lead will review site visitor notes, noting issues with quality to review and train on during check-in meetings.

To ensure high-quality administrative data, the study team will provide data TA to all grant recipients around data collection. We will tailor TA to the specific needs of each grant recipient, assessed through discussions with the grant recipients.

All project procedures have been reviewed by the Westat institutional review board (IRB). We will also have these documents approved by any other site-specific IRBs or similar ethics reviewers as required by each grant recipient’s jurisdiction. We will use only approved protocols.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

The interviews and focus groups are not designed to produce statistically generalizable findings, and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.

We will calculate missing data rates for program and participant data.

Nonresponse

Participants in the interviews and focus groups will not be randomly sampled, and findings are not intended to be representative. We do not plan to calculate nonresponse bias for these study components. Respondent demographics will be documented and reported in written materials associated with the data collection. We will also track all invited participants to evaluate whether there are trends in who does not consent to participate.

B6. Production of Estimates and Projections

We will use the data to document how the program is implemented, program challenges and facilitators, and whether the program achieves its intended short-term outcomes. This formative information and preliminary lessons learned may inform the DDDRP-funded programs and other diaper distribution programs efforts. Descriptive information will be used internally and also made publicly available in a report, project briefs, presentations, and other communications. The data will not be used to generate population estimates for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

For quantitative data (administrative data and outcome assessments), we will conduct basic quality assurance checks and procedures for identifying incomplete or missing data after receiving the data from each grant recipient. We will document missing data and delete or flag variables that do not make sense (e.g., someone working 80 hours while unemployed). For qualitative data, we will review each transcript to ensure the full content of the interview or focus group was recorded.

Data Analysis

For the qualitative data collected through interviews and focus groups, we will use NVivo to organize the data and thematically code interview transcripts and document abstractions. Where possible, we will use the same sets of codes across programs to facilitate cross-case analyses. We also will use data from participants, grant recipients, and grant recipient documents to develop client journey maps that function as flowcharts, documenting the steps and activities DDDRP participants experience at each grant recipient. Journey maps include steps such as eligibility assessment and criteria that made families ineligible. This mapping helps describe how caregivers experience the program and identifies opportunities to include random assignment procedures.

For the quantitative analysis using administrative data and outcomes assessments, we will compute basic descriptive analyses (frequencies, means, distributional analyses) and prespecified subgroup analyses (such as comparisons of dosage by number of children in a caregiver family). To inform the design of a future impact evaluation, we will examine which participant-level outcomes are most likely to show change and at what point after DDDRP enrollment they should be measured by calculating change scores for each outcome at each measurement level across grant recipients. We will include regressions, t-tests, and chi-square analyses.

Data Use

The primary intended uses of the data are to—

1. Understand how DDDRP is implemented within and across grant recipients distributing diapers.

2. Better understand diaper distribution operations.

3. Establish the foundation for rigorously measuring the impact of diaper distribution programs.

4. In conjunction with other research and evaluation efforts, inform future program administration.

The data will be used for internal and public-facing reports (e.g., on websites or social media, in journals), briefs, and presentations about the DDDRP implementation. Presentations may include internal presentations, presentations to other government organizations or departments, and professional conferences. Public-facing reports will be shared with the goal of helping other programs understand how the DDDRP was implemented and identify potential challenges. These reports will also share other lessons learned, including potential recommendations for outcome measures based on the exploratory outcomes assessments (e.g., which data could be collected, how were outcomes measured). The data collection will enable ACF to understand how the DDDRP was implemented and identify areas for improvement. The data will also be used by ACF to inform the design of an impact evaluation focused on participant outcomes.

The analytic data will not be released to the public.

B8. Contact Persons

Table 3 summarizes individuals who can answer questions about the statistical aspects of the survey and who will collect, process, and/or analyze the information for the agency. The protocol and instruments were developed and reviewed extensively by the ACF and Westat study team. Westat will also be responsible for the collection and analysis of the study’s data, in coordination with ACF.

Table 3. Individuals Involved in Data Collection and Analysis

Staff	Title	Contact Information (phone or email)
Allison Hyra	Project Director	allisonhyra@westat.com
Hilary Wagner	Deputy Project Director	hilarywagner@westat.com
Debra Rog	Co-Principal Investigator	debrarog@westat.com
ACF staff
Erin Cannon	Contracting Officer’s Representative	Erin.Cannon@acf.hhs.gov
Shirley Adelstein	Contracting Officer’s Representative	Shirley.Adelstein@acf.hhs.gov
Consultants
Justine Wolitzer	Co-Principal Investigator	justine@publicprofit.net
Jennifer Randles	Senior Advisor	jrandles@csufresno.edu

Attachments

Instrument 1. Grant recipient staff interview protocol 

Instrument 2. Grant subrecipient staff interview protocol 

Instrument 3. DDDRP participant focus group protocol 

Instrument 4. Site visit scheduling template 

Instrument 5. DDDRP focus group participant recruitment tool 

Instrument 6. DDDRP participant follow-up data guidance–subset of cohort 1

Instrument 7. DDDRP participant photo release and instructions

Attachment A. Template flyer for focus group recruitment

1 GenIC title: Diaper Distribution Demonstration and Research Pilot Baseline Data Collection

2 Public Profit. (December 2021). Help a Mother Out’s San Francisco Diaper Bank Final Evaluation Report.

11