Constituent Engagement Related to Use of AI
Administration for Children and Families Generic for Engagement Efforts
0970 – 0630
Supporting Statement Part B
July 2024
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Overview of Study Objectives
The Administration for Children and Families (ACF) Office of Planning, Research, and Evaluation proposes to collect information to better understand human services field constituents’ wants and needs related to the use of artificial intelligence (AI), particularly relating to the administration of public benefits and services.
The primary purpose of this information collection is to obtain rapid ACF constituent input and feedback on prioritization efforts related to:
ACF’s position and response to HHS-level mandates on AI, including the following requirements from EO 14110:
Identifying and prioritizing what guidance and support the human services field wants and needs related to use of AI, particularly relating to administration of public benefits and services per E.O 14110 Section 7.2(b)(i);
Providing human service perspectives to an HHS strategic plan on AI, per E.O 14110 Section 8 (b)(i);
Informing ACF's AI strategy and ongoing constituent engagement channels.
The gathered information will be used to develop an internal memo synthesizing findings from constituent engagement. This memo will be used to inform ACF’s AI strategy and determine on which topics to issue guidance on the use of AI to ACF grantees.
The three proposed instruments that will include minimal statistical analyses include the following:
Instrument 3: Survey of AI opinions—participants
Instrument 4: Survey of AI opinions—nonparticipants
Instrument 5: Screener for participants
As such, this document primarily focuses on these three instruments with general descriptive information provided for the other two instruments, which are protocols to support focus groups.
Respondent Universe and Sampling Methods
Target Population
For focus groups using Instrument 1, the target population is state leaders, local leaders, and frontline service providers from a range of human services programs
For focus groups using Instrument 2, the target population is participants in a diverse set of public benefit programs
For the Survey of AI opinions—participants (Instrument 3) the target population is participants in a diverse set of public benefit programs.
For the Survey of AI opinions—nonparticipants (Instrument 4) the target population is state leaders, local leaders, and frontline service providers from a range of human services programs.
The Screener for participants (Instrument 5) is for potential public benefit program participants who might meet the eligibility criteria to participate in focus groups using instrument 2 or the Survey of AI opinions—participants (Instrument 3). A recruiting panel will use this screener to identify eligible program participants, who will then be invited to participate in the focus group and/or take the survey.
Sampling and Site Selection
The contractor will conduct focus groups (Instruments 1 and 2), surveys (Instruments 3 and 4), and screening (Instrument 5) with purposive samples of the target population. The Contractor will attempt to recruit a respondent group that has a diversity of racial/ethnic backgrounds, geographic locations (primarily rural/urban), and roles within public benefits programs. However, ACF and its contractor do not intend to use a formal sampling method to achieve this.
Appropriateness of Study Design and Methods for Planned Uses
The data collected will be used to inform ACF’s AI strategy and determine on which topics to issue guidance on the use of AI to ACF grantees. The information is purely descriptive in nature; the data are not intended to be representative of the opinions of all public benefits programs or their enrollees. These limitations will be included in any written products resulting from this data collection.
This information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
Procedures for Collection of Information
Data Collection Processes
ACF and its contractor will field one online survey of AI opinions for participants (Instrument 3) and one for nonparticipants (Instrument 4) using Qualtrics. ACF hopes to receive approximately 400 complete surveys consisting of approximately 300 from state leaders and other program non-participants, and 100 from program participants.
ACF’s contractor will administer one online screener to potential public benefit program participants to determine whether they are eligible to participate in a focus group (Instrument 2) and/or complete a Survey of AI opinions (Instrument 3). The contractor anticipates administering the screener to 280 people.
Focus group data (instruments 1 and 2) will be collected using the Zoom meeting platform (instrument 1) and meeting technology provided by SAGO (instrument 2). Recordings from focus group meetings will be stored securely on the contractor’s website server, using federally approved data encryption. Transcripts developed from recordings will remove any names or other personally identifying information to protect the privacy of respondents. Access to recordings and transcripts from focus group meetings will be restricted to authorized Contractor study team members. Transcripts will be shared with ACF using secure platforms for data transfer.
Survey data (instruments 3 and 4) will be collected using the Qualtrics platform, which will be hosted on the ACF website. Survey respondents will complete the survey online and all data will be securely stored on ACF’s website server, using federally approved data encryption. Access to the survey data will be restricted to approved staff at ACF and authorized Contractor study team members who will monitor data collection. No personally identifiable information will be collected on the survey.
The screener (instrument 5) will be conducted via a web survey hosted on the recruiting panel SAGO’s secure HTTP site Methodify. Sago is ISO 27001 certified, and as such, their operating systems and applications comply with high international and industry standards and best practices for data security and protection. Their systems are regularly updated with the latest stable releases and security patches, service packs, and hot fixes as they are made available by the vendors. No personally identifiable information will be collected for the screener.
SAGO complies with the EU-U.S. Data Privacy Framework (EU-U.S. DPF) and the UK Extension to the EU-U.S. DPF as set forth by the U.S. Department of Commerce. In the United States, SAGO adheres to all relevant federal and state privacy laws, including the California Consumer Privacy Act (CCPA) and the amended California Privacy Rights Act (CPRA). SAGO uses physical, electronic, and procedural safeguards when collecting and storing data, encrypting transmissions using secure socket layer (SSL) technology.
Data Analysis
The contractor team will use thematic analysis for qualitative data and descriptive statistics to tabulate survey responses. No statistical tests are planned.
Use of Information Collected
ACF’s contractor will synthesize the information from the virtual focus groups it conducts and the survey responses into a memo for internal ACF use. At a later time, ACF may use information from the focus groups and survey to develop publicly available guidance for grantees.
Methods to Maximize Response Rates and Deal with Nonresponse
Response Rates
The survey is not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented to the extent feasible and reported in written materials associated with the data collection.
Test of Procedures or Methods to be Taken
Development of Data Collection Instruments
Because the use of AI in human services programs is an area of nascent study, ACF developed the data collection instruments in consultation with its contractor. Efforts were made to reduce respondent burden by limiting focus groups to only 60 minutes and limiting the survey to 5 minutes.
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Annalisa Mastri (amastri@mathematica-mpr.com) can answer questions about the statistical aspects of the information collection. In addition, the following people will collect, process, and/or analyze the resulting information:
Caitlin Lowery (ACF, OPRE)
Lauren Griffin (ACF, OPRE)
Nicole Deterding (ACF, OPRE)
Joshua Williams (ACF, OPRE)
Jane Yang (ACF, OCIO)
Heather Zaveri (contractor, Mathematica)
Megan Shoji (contractor, Mathematica)
Samantha Vance (contractor, Mathematica)
Katie Hunter (contractor, Mathematica)
Tiffany Waits (contractor, Mathematica)
Instrument 1: Focus groups with state leaders and others
Instrument 2: Focus groups with public benefit participants
Instrument 3: Survey of AI opinions—participants
Instrument 4: Survey of AI opinions—nonparticipants
Instrument 5: Screener for participants
Appendix 1: Outreach materials
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Annalisa Mastri |
| File Modified | 0000-00-00 |
| File Created | 2025-02-16 |