Instrument 1: Focus groups with state leaders and others
Registration questions
When registering for a focus group within the Zoom.gov platform, respondents will be asked to provide the following information:
Name
Email address
In what state do you currently live?
What is the name of organization where you currently work?
In your current job, how would you best describe your role?
SELECT ONE ONLY
Executive or administrator
Manager
Direct service staff
Information technology (IT) staff
Researcher or data scientist
Other (please describe)
Focus group introduction
Good morning/afternoon and thank you for joining us.
We are from an organization called Mathematica and we will be guiding the conversation today. The purpose of this focus group is to get your perspectives on how public benefit programs may consider using artificial intelligence, or AI.
I’ll explain a little more about what we mean by AI in a minute. We are hosting this session on behalf of the Administration for Children and Families. The Administration for Children and Families, or ACF, is a federal agency that administers a number of public benefit and other programs, including Temporary Assistance for Needy Families, Head Start and Early Head Start, teen pregnancy prevention programs, and many others.
The Administration for Children and Families is interested in hearing the perspectives of organizations and people who could use AI to administer and operate public programs. This is one of several focus groups and listening sessions being conducted with different groups. ACF will compile all the feedback received from the sessions to help it prioritize issuing guidance on AI use. ACF might also use the feedback to develop resources to support responsible AI use among the organizations it funds.
There are no right or wrong answers to the questions we’ll ask today, ACF is really interested in hearing your opinion, whatever it is. Information collected will be kept private to the extent permitted by law. Participation is voluntary and you can participate in this session to the extent you feel comfortable, you do not need to answer any questions you don’t want to. There are no risks associated with participating in this session.
I’m going to read a statement for your awareness: An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number and expiration date for this collection are OMB #: ADD, Expiration: ADD.
We would like to record this conversation to ensure we capture the key points. We will also transcribe it, take notes of the discussions happening in breakout rooms, and share with people at ACF. In the transcript and notes, we will remove names and any other personal identifiers to protect your privacy.
Format of the session
We'll start with an overview of what we mean by AI and a brief “pulse check” using audience polls, then move into discussing some scenarios of how AI might be applied in program operations and service delivery. We'll first discuss these scenarios in small breakout groups then come back together for a full group discussion. The breakout rooms will be facilitated by myself and my Mathematica colleagues, X, X, X, and X.
A. Background
First let’s set the stage so we all know what we mean when we say “AI.” In broad terms, AI is technology that enables computers and machines to simulate human intelligence and problem-solving abilities. When we say “AI,” we aren’t referring to far-fetched future technologies you might see in movies, but rather technology that currently exists and is accessible to most people. There are lots of different kinds of AI, and you have probably interacted with some of them.
Examples are talking to digital assistants, like Alexa, Siri, or “Hey Google.” You ask them a question and they return an answer to you. Other examples include personalized recommendations and content offered up by Netflix, Instagram, YouTube, or Amazon.
For our conversation today, we want to think about those types of activities in the context of public benefits and services.
B. Pulse check
Before we move into breakout rooms, we’d like to get a sense of where everyone is in terms of knowledge and perceptions of AI in public benefits programs. We will use Zoom’s polling feature to conduct a quick pulse check.
Knowledge Level
How would you rate your knowledge about current AI applications in public benefit programs?
I know a lot
I know a little
I do not know anything
Current or Planned AI Use
How would you describe the current or planned use of AI in your program/agency?
Currently using AI
Actively planning AI implementation
Discussing AI, no specific plans yet
Not using or planning to use AI
Possibilities for AI Use
What, if any, are ways you think AI could be used in public benefits programs?
[Select all you think are appropriate.]
Improve efficiency and reduce administrative burden
Enhance decision-making and service delivery
Provide more personalized services to clients
Improve accuracy and reduce errors
Other uses not listed above [fill text]
No uses
Don’t know
Concerns about AI Use
What is your greatest concern about using AI in public benefits programs?
Data privacy and security
Potential bias and fairness issues
Loss of personal interaction with clients
Too much reliance on technology
Risk of technical failures
Cost to continue operating, monitoring, and optimizing AI systems
Other concerns not listed above [fill text]
No concerns
Don’t know
Thank you. That was a very helpful starting point for us. Does anyone have any questions before we move into breakout rooms? Okay, in a moment you will be invited to join your breakout room. You will see a pop-up inviting you to join a breakout room. Click “ok” and it will take you to the room. We will see you all back here in the main virtual room in about 40 minutes.
C. Breakout groups
[Transition to breakout group facilitator]
Let’s start with some short introductions - Name, location, role/title, organization
(1 minute for each attendee to introduce themselves; approx. 5 per breakout group)
Overview of breakout discussion
I’m going to present a few scenarios and I’d love to hear your thoughts about how you might respond. There are no right or wrong answers here—we are just interested in what you have to say. For each scenario, take a moment to consider it. When you are ready to share, raise your hand in Zoom or, if you prefer, type your thoughts into the chat.
Breakout group type A: Internal operations
Scenario 1: Administrative assistance (priority)
Your organization introduces a set of AI tools to help staff with routine administrative tasks to process benefits. The AI tools would handle things like identifying duplicate documents submitted by clients, verifying signatures across forms, and automatically filing approved documents in the appropriate electronic case folders.
What are some potential advantages to having AI support these types of administrative tasks?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 2: AI-assisted pre-hiring screening (priority)
An AI system is available to assist with pre-hiring screening of staff in a public benefits program. The AI would analyze applicants' resumes, cover letters, and other application materials to identify top candidates for interviews.
What potential benefits do you see in using AI for pre-hiring screening?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 3: AI-assisted performance evaluation (priority)
An AI system is available to assist with performance evaluations of staff in a public benefits program. For current employees, the AI would analyze data such as work outputs, communication patterns, and feedback from colleagues to provide performance evaluation insights to managers.
What potential benefits do you see in using AI for performance evaluations?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 4: AI-assisted biometric sign-in to systems
Your agency is exploring the implementation of an AI-assisted biometric sign-in system, such as facial recognition or fingerprint scanning, for access to computer systems and physical facilities. The AI would be involved in the development, validation, and ongoing monitoring of the biometric sign-in process, analyzing the collected biometric data to ensure accuracy and security. This system would replace the current system of usernames, passwords, and key cards.
What potential benefits do you see in using AI-assisted biometric sign-in?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Breakout group type B: Benefits administration
Scenario 5: Application processing (priority)
To determine eligibility for public benefits, applicants complete an online application and upload required documents. AI is used to automatically determine eligibility for most applicants based on the submitted information. Cases are auto enrolled if eligible, with a note that eligibility may be re-reviewed by a human worker.
What are some advantages of using AI to process and assess applications?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 6: Case management (priority)
When a client case is transferred to a new caseworker in your organization, the worker receives both the full detailed case notes and an AI-generated summary highlighting key points from the prior notes.
What are the potential advantages of AI-assisted case note summaries?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 7: AI-assisted risk assessment (priority)
Your agency is considering using an AI program to scan all case files of clients receiving public benefits. The AI would be trained to identify and flag cases with potential risks related to mental health issues or child abuse based on patterns in the data. Based on its analysis, it can identify potential changes to the client's benefit levels and recommend additional services that may be beneficial for the client's unique circumstances and needs.
In what ways could this AI risk assessment potentially improve service delivery and client outcomes?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 8: AI-optimized shelter bed assignments
A shelter for people experiencing homelessness is considering using an AI system to manage room and bed assignments. The AI would analyze client data, including demographics, health conditions, and case history, to determine the most suitable room and bed assignments while considering factors such as client compatibility and special needs.
In what ways could AI-optimized bed assignments potentially improve client outcomes and shelter operations?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Breakout group type C: Customer service
Scenario 9: Translation services (priority)
Your agency is considering implementing an AI-powered language translation tool to facilitate communication between staff and clients who have limited English language skills. The tool would allow for near real-time translation during in-person or virtual meetings and cover the range of primary languages spoken by your clients.
How might this positively impact service delivery and client engagement compared to current processes?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 10: Emergency hotline assistance (priority)
Your organization operates a crisis hotline. For hotline calls where the caller is seeking immediate help for a dangerous situation, AI is used to analyze the call in real time and provide recommendations to the hotline staff on potential next steps. This includes suggesting relevant community resources for the hotline staff to share with caller as well as a recommendation on whether the situation may warrant dispatching first responders based on the AI's risk analysis.
What potential advantages could AI-generated recommendations provide in this type of high-stakes situation?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 11: Service center/help desk management (priority)
Your organization uses a type of AI called natural language processing models to perform routine customer service interactions. This could include responding to emails that do not require action on the part of a human, such as simple "thank you" emails, or sharing standard responses to frequently asked questions. It could also include an interactive phone tree for routing callers to the appropriate staff.
What potential advantages could AI-generated customer service interactions provide in this scenario?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Scenario 12: AI job skills coach
You manage an employment services program. To supplement the existing job skills training program and coaching sessions, clients are able to use an AI-powered virtual job skills coach in addition to their sessions with a human coach. This AI coach would be available 24/7 for additional practice and support. For example, the AI coach could simulate job interviews by asking common questions and providing feedback on the client's responses, similar to the human coach.
What are some advantages you could see to offering an AI coach?
What concerns, if any, would you have about this use of AI?
What kinds of guidance and resources from ACF would help your team feel more confident in assessing and/or using AI for this purpose?
Prompts: Clear do's and don'ts tailored to specific types of ACF-funded programs and services; clarification of oversight from and reporting requirements to ACF; procurement resources such as proposal evaluation checklists and lists of vendors offering related products; technical assistance to design, develop, and implement pilots; community of practice to connect with peers working on similar AI projects; real-world case studies
Breakout group discussion wrap-up
As we head back to the large group, take a moment to reflect on some of the most promising opportunities or important concerns raised, along with any key points that stood out to you. We'll have a chance to share some of these reflections with the full group.
D. Full group discussion
[Transition to full group facilitator]
Welcome back, everyone! I hope you had engaging conversations in your breakout groups. In those small groups, you dove into several scenarios of how AI might be used in public benefits programs.
We have time now to debrief and hear highlights from those discussions. I'd like to open it up for some reflections on what stood out to you. Note that not all of your groups discussed the same scenarios. What were some of the most promising opportunities or important concerns that emerged in your conversations? [Small group facilitators will be prepared to share out if participants don’t volunteer]
Promising opportunities
Would anyone like to share an AI application that your group found particularly promising or intriguing?
What potential benefits did you see?
Potential concerns
On the other hand, what applications did you find most concerning?
What were some of the key concerns or risks that came up in your conversations?
Considerations and safeguards
Did any important considerations emerge around how to ensure AI tools are designed and used responsibly?
What sorts of safeguards, oversight, or human involvement did you discuss?
Communications and support
What kinds of communication and staff/client support would be needed to introduce the kinds of AI tools you discussed?
E. Closing questions
We’ve given you some examples of how AI might be incorporated into public benefit programs.
Are there other ways AI could be applied in public benefit programs that we haven't discussed? Feel free to say them out loud or type into the chat.
What situations related to public benefits and services do you really not want AI to have any part in, compared with working exclusively with people?
Are there any types of resources/information on the use of AI within public benefits programs that you think would be useful or would like to see developed?
F. Thank you
Thank you all for sharing your perspectives and insights. The issues and ideas you've raised will be immensely valuable as ACF continues to explore the responsible use of AI in public benefits programs. This is an ongoing conversation, and your input will help shape ACF’s guidance on AI use and future resources to support responsible AI use among the organizations it funds.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Williams, Joshua (ACF) |
| File Modified | 0000-00-00 |
| File Created | 2025-02-17 |