Temporary Assistance for Needy Families (TANF) Data Innovation Project – Formative Data Collection

Attachment 1

TANF Data Innovation (TDI): Stakeholder Interview Protocols as of May 2018

This instrument lists questions that will be asked in interviews of key stakeholders with interests in seeing state TANF data used effectively to help improve program administration, payment integrity, and outcomes for participants. We divide stakeholders into five groups: 1) federal leaders and organizations focused on TANF, 2) federal leaders and organizations focused on other family self-sufficiency programs (such as SNAP and WIOA), 3) state health and human services agency leaders; 4) researchers, and 5) vendors. The questions are divided into the following sections: 1) background, 2) data use and opportunities, 3) research and analytic capacity, 4) federal reporting, 5) data quality, 6) technology and data system, 7) data sharing and governance, and 8) closing.

To minimize burden and optimize the utility of the responses we receive, the interview protocol has been customized for each of the stakeholder groups. No stakeholder will be asked every question. Figure 1 summarizes the questions to be asked to each stakeholder group. For example, there are several different versions of data use questions tailored to each stakeholder group (e.g. Version A is for national TANF organizations, Version B is for other family self-sufficiency organizations, Version C is for state HHS/TANF leaders and selected staff, and Version D is for vendors). Additionally, given that the exact topics we cover will depend upon the experience of the interviewee, not all questions slated for each stakeholder group will be covered in any given interview.

The format for most of the data collection will be in-depth interviews lasting an hour or less. The exception will be stakeholders in the state TANF staff group. In order to minimize burden, we may conduct these interviews in small groups, most likely during a national TANF leaders convening.

Figure 1: Questions for each stakeholder group

TOPICS		STAKEHOLDERS
(number of questions)		Federal TANF Leaders and Organizations	Federal Self-sufficiency Leaders and Organizations	State HHS/TANF Leaders & Staff	Researchers	Vendors
Background	(4)	Q1-Q3	Q1-Q3	Q1-Q3	All	Q1-Q3
Data Use	(varies)	Version A (9)	Version B (5)	Version C (8)	None	Version D (9)
Research Capacity	(3)	Q1, Q3	None	Q1, Q3	Q2	All
Federal Reporting	(2)	All	None	All	None	None
Data Quality	(2)	All	None	All	All	All
Technology	(2)	All	None	All	None	All
Data Sharing	(4)	All	Q2, Q3	All	Q2, Q3	Q2, Q3, Q4
Closing	(4)	All	All	All	All	All
Total Questions		26	12	26	13	24

PROTOCOLS

Introduction

Thank you for taking the time to talk with us. I am [name] from [organization conducting interview]. As mentioned in the background materials that we emailed/sent to you, we are part of a team implementing a new project supported by ACF/HHS called TANF Data Innovation (TDI). Through TDI, we will design and implement a new training and technical assistance (TTA) approach to support state TANF agencies as they use data for decision-making that improves the lives of low-income families. Partners in implementing this TTA program include MDRC, Actionable Intelligence for Social Policy (AISP) at the University of Pennsylvania, the New York University Center for Urban Science and Progress, and Chapin Hall at the University of Chicago.

As part of our initial assessment, we plan to survey TANF staff about their current practices, priorities, challenges, and perceived areas of need. We also are having discussions with external stakeholders (like you) to document the broader context shaping how TANF and other human services data are used in program administration. The topics we will cover with you today will help us better understand that context. Your insights will also help ensure that TDI’s Technical Assistance will make a meaningful contribution in both content and form.

Participation in the interview is completely voluntary and will require a time commitment of approximately one hour. [If audio-recording:] We will ask to audio record our conversations for our data to accurately reflect what is said. Audio-recording is voluntary; you may request that we stop recording the conversation at any time. Audio files may be transcribed. If they are, they will be stored on a secure server and then deleted from the recorder. Data files will not be emailed as attachments. The information you provide is private. Your name and position will not be connected to responses in any written materials without your express permission.

According to the Paperwork Reduction Act, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0356 and the expiration date is 06/30/2021.

Do you have any questions before we begin?

Background

1. Please start by stating your name, current role, the organization, agency or firm you work for, how long you have held this position, as well as any relevant previous roles you have held.

2. Is TANF your main area of focus, or just one of the human services programs with which you are involved? 

3. In what ways do you interact with TANF program and public assistance data (e.g., conducting research, working with TANF data files, reviewing reports, using data to make policy decisions)?

   • Are you directly responsible for working with data or do you manage people who work with data? Or would you describe your role another way?

   • Is your primary focus: TANF recipients/leavers, other economically vulnerable populations, or both?

   • Is your primary focus: adults in TANF households, children in TANF households, or both?

   • Can you provide some examples of how you use TANF data?

4. [For Researchers only]: Have you ever worked with individual-level TANF data? With TANF data integrated with another source? Where did you get the data? How did you get the data (i.e. what were the relationships, etc. that made it happen) and how was that experience?

Data Use and Opportunities

Version A: Federal Leaders and Organizations Focused on TANF

1. From your perspective, what are the key current policy or programmatic challenges facing the TANF program? Facing TANF applicants/recipients? [Prompt if needed: For example, prior research has shown that, as the labor market improves, individuals and families who apply for or remain on public assistance tend to have more severe barriers to employment or more complex service needs. Coordinating assistance under these circumstances could become a major challenge, requiring more intensive and extensive use of data.]

2. Data on TANF recipients can serve different purposes:

• Basic administrative functions like determining eligibility for assistance and calculating benefits and Emergency Assistance.

• Case management, coordinating assistance among different benefit programs (e.g. SNAP, Medicaid, SSI) or with agencies or organizations that provide employment and training services or social and legal services (including child support).

• Performance measurement and management, including understanding the workload of eligibility workers or employment specialists or planning for additional or fewer human resources.

• Meeting federal reporting requirements.

• Data analysis to determine which individuals and families are at the greatest risk or greatest need for interventions, to examine whether the caseload is growing or shrinking over time, to examine if the caseloads’ characteristics are changing, or to explore trends in benefit receipt, employment, and program evaluation.

If you were advising states on how to invest their resources to improve use of data, which area(s) would you prioritize? What areas would you add? Why?

2a. Do you believe agencies would benefit from technical assistance and training on using TANF data for any of the purposes I mentioned above? Again, those areas are:

• Basic administrative functions □ Yes □ No

• Coordinating assistance □ Yes □ No

• Performance measurement/management □ Yes □ No

• Meeting federal reporting requirements □ Yes □ No

• Data analysis □ Yes □ No

• Another purpose (please specify)? □ Yes □ No

3. Sometimes investments in technology and training pay off through more efficient use of resources or better coordination of services. Do you know of any state-level examples of these investments? Can you envision ways in which greater efficiencies or better coordination could be achieved through better use of data or improved training? Do you know of any examples that highlights risks and pitfalls that we should watch out for?

4. What states do you perceive as strong users of TANF data for program improvement, analyses of program effectiveness, or payment integrity? In your opinion, which states need extra assistance? Please explain.

5. What states do you perceive as strong users of multi-agency data? Weak users? Please explain. [Prompt with examples as needed: North Carolina, South Carolina, Michigan, Washington, Mississippi, etc.]

6. Are you familiar with any past or current efforts across TANF agencies to invest in the agencies’ capacities to effectively use data? If so, where?

7. TDI is intended to provide several years of training, technical assistance, and technical consultation to state and local TANF agencies to improve their use of data and build evidence to help low-income families. What types of technical assistance would TANF agency data staff find most helpful to improve data use and data analysis to improve the quality of programs? Are there specific topics we should support? [Prompt as needed: such as caseload dynamics, predictive analytics to identify those most at risk, integrating TANF with other relevant data?] Are there specific ways to deliver technical assistance that you find particularly helpful? [Prompt as needed: in-person vs. virtual; webinars; written tool kits or other documents like infographics; videos; hands-on support; working on specific projects with one-on-one technical consultation?]

8. Do you know of organizations or networks that you feel are providing strong training and/or technical assistance in these areas?

Version B: Federal Leaders and Organizations Focused on Family Self-Sufficiency

1. What are the key areas where your programs interact or link with TANF in terms of uses of data and integration, research, and program performance?
   
   

2. Can you give some examples of data integration? (Prompt to get examples related to eligibility/enrollment AND analytic use. Where appropriate, get references or ask if they will send some examples.)

3. What are the priority questions that your organization/agency is trying to answer about TANF recipients? These might include questions about how to improve the effectiveness of services, how to promote payment integrity, or other policy priorities. And how would your organization/agency benefit from data integration and analysis to answer these questions?

4. TDI is intended to provide several years of training, technical assistance, and technical consultation to state and local TANF agencies to improve their use of data and build evidence to help low-income families. What types of technical assistance would TANF data agency staff find most helpful to improve data use and data analysis to improve the quality of programs? Are there specific topics we should support? [Prompt as needed: such as caseload dynamics, predictive analytics to identify those most at risk, integrating TANF with other relevant data?] Are there specific ways to deliver technical assistance that you find particularly helpful? [Prompt as needed: in-person vs. virtual; webinars; written tool kits or other documents like infographics; videos; hands-on support; working on specific projects with one-on-one technical consultation?]

5. Do you know of organizations or networks that you feel are providing strong training and/or technical assistance in these areas?

Version C: State/Local TANF Administrators and Staff

1. To what extent are your TANF programs and/or data systems integrated with other state or local agencies and organizations, including WIOA, SNAP, housing, criminal justice, UI earnings, IRS, etc.? What are the challenges? What plans do you have for future integration of data? What are the main purposes served by the current or planned data integration? [Prompt to categorize each example by purpose: Enrollment/Eligibility verification? Case Management? System management (resource allocation; performance reporting; research & evaluation)]

2. What types of “research” or “program” reports do you get/produce? For example, what topics do they cover? How often do you get them? Who reviews them? (Note: Do you have copies or references you are willing to share with us?) What types of reports or information would you like to have in the future?

3. Do you work with universities, other research organizations, or vendors to produce program performance reports or do research on these programs? Could you provide us with contact names? (Note to interviewer: This is the place to capture deep partnerships with university-based research centers like IRP in Wisconsin.) Do you do this on a regular basis? What are some of the challenges? Do you have data sharing agreements with these other organizations? Are you willing to share TANF data with other agencies or researchers? What are the challenges?

4. What firms, vendors, or in-house staff develop and maintain your data systems (e.g., private firms like Deloitte, Accenture, Maximus)? What areas of your data systems or program performance do they support? What are the challenges?

5. What is your staffing capacity to support data analysis for program improvement and for providing evidence on improving low-income families’ lives? What are the challenges? If you had more resources, what would you most like to learn more about (e.g., your caseload, predicting risk, outcomes of programs recipients, participation, how to target services)?

6. TDI is a federal investment intended to provide several years of training, technical assistance, and technical consultation to state and local TANF agencies. What types of technical assistance do you believe would best support TANF data agency staff to better use their data? Are there specific topics we should support? [Prompt as needed: such as caseload dynamics, predictive analytics to identify those most at risk, integrating TANF with other relevant data?] Are there specific ways to deliver technical assistance that you find particularly helpful? [Prompt as needed: in-person vs. virtual; webinars; written tool kits or other documents like infographics; videos; hands-on support; working on specific projects with one-on-one technical consultation?]
   
   

7. What can TDI do to encourage you and your staff to join the TDI-Learning Collaborative and participate in TTA events and training? What would be a selling point for you and your staff? What concerns do you have?

8. Would you be interested in being a pilot site – involving 18 months of working intensively with the TDI team on projects specific to your needs, plus resources for your state to ensure staff capacity? What appeals to you? What concerns you?

Version D: Vendors

1. When answering this set of questions, please think about some of the challenges you have faced when working with social service agencies in a state:

• Did you have any difficulties finding people who could answer your questions and give you the information you needed? If so, please describe the main problems.

• What hurdles did you face in starting your work?

• Do you think things would have gone more smoothly if agency staff had been better trained? If so, what type of training would have helped for administrators? IT staff members? Human service staff members?

• Have you faced significant challenges integrating data from multiple agencies?

• What were the key data quality issues you faced?

• What challenges have you faced in implementing effective data security measures?

• What ware some key challenges you have faced in keeping projects on schedule?

• Which states have systems that you think are working well and why?

2. What state TANF agencies do you currently work with? In what capacity (e.g., developing and/or maintaining their TANF benefit system or a management information system for program services, conducting research projects)? (Note: Remind vendor their particular answers will be kept private.)

3. Have you worked with research firms, academics, or other organizations to conduct research or produce program performance outcomes in a particular state or locality? What was your role? What worked? What were some of the lessons learned? (Note: If there is a concrete research project, please ask for a reference to any relevant documents.)

4. What states do you perceive as strong users of TANF data (considering area such as program improvement, analyses of program effectiveness, program integrity, and research)? [Prompt: Examples of strong users could be states or localities that utilize predictive analytics to identify cross-cutting issues affecting multiple agencies serving the same participants or reporting on caseload dynamic trends.] Weak users? Please explain.

5. What states do you perceive as strong users of multiple agency data? Weak users? Please explain and identify challenges. [Prompt with examples as needed: North Carolina, South Carolina, Michigan, the Northern Marianas, Washington, etc.]

6. What are the major data quality issues with TANF data? What’s missing from TANF data?

7. What do you view as the major barriers to improved usage of TANF program data for program performance and assessing the effectiveness of TANF programs?

8. What single change would make the integration of TANF data with other systems easier for determining eligibility and enrollment? What change would make it easier to use data for analytic purposes?

9. TDI is a federal investment intended to provide several years of training, technical assistance, and technical consultation to state and local TANF agencies. What types of technical assistance do you believe would best support TANF data agency staff in better use of data? Are there specific topics we should support? [Prompt as needed: such as caseload dynamics, predictive analytics to identify those most at risk, integrating TANF with other relevant data?] Are there specific ways to deliver technical assistance that you find particularly helpful? [Prompt as needed: in-person vs. virtual; webinars; written tool kits or other documents like infographics; videos; hands-on support; working on specific projects with one-on-one technical consultation?]

Research and Analytic Capacity

1. [For federal TANF leader and organizations, state TANF leaders and staff, and vendors]: Do you observe states regularly using TANF data for research or performance management purposes, and if so, how?

2. [For researchers and vendors]: What are the TANF-related research or program performance questions you’d most like to see answered? Where have you been successful? Where have you been unable to address such questions and why (e.g., due to current data limitations)? Can you give us some examples?

3. [For federal TANF leader and organizations, state TANF leaders and staff, and vendors]: What kinds of skills would you like to see TANF staff and/or graduate trainees have more training on? (For example, fostering productive research/practice partnerships, developing interagency data sharing agreements, techniques to link or integrate multiple databases, approaches to measuring performance and effectiveness – such as rapid cycle tests, studying caseload dynamics, predictive analytics to help in targeting services, behaviorally informed practices & determining effectiveness, etc.)

Federal Reporting

[Ask federal TANF leaders and organizations and state TANF leaders and staff only.]

1. In your experience, what challenges do states find in federal TANF data reporting?

2. Perhaps you are aware that states have an option to submit either a universe of TANF recipients and their characteristics or a stratified sample. (If they confirm they do:) What factors motivate [depending on Federal vs State: the/your agency’s] decision to submit data for a sample of individuals served by the agency versus data for the whole universe of individuals served?

Data Quality

[Ask federal TANF leaders and organizations, state TANF leaders and staff, researchers, and vendors only.]

Thinking about the data and expertise in the agencies you’ve worked with [draw on their experience in the background information section]...

1. What are the most serious TANF data quality issues that need to be addressed?

2. In what ways could the quality and usefulness of TANF data be improved? Please be as specific as possible about individual data elements.

Technology and Data Systems

[Ask federal TANF leaders and organizations, state TANF leaders and staff, and vendors only.]

Thinking about the data and expertise in the agencies you’ve worked with [draw on their experience in the background information section]…

1. How would you characterize the usability and security of TANF agency technology and data systems?

2. Are you aware of any past or current efforts to improve TANF-related information technology (e.g. a movement to cloud computing, integrated systems)?

Data Sharing and Governance

Thinking about the data and expertise in the agencies you’ve worked with [draw on their experience in the background information section]…

1. [For federal TANF leader and organizations and state TANF leaders and staff]: How would you characterize TANF agencies’ comfort with sharing data with other agencies?

2. Are there any states you perceive to be particularly strong in terms of integration of TANF data with other data sets? Please explain.

3. What do you think are the primary barriers to data sharing and integration across human services agencies?

4. [For federal TANF leader and organizations, state TANF leaders and staff, and vendors]: Are there any states you perceive to be particularly strong in terms of TANF data analysis? If so, what topics were addressed and how? If not, what do you think are the barriers to more data analysis?

Closing

1. What is the most critical challenge for efforts to use data to improve the efficiency and effectiveness of TANF? What do agencies need to get there?

2. What is the biggest barrier to better data use?

3. Is there anything else you’d like to share?

4. Can you recommend other individuals and organizations that we should speak with to characterize the use of data and analysis in TANF and opportunities for training and technical assistance?

10