In-depth participant interviews

Attachment F
IN-depth participant interview guide and materials

f1. Telephone Recruitment Script

f2. Follow-Up Email

f3. Interview topic guide

This page has been left blank for double-sided copying.

15a. Telephone Recruitment Script

Evaluation of Employment Coaching for TANF and

Related Populations

Implementation Study

Protocol for Recruitment Telephone Calls for Participant Interviews

Hello, my name is [INTERVIEWER NAME], and I’m calling from [Mathematica]. May I speak with [FIRST NAME, LAST NAME]?

Hi [FIRST NAME]. I’m calling from [Mathematica], which is an independent evaluator assisting the U.S. Department of Health and Human Services on a national study to learn more about whether providing people with a coach to assist them with setting and meeting goals helps them get and keep a job. The study will also help us learn whether there are ways to make the [PROGRAM] work better.

[If participant previously participated in an interview: Thank you for participating in an in-person interview about your experiences last spring.] Now that the world has changed so drastically because of COVID-19, we are [again] talking to people like you who work with a coach to understand your experiences with the program during this time and how they may have changed. We’re also interested in understanding how your life has changed as a result. When you enrolled in the program and consented to be a part of the study in [MONTH] [YEAR], you learned that you may be asked to participate in an interview.

We would like you to participate in a phone or video interview during [INSERT POTENTIAL INTERVIEW DATES]. This interview will last approximately two and a half hours. We will give you a $60 gift card when you complete the interview in appreciation for your participation. We would like to talk to you about your experiences with and views of [PROGRAM] during COVID-19. Your participation in this study is very important and will help improve these programs for people like you.

Your participation in this study is voluntary. During the interview, anything you say will be private and we won’t use your name or any other identifying information when we report the results of our study. You may also choose to not answer any question you do not want to answer. In other words, no one will know who you are, but your feedback will be a valuable part of information shared with policymakers and practitioners to improve programs.

1. We would like to schedule an interview with you during [INSERT POTENTIAL INTERVIEW DATES]. What day and time is best for you?

Respondent provides a day and time 01 CONTINUE TO 1a

Respondent is not available that week 00 GO TO END

INTERVIEWER: TIME SLOTS ARE 9AM, 1 PM, AND 4PM. TRY TO USE THOSE TIMES. IF RESPONDENT CANNOT MEET AT THOSE EXACT TIMES, FOLLOW THE GUIDELINES BELOW:

	9 AM	1 PM	4 PM
For this time slot, do not schedule an appointment later than:	10:30 AM	2:30 PM	5:30 PM

1a. INTERVIEWER: CHECK SCHEDULE FOR AVAILABILITY. WORK WITH RESPONDENT TO FIND CONVENIENT TIME AND DAY.

Appointment available 01 CONTINUE

Appointment not available 00 GO TO END

1b. Slot is available: Great. That time is available and I’ve added you to the schedule. You will be talking with an interviewer from Mathematica, «Interviewer», on [REPEAT DAY AND TIME]. The interview will take place over the phone or by video. Would you prefer to talk to «Interviewer» over the phone or using video?

2. Contact Confirmation: We will email you a confirmation letter [if choose video: with instructions for joining a video call with «Interviewer»]. I would just like to take a moment to make sure we have the correct contact information for you.

CONFIRM ALL INFORMATION AND MAKE CORRECTIONS IF NECESSARY.

«Interviewer» will call you the day before the interview to introduce [him or herself] and re-confirm the day, time, and meeting information for the interview.

Please contact us at [INSERT TOLL-FREE NUMBER] if you have any questions or need to reschedule. Thank you! END CALL.

END. If not available during this time frame: I’m sorry that those days are not convenient for you. Right now I’m scheduling appointments only for those days. I will call you back if we decide to schedule appointments on other days. Thank you for your time.

15B. FOLLOW-UP Email

Subject line: Mathematica Interview Confirmation

Hello!

This email is a reminder that an interviewer from Mathematica, «Interviewer», is scheduled to talk with you on «APPT_DATE» at «APPT_TIME». The interview will be conducted [over the phone/by video]. [If by phone only: «Interviewer» will call you at that time.] [If by video: Please join the meeting at that time by clicking this link «Webex link»]. Mathematica is conducting these interviews as part of a study on behalf of the United States Department of Health and Human Services.

The interview will take about two and a half hours and we will give you a $60 gift card to thank you for your time. All the information that you give in this interview is private and your participation is completely voluntary and will not affect any government assistance you may be receiving.

We look forward to speaking with you soon! Thanks for your cooperation and help! If you have any questions, please call [INSERT TOLL-FREE NUMBER].

Sincerely,

An Affirmative Action/Equal Opportunity Employer

15C. INTERVIEW topic guide

Evaluation of Employment Coaching for TANF

and Related Populations

In-depth Interview Topic Guide

Introductory Statement and Consent

Thank you so much for meeting with me today. My name is __________ and I am with a company called [Mathematica]. We are an independent evaluator assisting the U.S. Department of Health and Human Services on a national study to learn more about whether providing people with a coach to assist them with setting and meeting goals helps them get and keep a job. The study will also help us learn whether there are ways to make the [coaching/program] work better.

Now that the world has changed so drastically because of COVID-19, we are talking to people like you who work with a coach to understand your experiences with the program during this time and how they may have changed. We’re also interested in understanding how your life has changed as a result.

Your participation in this study is voluntary. Our conversation will take about two and a half hours. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB number for this information collection is 0970-0506 and the expiration date is 03/21/2021. During our conversation, anything you say will be private, except if you say something that suggests you are very likely to harm yourself, that you are planning to hurt another person or child, or that someone is likely to harm you. We also won’t use your name or any other identifying information when we report the results of our study. We also won’t share what you say with your program or your coach. You may also choose not to answer any question you do not want to answer. In appreciation for participating in the interview, we will send you a $60 gift card.

Please remember that we are interested in your experiences and opinions; there are no right or wrong answers. My hope is that you will feel comfortable talking with me, but, if at any time I ask you a question and you don’t feel comfortable talking about the topic, feel free to say so and we will move on.

You may feel uncomfortable answering some questions. You can refuse to answer those questions if you wish, and it will not change your participation in the study or program. Although researchers will take many steps to protect all study information, there is a small risk that non-researchers could see it. In addition, representatives from the New England Independent Review Board may inspect and have access to private information as they ensure your rights as a study participant are protected.

Do you agree to be interviewed for this study? [IF YES:]

I would like to record our conversation so I don’t miss anything. No one will hear the tape except for researchers and the person who types it up. Is it okay with you if I tape this conversation? If you want me to turn the tape off for any reason or at any time, just say so.

[INTERVIEWER: IF RESPONDENT CONSENTS, PRESS THE RECORD BUTTON]

PAPERWORK REDUCTION ACT OF 1995 (Pub. L. 104-13) STATEMENT OF PUBLIC BURDEN

Through this information collection, ACF is gathering information to understand how coaching programs are serving individuals. Public reporting burden for this collection of information is estimated to average 150 minutes per respondent, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection. This is a voluntary collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information subject to the requirements of the Paperwork Reduction Act of 1995, unless it displays a currently valid OMB control number. The OMB # is 0970-0506 and the expiration date is 3/21/2021. If you have any comments on this collection of information, please contact Sheena McConnell, Mathematica, 1100 First St. NE, Washington, DC 20002-422.

Note to interviewers: Complete the table below before for each interview.

Date of Interview
Interviewer
Coaching program

A. Relationship building, participant background

1. Tell me a little bit about yourself.

1. Family composition (number and ages of children)

2. Favorite activities and hobbies

B. Client hopes, expectations, and service needs

1. Tell me about what brought you to [name of program]. How did you find out about the program? What did people say about the program?

2. When did you start participating in [name of program]?

3. What were your hopes when you first came to [name of program]?

4. What do you think is the purpose of [name of program]?

5. Tell me about what you have been doing as part of [name of program] during the COVID-19 pandemic (from March 2020 to today).

1. Orientation

2. Assessment

3. Goal or activity planning

4. Activities engaged in (job search, work experience, coaching sessions, training/education, etc.)

5. Work support services received (e.g., child care assistance including vouchers or funds; transportation assistance such as gas cards or bus passes; clothes, uniforms, tools or other supplies and equipment; tuition assistance; assistance finding stable housing; assistance with budgeting, credit, banking, or other financial matters; or assistance expunging a criminal record or other legal assistance)

6. Personal support services received (e.g., mental health counseling, domestic violence counseling or support, substance abuse treatment, or services to help with stress reduction or other approaches to improve emotional well-being)

7. What other kinds of help do you get from [name of program]? (e.g., help finding a job, help finding other services you need, help setting goals for your future.) Is the help useful? Why or why not?

6. Are these the same things you were doing as part of the program before COVID-19 started? If no, why are you newly participating? Probe for whether the activities were not offered before, whether the participants’ needs or availability changed.

C. Relationship with coach and other workers

1. Tell me all of the people you are working with at [name of program].

1. Role and types of help each worker provides

2. Tell me about [name of coach].

1. What is the name of your coach?

2. Has this been the same person throughout your engagement with the program?

   1. Have you had a different coach who has been assigned to work with you since COVID-19 started?

3. Whether currently working with coach

1. If relationship has ended, reasons why

4. Length of time working or worked with coach

5. How would describe coach to a friend

6. How would describe what coach does (e.g., coach’s role), and changes to role since COVID-19 started

7. How would describe overall relationship with coach and changes in relationship during COVID-19

8. Since COVID-19 started, how often contact occurs, how long sessions last, which modes of interaction are used (in-person meetings, phone calls, email, text, video conference), how often each mode is used, who initiates contact (coach or participant)

   1. Was this different before COVID-19? Why did things change?

   2. [If have contact by video:] What application or program did you use for your video meeting? For example, FaceTime, Zoom, WebEx, Google Meet.

9. If meet in-person, where in-person contact typically occurs and how often. Length of typical session. Was this different before COVID-19?

10. If in-person contact is one-on-one or in group settings. Was this different before COVID-19?

11. If coach’s availability has changed during COVID-19

3. What do you talk about when you meet with [name of coach]? Walk me through a recent session with your coach.

1. Content of session

2. Amount of time during coaching session spent on goal setting

3. Tools, activities, and types of materials used during coaching session

   1. Opinions of usefulness

4. Extent to which the participant determines what the participant and coach discuss

4. How did this meeting compare to your meetings with [name of coach] before COVID-19?

1. Did you talk about different things?

2. Was where or how you met different? (For example, over the phone instead of in-person?)

3. Was the length of the meeting different?

5. Do you meet with a case manager in addition to a coach?

1. Whether currently working with a case manager

   1. If relationship has ended, reasons why

2. How long participant has been or was meeting with case manager. How often contact occurs, how long sessions last, and which mode of interactions are used (in-person meetings, phone calls, email, text). Was this different before COVID-19?

3. If meet in-person, where in-person contact typically occurs and how often. Length of typical session. Was this different before COVID-19?

4. If in-person contact is one-on-one or in group settings. Was this different before COVID-19?

6. [If have case manager] What do you talk about with your case manager? Walk me through a recent session with your case manager.

1. Content of meeting

2. Amount of time during meeting spent on goal setting

3. Tools, activities, and types of materials used during meeting

1. Opinions of usefulness

4. To what extent the participant determines what the participant and case manager will discuss and the next steps in the conversation

5. Perceived difference between participant’s coach and case manager

6. Whether the guidance the participant receives from caseworker and coach ever conflict (e.g., around what you need to do)

7. Going back to talking about [name of coach]. Do you ever leave your meeting with [name of coach] with things to do before your next appointment?

1. Types of tasks. Did the types of tasks change during COVID-19?

2. Who determines the tasks. Did who determines the tasks change during COVID-19?

3. What specific tasks did participant find most and least useful

4. Whether participant typically completed tasks. If did not complete certain tasks, why not?

5. Did the likelihood of the participant completing tasks change during COVID-19? If so, how and why did it change?

8. Does what you talk about during a session with [name of coach] influence your behavior after you leave? How so?

9. What do you think about the meetings you have had with [name of coach] during COVID-19?

1. Likes and dislikes

2. Opinions of usefulness during COVID-19 and compared to before the pandemic

10. Sometimes people miss meetings they have scheduled because other important things come up, or they are busy and forget about their meeting, or unexpected things happen like their car breaks down or their child care falls through. Have you missed any meetings with [name of coach] or faced challenges to receiving other program services during the pandemic?

1. Reasons meetings have been missed or engagement was challenging

2. Consequences of missing meetings

3. Whether and how started attending meetings again

4. How participants ability to attend meetings or receive other program services differed before the pandemic. Were they more or less likely to miss meetings before the pandemic? Why?

11. We just talked about missing meetings with [name of coach]. Has there ever been a time when you stopped participating in the program altogether (including not meeting with your case manager, if applicable) since the pandemic started? Can you tell me more about that? What was happening in your life at the time? What changed that made you stop participating?

1. Has participant ever lost some or all of their benefits because of not completing program requirements—that is, been sanctioned

12. What has helped you to continue to participate during the pandemic?

D. Service delivery and the goal achievement process

1. Do you currently have a goal or goals that you are working toward with your coach? What is that goal (those goals)?

1. [If participant answers no] Did you have a goal or goals you worked toward with your coach in the past? What was [that goal/those goals]? [If they have no goals they worked toward with their coach, skip to question 7]

2. [If multiple goals] Which goal is the most important to participant? (Focus on this one in next questions)

3. When was the goal developed?

4. How was the goal developed?

1. Role of coach in developing goal

5. How important is the goal to the participant? How motivated is the participant?

2. Did anything get in your way when you were figuring out what your goal would be?

1. Challenges faced

2. Strategies for addressing challenges

3. What happened if the participant didn’t have a goal when they were in the program?

3. What did you do after you identified your goal?

a. Action steps developed to achieve goal

b. Who developed the action steps and how

i. Role of coach in developing action steps

4. [If action steps were planned] Did anything get in your way while planning the steps you were going to take?

1. Challenges faced

2. Strategies for addressing challenges

5. Tell me about your experience with working on your goal during COVID-19

1. Challenges faced getting started and strategies for addressing challenges

2. If action steps were developed, whether they were taken to meet the goal. If action steps were not developed or the steps developed were not taken, how else did the participant work toward the goal?

3. Things that helped with making progress (e.g., things internal to the participant, like motivation and hard work, and things provided by the program, like support from a coach)

4. Challenges with completing the action steps or goal (e.g., having difficulty getting started, having difficulty finishing tasks); new or worse challenges as a result of COVID-19

5. What did participant do when they faced these challenges?

6. How did what they actually do differ from what was planned?

7. What rewards, incentives, or other forms of motivation did the participant receive from the coach, if any

6. As a result of COVID-19, did you change the goal you were working on at [name of program] or the planned steps (if steps were developed)? If yes, how so? What were you working on or planning to do before?

1. Process for reviewing and revising goals and action steps (how often, how communicated)

2. Role of coach in helping to refine and change goals and action steps

3. Why goal or steps changed

7. Before working with your coach, did you have goals? What were they?

8. Please tell me about a goal that you worked on but did not achieve since you started working with your coach.

1. What was the goal?

2. When did you set the goal?

3. Did the coach help set the goal?

4. What prevented you from achieving the goal?

1. Other assistance or services that would have been helpful

5. Did you revise your goal, or did you just decide not to continue to work toward it?

   1. Process for reviewing and revising goals (how often, mode of communication)

   2. Role of coach in helping to refine and change goals

9. Please tell me about the goal that you have achieved that you are most proud of.

1. What was the goal?

2. When did you set the goal?

3. Did the coach help set the goal?

4. If achieved goal, what helped the participant achieve the goal? Was the coach helpful? Were other people or services helpful?

5. If did not achieve goal, what prevented the participant from achieving it?

1. Other assistance or services that would have been helpful

10. Did you set a new goal after you reached the goal you were working on? If so, what was it? Did it build off the last goal you achieved?

11. Does/Did [name of coach] do anything to motivate you when you are/were working with them on your goals? [Note: This question should be asked about the goal-setting process in general, not about the specific goal they have been discussing.]

1. Role of coach in motivating participant to achieve goal(s)

2. Receipt of rewards, incentives, or other financial support (e.g., things like gift cards, gas money, certificates, or praise). If received:

1. How much received

2. Uses of the funds

3. Changes in availability since COVID-19

3. Perceived effectiveness/usefulness of rewards, incentives, or other motivation techniques

12. Does/did [name of coach] ask you about how you are/were doing on your goal(s)? What happened if you had not met a goal? What about if you had met a goal? [Note: This question should be asked about the goal-setting process in general, not about the specific goal they have been discussing]

1. Did coach ask whether you had taken the action steps and met goals? What happened when you faced challenges?

13. Did [name of coach] do anything to celebrate your achievements? What did he or she do?

E. Progress toward improving work and life outcomes

1. Tell me about your work experience before you started [name of program]?

1. Employment status when started the program

2. Employment history in two years prior to starting program -- unemployment and employment

2. Are you working now?

3. [If working] Tell me more about the job you have now.

1. Type of job, how long been with employer; views and expectations on how long participant expects to keep this job

1. How got this job; whether and how coaching program helped in finding or maintaining a job; other sources of help

2. Training or education programs participated in to prepare or qualify for job

3. Satisfaction with current job, major positive aspects and negative aspects

4. Has COVID-19 affected your job or the work you do? How so? Probe for working from home, safety measures, social distancing, fewer or more hours.

5. Whether currently looking for a different job

6. [If looking for a different job] Current or most recent job search experience (for example, what job search activities they are doing, challenges related to COVID-19, help received with it, type of job looking for, whether and how coaching program is helping with finding a new job)

4. [If not working:] When was the last time you worked?

1. What kind of work was it?

2. How long did you work there?

3. Why did you stop working there? Probe if COVID-19-related.

   1. [If related to COVID-19:] Is your [job loss, layoff, or furlough] temporary?

   2. [If yes:] Do you know when you will be able to return to work?

   3. [If yes:] Do you have concerns about returning to work? Describe.

5. [If not working] Are you doing anything to look for or prepare for a job?

1. Type of job looking for or preparing for

2. Has COVID-19 impacted your job search? Probe for if there are fewer jobs available, concerns about safety, interested in new/different industries due to COVID-19, challenges with child care or transportation

3. Current or most recent job search experience (for example, what job search activities they are doing, challenges related to COVID-19, help received with it, whether and how coaching program is helping with finding a job)

4. Training or education programs participating in to prepare or qualify for job

6. Has [name of coach] helped you get closer to reaching your work goals? To reaching other goals for your life? How so?

7. What are your goals for your next job/career?

a. Aspirations, goals, and plans related to future employment/career

F. Service and information needs related to COVID-19

1. During COVID-19, we’ve been asked to stay home, and we can’t do a lot of what we normally do in person. Have you been able to use technology, like a cell phone or computer, to help with this?

   1. [If yes:] What forms of technology have you used? Probe for cell phone, smart phone, computer, social media, video conferencing.

   2. Have you used this technology to stay in touch with your coach?

   3. Have you used it to access other services you or your family need?

   4. Have you faced any challenges to using the technology? Probe for lack of hardware or software, lack of internet, lack of phone minutes, literacy issues.

   5. Did [program] help you with getting access to the technology you need?

   6. Did you get help from somewhere else? For example, free Wi-Fi in the community or a laptop provided by a children’s school.

2. COVID-19 has had a big impact on all our lives. What are the major challenges you and your family have faced during this time? Probe for challenges related to stress (including financial) and personal and family well-being, mental and physical health, meeting basic needs, and child and dependent care.

   1. Have you been able to resolve these challenges? How so?

   2. Has [program] been able to help you? How so?

   3. Has anyone else or any other program helped you? Probe for family or friends, other government or community programs or benefits, church.

3. Before the pandemic, were you receiving any support from the government, like cash assistance or TANF, food stamps or SNAP, health care, or unemployment insurance?

   1. Are you receiving any of those benefits now?

   2. [If receiving benefits before and during the pandemic:] Have your benefits changed as a result of the pandemic? Probe for changes in benefit amount or program requirements.

4. [If not already discussed:] Have you faced any employment challenges during this time? Probe for job loss, layoffs, safety concerns, working remotely, lack of job opportunities.

   1. Have you been able to resolve these challenges? How so?

   2. Has [program] been able to help you? How so?

   3. Has anyone else or any other program helped you? Probe for family or friends, other government or community programs or benefits, church.

5. Did [program] provide you any information about the virus?

   1. If so, what information did they provide?

   2. Was the information helpful?

6. Would additional information about COVID-19 be helpful that the program did not provide? Describe.

G. Post-program – what might be different as a result of the coaching experience

1. Would you say the coaching program has changed you or the way you do things? How so?

1. Differences in views, knowledge, skills or behavior related to:

1. Goal setting (for example, extent to which participant is goal setting on their own)

2. Employment (for example, job search, workplace behavior, long-term plans/career plans)

3. Feelings about the future

4. Motivation

5. Personal strengths and belief in ability to succeed

6. Stress management

7. Interactions and relationships with child(ren) and other family and friends

2. Assessment of what coaching program experiences and other factors may have contributed to any changes

2. Please think about other times you have worked with someone in a program who was helping you get a job, like a case manager, case worker, or a job specialist. How is working with [name of coach] different?

3. Has participating in [program] been helpful to you during COVID-19? How so?

1. Has it been more or less helpful than before COVID-19?

4. What are the three main benefits you’ve received from the coaching program during COVID-19?

5. What do you think the program did particularly well in operating during COVID-19?

6. What do you wish was different about how the coaching program operated during COVID-19?

H. Wrap up

1. What are your hopes for the future?

2. Is there anything else you’d like to share about your experience with the program or your coach during COVID-19?

I. Participant demographic information

Before we close, I’d like to ask just a few more background questions about you to help us describe who we talked to during these discussions. If you don’t feel comfortable, you don’t have to answer them, but it would be helpful for us to know. So, if you don’t mind sharing, could you tell me:

1. What is your age?

2. What is your sex?

3. Are you Hispanic, Latino/a, or of Spanish origin?

4. What is your race (e.g., American Indian or Alaska Native, Asian, Black or African America, Native Hawaiian or other Pacific Islander, White)?

5. Finally, can you tell me the email address where we should send your gift card?

1. If you don’t use email or won’t be able to access a gift card via email, we can mail you a gift card. To what address should we mail the card?

Do you have any questions for me? Thank you so much for your time. We really appreciate all that you have shared.