CQA Debriefing Focus Groups

Focus Group Moderator’s Guide

Date: ____/____/______ Start time: ___________ AM / PM

Location: Irving, TX Tamarac, FL Nashville, TN El Paso, TX Blythewood, SC Jacksonville, FL Tempe, AZ Pueblo, CO New York, NY Kansas City, KS

Notes: Inbound or outbound – need to tease out which type of call that comment is relevant to

Coaching: supposed to be doing it weekly good or bad;

[TRACIE HAS AGENTS SET UP NAME PYRAMIDS FOR EACH AGENT AS THEY ENTER THE ROOM SO EVERYONE CAN SEE EACH OTHER’S NAMES AS THEY SIT AROUND THE TABLE.]

Please write down your name and set up a pyramid.

INTRODUCTION

Welcome! Thank you for coming to talk with us today. We know we all lead busy lives so we really appreciate that you are taking this 90 minutes of your day to help us improve the agent desktop for the 2020 Census.

My name is [MODERATOR NAME] and I will be guiding today's conversation. I work for the Census Bureau in the Center for Behavioral Science Methods. I have a note taker who also works with me and there are some observers who want to hear the conversation. You all have already met Tracie, who is with you all in person.

We’ll be recording our session today so that we don’t miss anything important. Does anyone have any questions before we get started?

RULES

Great! Let’s go over a few basic rules.

• First, I will be asking a lot of questions. There are no right or wrong answers to my questions; we just want your honest opinions. We don't need to have everyone agree. In fact, if you have a view different from others it is especially important for us to hear it because you may be the only person who is representing a different point of view here today.

• Be respectful of other's people comments. It is fine to disagree with each other’s comments

We are interested in listening to everybody's opinion in a respectful manner.

• Only one person speaks at a time and we’d like everyone to get a turn. Please do not have a side conversation with the person sitting next to you or speak while somebody else is talking. Please speak up and let us know what you are thinking. If you are the type of person who always participates, please give a chance to others to talk.

• Due to limited time, I must move the discussion along and may sometimes have to interrupt in order to do so. Please do not be offended.

• Please keep what everyone says here private. During the group conversation, we use first names only. We keep what you say entirely confidential and I ask that each of you treat confidential what others say around the table. Your names will not be used in any reports.

• Please keep respondents’ information confidential. When you tell us about the experiences you had administering the survey to a respondent, please don’t share respondents’ names or any other information that could identify a respondent.

• Please turn off your phone and put it away in your bag or pocket before we begin.

ICEBREAKER

Let's go around the room and have each one of you tell us briefly:

• Your first name or nickname

• Tell us about your experience with surveys. How long you’ve been working as an CSR?

• Tell us about how many calls you received for the 2020 Census. About how many of those were where the respondent wanted to give their answer to fill out the survey?

Let me go first. My name is…

Now I’ll have Tracie introduce herself.

PROBES

Thanks for telling us a little bit about yourself. Today we’re going to talk about your experience as a CSR. We want to hear from you so we can better understand the experience you and callers had during the 2020 Census. If you completed any surveys with respondents in a language other than English, please keep in mind that we’re interested in your experience both with the survey in English and in other languages. We will follow the list of questions that you should have seen earlier. For the questions about the data capture tool, we can have pictures of those screens in the Skype window. We want to make sure we understand which questions you are referring to during the discussion. We know this was a very challenging time to be a CSR and there will be some time at the end to share how the COVID-19 pandemic affected your work experience.

Let’s get started!

Logistics and Training (10 minutes)

First, we’re going to the discuss your hiring and training experience.

1. Can you tell us more about your onboarding experience?

   1. IF NEEDED: How efficient was the onboarding process?

2. Were there any challenges that you faced that the training did not prepare you for?

3. Which training did you think was most helpful to prepare you for answering calls?

Communication

Let’s talk now about different ways of communicating updates, like alert emails, training microbursts, messages about new FAQs, desk-to-desk conversations with your supervisor, and daily hot topics.

4. How do you recall learning about updates?

5. Of all the methods we just talked about, which method do you think worked the best for giving you new information?

   1. What didn’t work very well?

6. Were there any periods when you did not have enough time to read emails that you received?

   1. IF NEEDED: Can you tell me more about that?

Quality

7. Can you tell us more about any floor support you received during difficult calls?

8. For the next census in 2030, we are thinking about using the virtual floor support method instead of the raise the flag in-person method for helping CSRs with questions.  What do you think of that idea? What changes should we make to improve virtual floor support for next time?  

9. What was your experience like with coaching?

10. What was your understanding of how you were being scored?

    1. [If needed] How fair or unfair did you think the scoring was?

Blue means: We will cover at least one of the three topics at each center. If there is more time, we can cover the other two topics. Topics will rotate by center

Data Capture Tool - Question Content – English, Spanish, and Other Non-English Languages

Now let’s discuss the data capture tool.

11. Were there any questions where you suspected that respondents gave the wrong answer?

    1. Which questions?

    2. How about any questions in other languages?

12. Were there any questions that respondents misunderstood or asked you to repeat?

    1. [IF BILINGUAL GROUP] Was it the same for the other languages?

13. [IF BILINGUAL GROUP] Now let’s talk about the non-English languages that you used. Were there any translated questions in the data capture tool that weren’t conversational or that were awkward to read out loud?

    1. Which language and which questions?

User Interface

We are also interested in how well the data capture tool worked for you.

14. If you could change anything about the design or layout of data capture tool, what would it be and why? Think about coding inbound calls, starting an enumeration, coding the disposition, linking FAQs, etc.

15. [If not already mentioned] When searching for answers to FAQs, did the interface work well for you or did it frustrate you?

    1. Tell me more.

16. What about other features of the agent desktop, like the scripting panel, resource library, and CSR dashboard:

    1. What worked well and what would you have changed, if you could have?

FAQs

Now we’re going to talk about the FAQs.

17. What were the most frequent questions and concerns callers had?

18. Were there any FAQs that worked really well?

19. Did respondents have any questions that you could not answer using the FAQs?

    1. What were they?

20. Were you able to submit feedback on which FAQs were missing or needed to be updated?

21. [IF BILINGUAL GROUP] Now let’s talk about the non-English languages that you used. Were there any translated FAQs that weren’t conversational or that were awkward to read out loud?

    1. Which language and which FAQ?

COVID 19 Pandemic: (last 10 minutes)

We know that you all were working during the COVID 19 pandemic, and we’d like to learn more about that experience. Some of you may have been out of the office for a few weeks at a time.

22. What was your experience like with the return to work training?

23. When you came back, were you able to get up to speed quickly or did you feel like you missed out on important information?

24. Did the safety precautions and the way that COVID related precautions and updates were shared throughout the call center meet your expectation or not?  How so?

[IF NEEDED] If not, what would you have done differently?

[IF NEEDED] What communication or procedure did you feel was the most reassuring or informative?

Anything else

1. Is there anything else you’d like to tell us about that you haven’t had a chance to share yet?

Thank and End

Thank you for your feedback. This information will help us make improvements for the 2030 Census.

7