WIC Child Retention (Private Sector) (2 of 2)

Appendix A.3 Pretest – Clinic Staff OMB Number: 0584-0613

Expiration Date: 02/28/2021

WIC Child Retention Survey

Cognitive Interview Pretest Protocol

This information is being collected to assist the Food and Nutrition Service in understanding why children ages 1 to 4 years leave the Special Supplemental Nutrition Program for Women, Infants, and Children (WIC). This is a voluntary collection and FNS will use the information to better understand Local Agency strategies related to improving child retention in the WIC Program. This collection does not request any personally identifiable information under the Privacy Act of 1974. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-0613. The time required to complete this information collection is estimated to average 3.34 hours per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302 ATTN: PRA (0584-0613). Do not return the completed form to this address.

Sponsored by the U.S. Department of Agriculture, Food and Nutrition Service

May 20, 2019

PRE-INTERVIEW STEPS

• 2M will schedule the appointment and copy Amanda on the confirmation email with the participant stating that Amanda will send the survey about 15 minutes before the interview.

• If needed, Sarah is Amanda’s backup.

• Amanda will email the survey, confirm the telephone number she will call from, and ask the participant to print or pull up a copy of the survey but not to review it.

• Amanda will call the participant at the scheduled time.

INTRODUCTION

Hello my name is [NAME] and I am with Mathematica Policy Research. Thank you again for agreeing to help us test our survey questions. First, did you get the email with the questionnaire in it? [If not, wait until they get it.] Great! Were you able to print it out or do you have it on the computer only? Now I will give you a little background information and tell you more about what we will be doing today.

We would like your feedback on the questions to make sure they are clear, use words or phrases that make sense to WIC staff, and request information that is relevant and relatively easy to recall. We want to learn from you what we can do to improve the questions.

When we’re ready to start, I will ask you to complete the survey as you normally would but with one major difference: I would like you to read and think aloud as you answer the questions. I am interested in your answers, but I am also interested in your thought process as you answer the questions. As you take the survey, I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

For example, if there were a question such as “How many windows in your house?” and you were thinking aloud as you answered that question, you might say things like “I’m picturing my living room and there is a bay window, so I’m not sure if that’s considered one window or three. There is a small window in the bathroom, and a small one in the front door, two in the kitchen, etc.” You walk me through how you get to your answer. Any questions about thinking aloud as we go along?

From time to time, I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no wrong answers. I really want to hear your opinions and reactions, so don’t hesitate to speak up whenever something is unclear, is hard to answer, or doesn’t seem to apply to you. This will help us improve the questions.

I want to let you know one more thing about the survey. People will complete it on the web. The web survey will be able to present the appropriate questions based on someone’s earlier answers. We can’t do that on paper today, so I will occasionally let you know which question you should go to next.

With your permission, I’d like to record the interview so that I can focus on listening to you instead of note-taking. The recording will not be shared beyond the study team. Your answers to the survey questions and any information you provide to me today will be kept private. Your name and any personal information you may provide will not be used in our reports. May I have your permission to record our interview today? Thank you.  START RECORDING.

Finally, when we finish the interview I will confirm your mailing address so we can send you a $30 gift card as a thank you for your help. Do you have any questions before we begin?

Okay, let’s begin. Please go ahead and open the survey and start on the Introduction page. Remember, you’re completing it as you normally would but reading and thinking out loud and with questions from me. Please keep your WIC site in mind as you answer the questions. You can go ahead and start.

General Probes: Use the following probes as needed to elicit more feedback from participants during the think-aloud process. Selected probes will be used throughout the interview in addition to the question-specific probes that are provided underneath individual survey questions.

• In your own words, please tell me what you think this question is asking.

• Can you tell me more about that?

• What makes that clear or unclear?

• What level of difficulty did you have answering this question?

• Please tell me what you think about these answer choices.

  • Are any unclear or difficult to understand? If so, please explain.

  • Are there any answer choices you think don’t belong on this list?

  • Are any answer choices missing?

• What led you to select don’t know to that question?

• Do you think there are any additional questions the survey should ask about this topic? Why is that?

INTRODUCTION

Thank you for taking the time to complete this important survey.

As you likely know, child retention, or keeping children in WIC after their first birthday, is a challenge for WIC clinics across the country.

The goals of this survey are to:

(1) Understand why children ages 1 to 4 years leave WIC, and

(2) Identify solutions to keep eligible children ages 1 to 4 years on WIC.

Because you work directly with WIC participants, your input is very important. By participating in this survey, we hope you can help us understand and address the barriers to continued child participation.

This survey takes about 20 minutes to complete. You may complete the survey in one sitting or come back to it later—your answers will save automatically.

Findings from this survey will be summarized across all participants. We will not associate responses with a specific person. Your answers will be kept private to the extent allowed by law.

If you have any questions about the survey—or if you experience any problems with the survey—contact 2M Research at 1-877-440-0050 from 9:00 AM to 4:00 PM EDT Monday through Friday, or by email at usdawicretention@2mresearch.com.

Are the instructions clear or unclear? Why or why not?

Let’s take a closer look at the first goal of the survey. It says we want to understand why children “leave WIC.” What do you think about “leaving WIC” versus “dropping out of WIC”? Which term is easier to understand? Why is that?

I’m also interested in the term “child retention.” Is that a term that you think the people completing the survey will undersatnd. Do you recommend calling it something else?

INSTRUCTIONS

Move forward through the questions by clicking on the forward arrow “Next” button on the bottom right.

If you need to go back, click the back arrow “Back” button on the bottom left.

To exit the survey and come back to it later, click the “X” on the top right. The survey can be accessed again by clicking on the survey link that was provided to you in the email notifying you about the survey.

When you reach the end of the survey, click “SUBMIT.”

CHILD RECERTIFICATION APPOINTMENTS

These questions are about your WIC clinic’s practices for child recertification appointments. Child recertification appointments are for children ages 1 through 4 years only.

If you work at more than one clinic site, please answer for the site where you work most of the time.

How easy or difficult do you think these instructions are to understand? Why is that?

How did you define “where you work most of the time”?

1. At your site, how do WIC participants schedule child recertification appointments?

	SELECT ONE RESPONSE PER ROW
	YES	NO
a. Telephone call	1 □	0 □
b. Text message	1 □	0 □
c. Email	1 □	0 □
d. Online	1 □	0 □
e. In person (for example, during a previous clinic visit)	1 □	0 □
f. As a walk-in for a same-day appointment	1 □	0 □
g. Other (specify)	1 □	0 □

IF Q1G OTHER SPECIFY=YES AND THE OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Do the answer choices cover all of the possibilities for scheduling appointments at your site, or are there other answer choices that you think should be included?

2. Sometimes participants cancel their appointments. When this happens for a child recertification appointment, how likely is it that the participant will reschedule their appointment within the same month?

1 □ Not at all likely

2 □ A little likely

3 □ Somewhat likely

4 □ Very likely

5 □ Don’t know

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

Do you feel the range of the scale is appropriate? Why or why not?

3. How does your site send routine reminders for upcoming child recertification appointments?

	SELECT ONE RESPONSE PER ROW
	YES	NO	DON’T KNOW
a. Telephone call	1 □	0 □	d □
b. Text message	1 □	0 □	d □
c. Email	1 □	0 □	d □
d. Letter or postcard	1 □	0 □	d □
e. Other (specify)	1 □	0 □	d □

IF Q3E OTHER SPECIFY=YES AND SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

Do the answer choices cover all of the possiblities for sending routine reminders at the clinic? Or are there other answer choices that you think should be included?

RESPONDENT: PLEASE ONLY SELECT ONE RESPONSE PER ROW.

4. Whether or not your site uses them, which types of appointment reminders do you believe are most effective in ensuring that participants keep their child recertification appointments?

	SELECT ONE RESPONSE PER ROW
	YES	NO	DON’T KNOW
a. Telephone call	1 □	0 □	d □
b. Text message	1 □	0 □	d □
c. Email	1 □	0 □	d □
d. Letter or postcard	1 □	0 □	d □
e. Other (specify)	1 □	0 □	d □

IF Q4E OTHER SPECIFY=YES AND SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

How do you define “effective”?

READ TO RESPONDENT: QUESTION 5 IS ASKED IF THE SITE SENDS ROUTINE APPOINTMENT REMINDERS, ACCORDING TO QUESTION 3. REGARDLESS OF WHETHER YOUR SITE SENDS THESE REMINDERS, LET’S TAKE A LOOK AT THIS QUESTION.

5. When are routine appointment reminders sent for child recertification appointments?

MARK ALL THAT APPLY

1 □ 2 weeks or more before the appointment

2 □ 1 week before the appointment

3 □ 1 or 2 days before or the same day as the appointment

d □ Don’t know

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

Do you feel the intervals are appropriate? Why or why not?

6. What are the top 3 reasons why parents/guardians say they miss child recertification appointments?

SELECT UP TO 3 RESPONSES

1 □ Forgot about the appointment or was unprepared

2 □ Had a conflict with the scheduled time

3 □ Could not get to the appointment

4 □ Decided to stop participating in the WIC program

5 □ Did not feel they could bring other children to the appointment

6 □ Had a negative experience at a previous visit

7 □ Other (specify)

IF Q6_7 OTHER SPECIFY IS CHECKED AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

What do you think would happen if we people completing the survey to mark yes or no to each item on the list?

How did you choose the “top 3 reasons”?

Are there other answer choices that you think should be included?

READ TO RESPONDENT: PLEASE ANSWER QUESTION 7 FOR THE TOP THREE REASONS YOU SELECTED IN QUESTION 6.

7. How likely are these reasons for missing child recertification appointments causing children to leave the WIC program?

	SELECT ONE RESPONSE PER ROW
	NOT AT ALL LIKELY	A LITTLE LIKELY	SOMEWHAT LIKELY	VERY LIKELY
a. [FILL FROM Q6]	1 □	2 □	3 □	4 □
b. [FILL FROM Q6]	1 □	2 □	3 □	4 □
c. [FILL FROM Q6]	1 □	2 □	3 □	4 □

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

In your own words, what do you think this question is asking? Can you think of a better way to phrase it?

8. How does your site routinely follow up with “no shows,” or participants who miss child recertification appointments?

	SELECT ONE RESPONSE PER ROW
	YES	NO
a. Telephone call	1 □	0 □
b. Text message	1 □	0 □
c. Email	1 □	0 □
d. Letter or postcard	1 □	0 □
e. Other (specify)	1 □	0 □

IF Q8E OTHER SPECIFY=YES AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

In this context how do you define “routinely”?

How do you define “no-shows”? Do you think clinic staff completing this survey will understand this term?

9. Does your clinic track a “no-show” rate specifically for missed child recertification appointments?

1 □ Yes

0 □ No

d □ Don’t know

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

In your own words, what do you think this question is asking? Do you find this question confusing? Why or why not?

How do you define “no-show rate”?

Was the infromation for this question readily available, why or why not? [If needed: who at your agency would know this information?]

How does your agency track the “no-show rate”?

What effect do you think the don’t know option may have for people like you who are completing the survey?

READ TO RESPONDENT: IN THE WEB SURVEY, RESPONDENTS WILL GET QUESTION 10 ONLY IF THE ANSWER TO QUESTION 9 IS NO OR DON’T KNOW. REGARDLESS OF YOUR ANSWER TO QUESTION 9, LET’S TAKE A LOOK AT QUESTION 10

10. Does your agency track a “no-show” rate specifically for missed child recertification appointments?

1 □ Yes

0 □ No

d □ Don’t know

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

In your own words, what do you think this question is asking? Do you find this question confusing? Why or why not?

How does your agency track the “no-show” rate?

Do you think the clinic staff completing this survey will understand the difference between traking the no-show rate at the agency versus the clinic level? Can you think of way to make the distinction clearer?

Do you find this question redundant?

RESPONDENT: ONLY ANSWER IF Q9 IS YES AND Q10 IS YES, OTHERWISE GO TO Q12.

11. Does your agency report the “no-show” rate specifically for missed child recertification appointments to the State WIC agency?

1 □ Yes

0 □ No

d □ Don’t know

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

Was the information for this question readily available, why or why not? [If needed: Who at your agency would know this information?]

READ TO RESPONDENT: PEOPLE COMPLETING THE WEB SURVEY WILLGET QUESTION 12 ONLY IF THEY SAY THEIR CLINIC OR THEIR AGENY TRACKS A NO-SHOW RATE.

IF CLINIC/AGENCY TRACKS A NO-SHOW RATE: BEFORE YOU READ QUESTION 12, LET ME JUST ASK YOU WHAT YOUR NO-SHOW RATE IS IN YOUR OWN WORDS.

IF CLINIC/AGENCY DOES NOT TRACK A NO-SHOW RATE: BEFORE YOU READ QUESTION 12, LET ME JUST ASK YOU WHAT YOU THINK A PERSON WOULD SAY THEIR RATE IS IF THEIR CLINIC TRACKS ONE? YOU CAN MAKE UP A RATE.

NOW LET’S CONTINUE TO QUESTION 12. PLEASE READ IT OUT LOUD AND VERBALIZE YOUR THOUGHT PROCESS AS YOU HAVE BEEN DOING.

12. What percentage of child recertification appointments (ages 1 to 4 years only) were no-shows at your clinic last month? Please enter the percentage below.

| | | | percent

(valid range 000-100)

d □ Don’t know or unable to report

IF THE NUMBER REPORTED IN Q12 IS GREATER THAN 40, THEN A SOFT CHECK WILL BE DISPLAYED: “You entered that [Q12 FILL] percent of child recertification appointments were no-shows. Please confirm or correct your response and continue.”

[HOVER DEFINITION OF “child recertification” WHEN RESPONDENT HOVERS THE MOUSE OVER TERM.]

Child recertification appointments are for children ages 1 through 4 years only.

IF R’S OPEN-ENDED RESPONSE WAS A PERCENTAGE: As you can see, the question in the survey asks for the rate as a percentage. I noticed you also reported it as a percentage. Do you think a perentage is the best format for answers to this question? For example, what if the question asked for the number of children who missed appointments last month and the total number of children who had appointments last month?

IF R’S OPEN-ENDED RESPONSE WAS NOT A PERCENTAGE: As you can see, the question in the survey asks for the rate as a percentage. I noticed you reported it differently from a percentage. What do you think is the best format for answers to this question? For example, what if the question asked for the number of children who missed appointments last month and the total number of children who had appointments last month?

Do you think many respondents will be able to report no-shows just for children ages 1 to 4? If not, what types of WIC participants do you think will be included in no-shows?

Do you think many respondents will be able to report no-shows for last month? Do you think using a different time period would be better to ask? Why is that?

Was the information for this question readily available, why or why not? [If needed: Who at your agency would know this information?]

How much time did it take you, or would it take a clinic staff member, to access the information needed to answer this question? Please walk me through the steps in that process.

REASONS CHILDREN LEAVE THE WIC PROGRAM

These questions are about the reasons children (ages 1 through 4 years) leave the WIC program.

13. Some child participants leave the WIC program shortly after their first birthday. In your opinion, what are the top 3 reasons child participants leave the WIC program at age 1?

select up to 3 responses

1 □ Embarrassed to participate in WIC

2 □ Too difficult to be physically present for appointments

3 □ Believed they were no longer eligible due to the child’s age

4 □ Believed they were no longer eligible due to household income

5 □ Felt they didn’t need WIC any more

6 □ Felt that needs were met by participation in other programs (for example, Head Start, CACFP, or SNAP)

7 □ Food package no longer includes the foods and beverages participants want

8 □ Negative experiences with the clinic or clinic staff

9 □ Negative experiences redeeming food benefits in stores

10 □ Other (specify)

IF Q13_10 OTHER SPECIFY IS CHECKED AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Let’s take a look at the first answer choice, “embarrassed to participate in WIC.” Some people refer to this as “stigma.” We are wondering what you think about using the term “stigma” in the survey—do you have a preference for the current answer choice or stigma? Which one do you think will be easier for the survey respondents to understand?

What do you think about the wording of the response options, are any unclear or difficult to interpret? Why or why not?

What do you think would happen if we asked people completing the survey to mark yes or no to each item on the list?

How do you choose the “top 3 reasons”?

Are there other answer choices that you think should be included?

14. Some child participants leave the WIC program between ages 2 to 4. In your opinion, what are the top three reasons child participants leave the WIC program at ages 2 to 4?

select up to 3 responses

1 □ Embarrassed to participate in WIC

2 □ Too difficult to be physically present for appointments

3 □ Believed they were no longer eligible due to the child’s age

4 □ Believed they were no longer eligible due to household income

5 □ Felt they didn’t need WIC any more

6 □ Felt that needs were met by participation in other programs (for example, Head Start, CACFP, or SNAP)

7 □ Food package no longer includes the foods and beverages participants want

8 □ Negative experiences with the clinic or clinic staff

9 □ Negative experiences redeeming food benefits in stores

10 □ Other (specify)

IF Q14_10 OTHER SPECIFY IS CHECKED AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Do you think people completing the survey will notice that this question is asking about ages 2 to 4 whereas the first question in this section asked about children at age 1? Do you think we can do anything in the web survey to make this distinction clearer?

Do you think there are different reasons why children leave the program at age 1 versus age 2-4? What are those reasons?

READ TO RESPONDENT: PEOPLE COMPLETING THIS SURVEY WILL GET QUESTION 15 ONLY IF THEY SAID CHILDREN LEAVE THE WIC PROGRAM IF IT IS “TOO DIFFICULT TO BE PHYSICALLY PRESENT FOR APPOINTMENTS.” LET’S TAKE A LOOK AT QUESTION 15 REGARDLESS OF YOUR EARLIER ANSWERS.

15. You indicated that one of the reasons children leave the WIC program is difficulty being physically present for appointments. Which of the following factors make it difficult for participants to be physically present for appointments at your site?

	SELECT ONE RESPONSE PER ROW
	YES	NO
a. Difficulty taking time off of work or school	1 □	0 □
b. Lack of transportation	1 □	0 □
c. Difficulty arranging child care	1 □	0 □
d. WIC site does not have toys or a designated play area to help occupy children	1 □	0 □
e. Inconvenient clinic hours	1 □	0 □
f. Inconvenient clinic location	1 □	0 □
g. Other (specify)	1 □	0 □

IF Q15G OTHER SPECIFY IS YES AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Are there other answer choices that you think should be included on the list for this question?

READ TO RESPONDENT: PEOPLE COMPLETING THIS SURVEY WILL GET QUESTION 16 ONLY IF THEY SAID CHILDREN LEAVE THE WIC PROGRAM BECAUSE OF “NEGATIVE EXERIENCES WITH THE CLINIC OR CLINIC STAFF” LET’S TAKE A LOOK AT QUESTION 16 REGARDLESS OF YOUR EARLIER ANSWERS.

16. You indicated that one of the reasons children leave the WIC program is negative experiences with the clinic or clinic staff. Which of the following factors make participants have negative experiences with the clinic or clinic staff at your site?

	SELECT ONE RESPONSE PER ROW
	YES	NO
a. It takes a long time to get an appointment	1 □	0 □
b. Long appointments	1 □	0 □
c. Difficulty contacting WIC staff to schedule an appointment	1 □	0 □
d. Language barrier	1 □	0 □
e. Did not feel they were treated respectfully	1 □	0 □
f. Other (specify)	1 □	0 □

IF Q16F OTHER SPECIFY IS YES AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Are there other answer choices that you think should be included on the list for this question?

RETAINING CHILDREN IN THE WIC PROGRAM

These questions are about the practices your site uses to retain children (ages 1 through 4 years) in the WIC program, and the effectiveness of child retention practices. We are interested in your opinions. There are no right or wrong answers.

What does the phrase “use to retain children” mean to you?

17a. Which of the following clinic services and environment practices do you think is or would be effective for retaining children in the WIC program?

CLINIC SERVICES AND ENVIRONMENT

MARK ALL THAT APPLY

1 □ Provide transportation services

2 □ Offer a WIC mobile clinic

3 □ Make the clinic child-friendly (for example, offering toys or play area)

4 □ Co-locate the WIC clinic with other medical or social service providers

5 □ None of these

Are there other answer choices that should be included on this list?

How do you think the “none of these” response option affects your ability to answer this question?

How do you define “effective”?

17b. Which of the following clinic services and environment practices does your site currently use to retain children in the WIC program? If you work at more than one clinic site, please answer for the site where you work most of the time.

CLINIC SERVICES AND ENVIRONMENT

MARK ALL THAT APPLY

1 □ Provide transportation services

2 □ Offer a WIC mobile clinic

3 □ Make the clinic child-friendly (for example, offering toys or play area)

4 □ Co-locate the WIC clinic with other medical or social service providers

5 □ None of these

Do you understand the instructions in italics for this question?

18a. Which of the following operational practices do you think is or would be effective for retaining children in the WIC program?

OPERATIONAL PRACTICES

MARK ALL THAT APPLY

1 □ Offer nutrition education online (for example, website, Skype, etc.)

2 □ Offer online recertification

3 □ Reload electronic benefits remotely so that participant does not need to visit the clinic to reload the card

4 □ Offer a 1-year certification period rather than a 6-month certification period

5 □ None of these

Are there other answer choices that should be included on this list?

18b. Which of the following operational practices does your site currently use to retain children in the WIC program? If you work at more than one clinic site, please answer for the site where you work most of the time.

OPERATIONAL PRACTICES

MARK ALL THAT APPLY

1 □ Offer nutrition education online (for example, website, Skype, etc.)

2 □ Offer online recertification

3 □ Reload electronic benefits remotely so that participant does not need to visit the clinic to reload the card

4 □ Offer a 1-year certification period rather than a 6-month certification period

5 □ None of these

19a. Which of the following scheduling practices do you think is or would be effective for retaining children in the WIC program?

SCHEDULING PRACTICES

MARK ALL THAT APPLY

1 □ Offer extended clinic hours (for example, early mornings, evenings, or weekends)

2 □ Minimize the required number of visits

3 □ Minimize the length of visits

4 □ Use text messaging or other technology for appointment scheduling, rescheduling, or reminders

5 □ None of these

Are there other answer choices that should be included on this list?

19b. Which of the following scheduling practices does your site currently use to retain children in the WIC program? If you work at more than one clinic site, please answer for the site where you work most of the time.

SCHEDULING PRACTICES

MARK ALL THAT APPLY

1 □ Offer extended clinic hours (for example, early mornings, evenings, or weekends)

2 □ Minimize the required number of visits

3 □ Minimize the length of visits

4 □ Use text messaging or other technology for appointment scheduling, rescheduling, or reminders

5 □ None of these

20a. Which of the following customer service practices do you think is or would be effective for retaining children in the WIC program?

CUSTOMER SERVICE

MARK ALL THAT APPLY

1 □ Provide customer service training for staff who interact with participants

2 □ Ensure staff who speak participants’ preferred language are available

3 □ Tailor food packages to meet participant preferences

4 □ Tailor nutrition education to meet participant preferences

5 □ Use creative ways to connect with participants (for example, sending birthday cards for children)

6 □ Engage with local WIC food vendors to improve the WIC shopping experience

7 □ None of these

Are there other answer choices that should be included on this list?

20b. Which of the following customer service practices does your site currently use to retain children in the WIC program? If you work at more than one clinic site, please answer for the site where you work most of the time.

CUSTOMER SERVICE

MARK ALL THAT APPLY

1 □ Provide customer service training for staff who interact with participants

2 □ Ensure staff who speak participants’ preferred language are available

3 □ Tailor food packages to meet participant preferences

4 □ Tailor nutrition education to meet participant preferences

5 □ Use creative ways to connect with participants (for example, sending birthday cards for children)

6 □ Engage with local WIC food vendors to improve the WIC shopping experience

7 □ None of these

21. Is there another practice that you think is or would be effective for retaining children in the WIC program?

1 □ Yes

0 □ No

What came to mind when you were thinking about “another practice”?

22. What other practice do you think is or would be effective for retaining children in the WIC program?

STRING 200

IF Q22 IS UNANSWERED, THEN THIS MESSAGE WILL BE DISPLAYED: “Please provide your response and continue.”

If answered, what category do you think your other practice belongs under? Clinic services and environment, operational practices, scheduling practices, or customer service practices? Why or why not?

23. Does your site use [Q22 FILL] to retain children in the WIC program?

1 □ Yes

0 □ No

READ TO RESPONDENT: IN THE WEB SURVEY, QUESTIONS 24 AND 25 WILL DISPLAY ONLY THE ANSWERS THAT THE RESPONDENT MARKED AS EFFECTIVE IN QUESTIONS 17A, 18A, 19A, 20A, 21A OR 22. PLEASE CONSIDER ONLY THE PRACTICES YOU SELECTED ALREADY IN THIS SECTION WHEN YOU ANSWER QUESTIONS 24 AND 25.

24. You selected the following practices as ones that are or would be effective for retaining children in the WIC program. Which three practices do you think are or would be the most effective for retaining children in the WIC program at your site? These may or may not be practices your site already uses.

select up to 3 responses

1 □ Provide transportation services

2 □ Offer a WIC mobile clinic

3 □ Make the clinic child-friendly (for example, offering toys or play area)

4 □ Co-locate the WIC clinic with other medical or social service providers

5 □ Offer nutrition education online (for example, website, Skype, etc.)

6 □ Offer online recertification

7 □ Reload electronic benefits remotely so that participant does not need to visit the clinic to reload the card

8 □ Offer a 1-year certification period rather than a 6-month certification period

9 □ Offer extended clinic hours (for example, early mornings, evenings, or weekends)

10 □ Minimize the required number of visits

11 □ Minimize the required length of visits

12 □ Use text messaging or other technology for appointment scheduling, rescheduling, or reminders

13 □ Provide customer service training for staff who interact with participants

14 □ Ensure staff who speak participants’ preferred language are available

15 □ Tailor food packages to meet participant preferences

16 □ Tailor nutrition education to meet participant preferences

17 □ Use creative ways to connect with participants (for example, sending birthday cards for children)

18 □ Engage with local WIC food vendors to improve the WIC shopping experience

19 □ [Q22 FILL]

In your own words, what do you think this question means?

What do you think of the wording of this question? Do you think it will be easy or difficult for respondents to understand? How so?

What kinds of WIC participants did you consider when you were answering this question?

How did you decide on your top 3 practices that are or would be effective in retaining children in the WIC program from all the practices you selected in the earlier questions?

25. Some practices need a lot of money and resources to implement, while others do not. Given the limited resources available, which three practices do you think are or would be the most cost-effective for retaining children in the WIC program at your site?

select up to 3 responses

1 □ Provide transportation services

2 □ Offer a WIC mobile clinic

3 □ Make the clinic child-friendly (for example, offering toys or play area)

4 □ Co-locate the WIC clinic with other medical or social service providers

5 □ Offer nutrition education online (for example, website, Skype, etc.)

6 □ Offer online recertification

7 □ Reload electronic benefits remotely so that participant does not need to visit the clinic to reload the card

8 □ Offer a 1-year certification period rather than a 6-month certification period

9 □ Offer extended clinic hours (for example, early mornings, evenings, or weekends)

10 □ Minimize the required number of visits

11 □ Minimize the length of visits

12 □ Use text messaging or other technology for appointment scheduling, rescheduling, or reminders

13 □ Provide customer service training for staff who interact with participants

14 □ Ensure staff who speak participants’ preferred language are available

15 □ Tailor food packages to meet participant preferences

16 □ Tailor nutrition education to meet participant preferences

17 □ Use creative ways to connect with participants (for example, sending birthday cards for children)

18 □ Engage with local WIC food vendors to improve the WIC shopping experience

19 □ [Q22 FILL]

In your own words, please tell me what you think this question is asking. How could we word the instructions differently to be clearer?

How do you define “cost-effective”?

How did you decide your ranking?

26. What other changes to WIC requirements at the local, State, or Federal level would help retain children in the WIC program? Please suggest up to 3 changes.

1. STRING 200

2. STRING 200

3. STRING 200

In your own words, please tell me what you think this question is asking.

How did you decide on which changes to include in your answer? Do you happen to know which changes are ones that USDA would need to make versus ones that your State WIC Agency could make on its own?

How do you define “WIC requirements”?

CLINIC CHARACTERISTICS

These questions are about your WIC clinic site. If you work at more than one clinic site, please answer for the site where you work most of the time.

27. When is your clinic site open?

	SELECT ONE RESPONSE PER ROW
	YES	NO
a. 5 or more days per week	1 □	0 □
b. During the weekend at least once per month	1 □	0 □
c. Early morning hours at least 1 day per week	1 □	0 □
d. Evening hours at least 1 day per week	1 □	0 □

IF Q19B_1 IS CHECKED AND Q27B, Q27C, AND Q27D ARE ALL NO OR UNANSWERED, THEN A SOFT CHECK WILL BE DISPLAYED: “Earlier you indicated your clinic offers extended hours but you did not select any of those options for this question. Please review your responses and continue.”

Do the intervals make sense for when your clinic is open? If not, how could we make this clearer?

28. After participants arrive at your clinic for a child recertification appointment, about how long do they wait, on average, before the appointment begins? Your best guess is fine.

1 □ Less than 15 minutes

2 □ 15 to 30 minutes

3 □ 31 to 60 minutes

4 □ More than 60 minutes

What do you think about these answer choices, do they accurately represent wait times? Why or why not?

What came to mind as you though about how long a participant waits before the appointment begins?

29. Not counting the time waiting for the appointment to begin, about how long does an average recertification appointment for one child take, starting from the beginning of the appointment to completion of benefit issuance? Your best guess is fine.

1 □ Less than 30 minutes

2 □ 30 to 59 minutes

3 □ 1 hour to under 1.5 hours

4 □ 1.5 hours to under 2 hours

5 □ 2 hours or more

What do you think about these answer choices, do they accurately represent appointment lengths? Why or why not?

What came to mind as you though about what is included from the start of the appointment to the completion of benefit issuance? Are there any aspects of child recertification visits that you are unsure about counting as either wait time in question 28 or appointment time in question 29?

30. In what type of facility is your clinic site located?

1 □ Health department

2 □ Other health-related facility, such as a hospital, Indian Health Service facility, Federally Qualified Health Center, or nonprofit health faciitly

3 □ Nonprofit facility, such as a community service agency, school or Head Start site, or faith-based agency

4 □ Stand alone WIC site

5 □ Other (specify)

IF Q30_5 OTHER SPECIFY IS CHECKED AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

Are there other types of facilities that should be on this list?

How easy or difficult was it to understand the descriptions of these facility types? Do you recommend using different terms?

31. What is the total number of WIC participants served per month at your clinic site? Your best guess is fine.

1 □ 300 or fewer participants

2 □ 301 to 900 participants

3 □ 901 to 2,500 participants

4 □ More than 2,500 participants

What came to mind as you thought about the total number of WIC participants that you serve per month at your clinic?

How easy or difficult was it for you to answer this question? Do you think survey respondents will be able to answer this off the top of their heads? Who would they need to ask if they don’t know the answer to this question on their own?

32. Compared to the number of WIC participants served at your clinic site, how would you describe the staffing levels?

1 □ Far too few staff for the number of participants

2 □ Too few staff for the number of participants

3 □ About the right number of staff for the number of participants

4 □ Too many staff for the number of participants

5 □ Far too many staff for the number of participants

What came to mind as you thought about which answer choice to select?

YOUR WIC EXPERIENCE

33. What are your primary responsibilities in your WIC agency? If you work at more than one clinic site, please answer for the site where you work most of the time.

MARK all that apply

1 □ Caseload management

2 □ Child certification

3 □ Following up with missed certification or recertification appointments

4 □ Nutrition education

5 □ Breastfeeding support

6 □ Food benefit issuance

7 □ Measure height and weight

8 □ Determine income eligibility

9 □ Determine nutrition risk

10 □ Anemia screening

11 □ Other (specify)

IF Q33_11 IS CHECKED AND OTHER SPECIFY IS BLANK, THEN A SOFT CHECK WILL BE DISPLAYED: “You selected ‘other’ but left the ‘specify’ field unanswered. Please provide your response and continue.”

What do you think about the response options for responsibilities at your WIC agency? Are there any that we should include to make this question more inclusive?

34. How many years have you worked at this local WIC agency?

| | | number of years

(valid range 00-50)

IF THE RESPONSE TO Q34 IS GREATER THAN 20, THEN A SOFT CHECK WILL BE DISPLAYED: “You entered that you have worked in a clinic at this local WIC agency for [Q34 FILL] years. Please confirm or correct your response and continue.”

35. On average, how many children (ages 1 through 4) do you personally certify or recertify per week?

| | | average number of weekly child certifications and recertifications

(valid range 00-99)

IF THE RESPONSE TO Q35 GREATER THAN 25, THEN A SOFT CHECK WILL BE DISPLAYED: “You entered that you personally certify or recertify [Q35 FILL] children per week. Please confirm or correct your response and continue.”

How did you come up with your answer?

36. Please share any additional comments you have about child recertification appointments, reasons children leave the WIC program, or strategies for retaining children in the WIC program.

STRING 200

Thank you for completing this survey! If you have any questions about the survey, please contact 2M Research at 1-877-440-0050 from 9:00 AM to 4:00 PM EDT Monday through Friday, or by email at usdawicretention@2mresearch.com.