6 Results and Recommendations

Attachment B6

Healthy Start

Pilot Survey Results and Recommendations

HRSA’s Healthy Start Evaluation and Capacity Building Support Project

October 2022

Pilot Test Results and Recommendations

Background

Purpose. Pilot testing was conducted on four data collection instruments that Westat, the evaluation contractor, developed in collaboration with the Health Resources and Services Administration’s (HRSA) Maternal and Child Health Bureau (MCHB) for the Healthy Start evaluation. These instruments include the Program Survey, Network Survey, Participant Survey, and Stakeholder Interview Guide. The purpose of the pilot test was to ensure the appropriateness of the questions and the usability of each instrument within the target audience.

Sample and Recruitment. Nine Healthy Start program directors were identified and contacted with the help of HRSA Project Officers. These directors were requested to be respondents for the pilot testing of the Program Survey. The director of one Healthy Start program was requested to identify nine members from their Community Action Network (CAN) for the Network Survey and another was requested to identify nine participants for the Participant Survey. Other program directors were requested to identify program staff and CAN members for testing the Stakeholder Interview Guide with a total of nine interviewees.

Implementation. The three surveys were programmed in SurveyMonkey Premier for web-based administration with respondents from Healthy Start programs. The pilot testing of all the instruments occurred between January 10 and March 8, 2022. The testing was conducted by Westat research staff during one-on-one interviews via Microsoft Teams or Zoom at a time convenient to the respondent. For the web surveys, the Westat interviewer first described the purpose of the testing using a script and obtained informed consent to proceed. The web link was then shared with the respondent who was asked to follow the web instructions, complete the survey, and then provide feedback on the questions and survey experience. The interviewer took notes on items skipped or those on which the respondent had questions or took unusually long. After the respondent finished the survey, the interviewer engaged in a discussion with the respondent and asked questions using a semi-structured discussion guide. For the Stakeholder Interview Guide, the interviewer similarly described the purpose of the interview and proceeded with the informed consent and the questions on the guide while another team member took notes. All team members entered their notes in an Excel file and met internally to discuss findings and suggest changes to the instrument.

Results

Time to Complete the Instrument

Program Survey. Out of the nine respondents who participated, the median time to complete the Program Survey was 30 minutes, ranging from 23 to 44 minutes. Half the respondents thought the length was fine. The others were concerned about the time it would take to find the data that they would need. Two respondents indicated that they would not be able to complete the survey on their own. Most thought it would be good to have a copy of the survey ahead of time so they could gather information before starting the web survey.

Participant Survey. Out of the nine respondents who participated, the median time to complete the Participant Survey was 14 minutes, and the time ranged from 8 and 16 minutes. The majority of respondents felt that the length of the survey was appropriate.

Network Survey. Out of the nine respondents who participated, the median time to complete the Network Survey was 23 minutes, and the completion time ranged from 16 to 45 minutes. Only one respondent took 45 minutes to complete the survey, and was therefore an outlier. This respondent experienced technical difficulties, requiring the Westat team member to navigate through the screen while the respondent verbally selected answers. During this time, the respondent also paused to ask questions and make comments. By excluding this respondent’s time, the median time to complete the survey was 22 minutes, with a range of 16 minutes to 26 minutes. The majority of respondents indicated that the time it took to complete the survey was reasonable, and one respondent expressed that the shorter length allowed them to remain focused on each question. Two respondents felt that the survey length was long. One respondent, who completed the survey in 18 minutes, noted that the survey was long for not receiving an incentive. The other respondent, who completed the survey in 26 minutes, mentioned that it will be important to manage expectations for how long the survey would take.

Stakeholder Interview Guide. Out of the nine respondents who participated, the median time to complete the Stakeholder Interview Guide was 45 minutes, and the time ranged from 30 to 80 minutes. The majority of the respondents were able to respond to all questions, though thorough responses took more time than expected.

General Recommended Changes

Westat recommend increasing the character limit to 200 for all the “Other (Specify)” options, and indicating below the box that the maximum characters permitted is 200. We also recommend formatting tables that are long to freeze the top row to ease scrolling up and down. In addition, we recommend adding the instruction in some questions: Select “Yes” or “No” per row or Select one per row so respondents do not skip questions.

Additionally, Westat recommend emphasizing the estimated survey duration in the Informed Consent portion of the survey by bolding the time, and recommend providing this information in a Frequently Asked Questions (FAQs) document that is sent to respondents with the survey invitation.

Finally, Westat recommend providing explicit definitions of the CAN and Healthy Start, when applicable, in the introduction of the survey, and recommended to have respondents acknowledge that they have read this information before proceeding.

Survey-Specific Recommended Changes

Recommendations based on pilot test feedback for specific surveys by question are summarized in Tables 1 through 4. We have made additional recommendations for the surveys based on feedback from HRSA staff.

Table 1. Program Survey Results and Recommendations, by Question

Program Survey Question Number and Original Text	Summary of Feedback	Recommendation
2. In which types of organizations are your Heathy Start sites located? Select all that apply.	A respondent was unclear about the term “site” and some suggested adding additional response options (e.g., university).	Change to: In which types of organizations does your Healthy Start program operate? Add as response option: University
Do you currently have formal agreements with the organizations where the Healthy Start sites are located? A formal partnership can be defined as a written agreement (usually involving a subcontract or memorandum of understanding (MOU )). Yes/No	A respondent was unsure of any agreement.	Delete question
3. During GRANT YEAR 2020-2021, what was the approximate average caseload for each full-time equivalent case manager/ care coordinator/ community health worker?	A few respondents preferred a range of numbers rather than open text. HRSA suggested using calendar year rather than grant year throughout the survey.	Change to: During CALENDAR YEAR 2022... and from open text to response options: Less than 25; 25 to 39; 40 to 54; 55 to 64; 65 to 74; 75 or more
5. What criteria do you use for defining high-risk? Indicate which criterion defines high-risk by itself and which define high-risk in combination with other criteria. Check all that apply. f. Domestic violence against women and/or her children m. Woman is a single parent with no partner	Some respondents were unsure of criteria/combinations of criteria or had sites that used different criteria. HRSA suggested changing language for option f to “family violence against adults or children” and option m to “single parent with no partner.”	Change to: What criteria do you use for defining high-risk? Remove columns: “Alone” and “In Combination with Other Criteria” and keep one check box for each option. Change options f and m to: f. Violence within the participant’s home; m. Single parent with no partner
6. Do you have a tool that you use to make and document the determination of high-risk? No; Yes, a standardized tool (Specify name of tool); Yes a locally developed tool	Some respondents wanted to check more than one “Yes” response option.	Change response options to: No; Yes (Specify name of tool or tools)
How often does your program conduct assessments for Healthy Start women?	Some respondents were confused because they used the required HS forms.	Delete question because all programs use the HS forms as required.
How often does your program conduct assessments for infant/family assessments?	Some respondents were confused about these assessments and used the HS forms.	Delete question because all programs use the HS forms as required.
9. What outreach strategies did you use to recruit Healthy Start participants during GRANT YEAR 2020-2021? Select all that apply.	Some respondents also used strategies other than those listed.	Add option: Other (Specify)
10. To what extent has recruitment of Healthy Start participants been a problem for your Healthy Start program? Select one only.	Some respondents wanted to explain the problem. HRSA staff suggested using open-text.	After options “Somewhat of a problem” and “A big problem” add “Explain” with an open-text box.
12. Using a scale from 1-5, (1=not at all challenging, 5=very challenging), how much of a challenge is it to retain female participants in your Healthy Start program? Select one response for each row.	Some respondents found it difficult to understand the scale.	Revise question to: Using a scale from 1-4, (1=Not at all a challenge, 4=A significant challenge), how much of a challenge is it to retain participants in your Healthy Start program? Select one response for each row. In the table, define each point in the scale: 1. Not at all a challenge; 2. Somewhat of a challenge; 3. A moderate challenge; 4. A significant challenge.
13. In GRANT YEAR 2020-2021, did your program provide home visits to your Healthy Start clients? Yes. If yes, what home visiting model do you use? (Specify); No	Some respondents noted that during the pandemic, they provided virtual visits which they considered home visits.	Change to: In CALENDAR YEAR 2022, did your program provide home visits to your Healthy Start participants? Check all that apply. Yes, in person visit; Yes, virtual visit; No
In GRANT YEAR 2020-2021, did your program provide telehealth visits to your Healthy Start clients? Yes; No		Delete question as it is covered in revised #13 above.
14. Do you track the use of clinical preventive services by all (preconception, pregnant, and postpartum) women who participate in your Healthy Start program?	The term “tracking” was unclear.	Change question to: Do you keep a record of the clinical preventive services that Healthy Start participants receive?
15. In GRANT YEAR 2020-2021, approximately how often do you make referrals for the following specific services/ support? Select one per row. j. Mental health services l. Reproductive health services	Some respondents were unclear if referrals meant outside the HS program. They also wanted examples for mental health and reproductive health services in the response options provided.	Change to: In CALENDAR YEAR 2022, approximately how often did you make outside referrals for the following specific services/support? Select one per row. Change following options to: j. Mental health services for anxiety and depression; k. Other mental health services; m. Family planning services
16. In GRANT YEAR 2020-2021, how many preconception, pregnant or post-partum women did your program refer to mental/behavioral health services for diagnosis or treatment for depression or anxiety?	Some respondents were unclear if referrals meant outside the HS program and wanted the text box to allow for numbers more than 200. HRSA staff suggested using inclusive language in the survey (e.g., pregnant people rather than pregnant women).	Change question: to In CALENDAR YEAR 2022, how many preconception, pregnant or post-partum people did your program refer to mental health services for diagnosis or treatment for depression or anxiety outside Healthy Start? Programming: Increase number limit to 300 in the text box.
17. Do you track and document whether Healthy Start participants receive the services to which your program refers them? Select one response per row.		To be consistent with other questions, revise to: Do you track and document whether Healthy Start participants receive outside services to which your program refers them? Select “Yes” or “No” for each service/support.
26. To what extent has the CAN made an impact on each of the following areas of community improvement? Select one per row.	Some respondents felt unsure if the CAN made an impact in some areas.	Add column with option: I don’t know
27. What types of activities does the CAN conduct and how often are they held? Select one per row.	Some CANs meet bimonthly. CANs were unable to conduct in-person activities during the pandemic.	Change question to: What types of activities did the CAN conduct in CALENDAR YEAR 2022 and how often were they held? Select one per row. Add column: Every Other Month
During GRANT YEAR 2020-2021, how many other local and state activities specifically focused on women’s health did you participate in GRANT YEAR 2020-2021?	Some respondents thought it was unclear if the question was referring to the CANs.	Delete question because covered in revised #31 (In addition to your CAN-specific activities, in what other state or local activities/committees did your program participate in CALENDAR YEAR 2022? Select all that apply.)
29. The following statements could be about your Healthy Start program’s experience with, and observations about, your CAN. Please indicate your agreement with the following statements on a scale from “Strongly Disagree” to “Strongly Agree.” Select one per row. a. The organizations involved in the CAN are the “right” organizations to be involved.	For this question in the Network Survey, HRSA staff suggested the respondent may think the current organizations are the right organizations. The option does not provide information on organizations that were not included.	Change option and move to i: The CAN includes representatives from all of the service areas that Healthy Start participants need. If respondent checks i (strongly disagree or disagree), add question #30: In the previous question, you indicated that the CAN does not include all organizations that are needed to provide services to the Healthy Start community. Select the types of services that these organizations would provide if they were included in the CAN. (CHECK ALL THAT APPLY) [15 options]
31. In addition to your CAN-specific activities, in what other type of related activities/ committees did your program participate in GRANT YEAR 2020-2021? Select all that apply.		Revise question so that it covers the deleted question on state and local activities: In addition to your CAN-specific activities, in what other state or local activities/committees did your program participate in CALENDAR YEAR 2022? Select all that apply.
33. Are women in your Healthy Start asked if they would like their male partners/fathers of their infants to be involved in the program?	Healthy Start participants can qualify for the program if they have children up through the age of 18 months, not just infants.	Change question to: Are women enrolled in your Healthy Start program asked if they would like their male partners/fathers of their children to be involved in the program?
34. What fatherhood curriculum does your Healthy Start program use?	Some programs used more than one curriculum.	Add after question: Check all that apply.
35. Does your program have a case manager for the fathers/male partners?	If programs had not filled the position, this question was difficult to answer.	Change question and options to: Does your program have a funded case manager position for the fathers/male partners? Yes and the position is filled; Yes but the position is vacant; No
37. Does your program cover the following topics with Healthy Start fathers/male partners? c. Self-feelings about being a father	HRSA staff noted that option c was not a Healthy Start topic.	Delete option c.
Do you refer male Healthy Start participants to any of the following services?	This question is addressed for all participants, including fathers, in an earlier section.	Delete question.
38. To what extent has retention of fathers/male partners in your Healthy Start program been a problem? Select one only.	A respondent felt that father involvement rather than retention was the issue.	Change question to: To what extent has ongoing involvement of fathers/male partners in your Healthy Start program been a problem? Select one only.
39. What strategies does your Healthy Start program use to facilitate father/male partner retention? Select all that apply. _ Frequent contact and follow-up with men		To be consistent with the previous question, replace “retention” with “involvement” and use “fathers/male partners” rather than men in the first option: In CALENDAR YEAR 2022, what strategies did your Healthy Start program use to facilitate ongoing involvement of fathers/male partners in the program? Select all that apply. _ Frequent contact and follow-up with the fathers/male partners
40. What data elements are included in your data system(s)? Select one “Yes” or “No” response per row. b. Case management/ care coordination/community health worker encounters (in-person, telephone, electronic) [except home visits] c. Home visits	Some respondents thought that some options were confusing particularly because home visits were replaced by virtual visits during the COVID-19 pandemic.	Separate options for virtual and in-person visits: b. Case management/care coordination/community health worker encounters that were by telephone or virtual c. Case management/care coordination/community health worker encounters that were in-person
Below is a definition of health equity. Is there anything that you would add or change to this definition?	HRSA staff noted that quotes in the definition were confusing.	Delete question and provide definition of health equity in plain language.
43. Using a scale from 1-5, (1=least likely cause, 5=most likely cause), to what extent do you think the following contribute to inequities in maternal and child health in your community?	A respondent found the scale a bit confusing and the option “racism” was very broad.	Change question and scale to: Using a scale from 1-4, (1=Not at all a contributor, 4=A significant contributor), to what extent do you think the following contribute to inequities in maternal and child health in your community? Select one per row. In the table, define each point: 1.Not at all a contributor; 2. Somewhat of a contributor; 3. A moderate contributor; 4. A significant contributor Define racism by adding examples: Racism (overt, perceived, structural/systemic, cultural, educational, employment, etc.)
55. Healthy Start staff who participated in the completion of the survey (Check all that apply)	A responded was unclear whether the question was asking who entered the responses or who contributed to the survey.	Change to: Healthy Start staff who contributed to complete the survey (Check all that apply)
	At the end of the survey, HRSA staff suggested not using “hit” due to its violent connotation.	Change to: Please press send to submit your survey.
	Some respondents suggested adding questions about COVID-19 and additional funding that impacted their programs. HRSA staff also suggested including questions about the impact of COVID-19 on programs.	Add 5 additional questions in new Section VIII. Other Questions: 50. Did you receive any supplemental funds from HRSA? Check all that apply. Infant health equity plans; Doula services; Clinical providers 51. Describe how you used these funds in your program? ___. 52. How did the COVID pandemic affect your program operations? Check all that apply. We added COVID to the topics we cover with Healthy Start participants; We changed in-person meetings to virtual meetings and/or telephone calls; We changed home visits to virtual meetings and/or telephone calls; We had to change our outreach strategies; We diverted some Healthy Start funds to COVID activities such as vaccinations, testing and contact tracing; We diverted Healthy Start staff to COVID activities such as vaccinations, testing and contact tracing; Other (Specify)__. 53. How did the COVID pandemic affect your new enrollment numbers and involvement of Healthy Start participants in the program? Check all that apply. The pandemic did not affect our new enrollment numbers; The pandemic did not affect participation in our Healthy Start program; Our new enrollment numbers went down; Participation in Healthy Start went down; Other (Specify)__. 54. How did the COVID pandemic affect your Healthy Start CAN? Check all that apply. The pandemic did not affect our Healthy Start CAN; We changed our in-person events to virtual meetings; We reduced the number of meetings and/or events that we held; Other (Specify).

Table 2. Participant Survey Results and Recommendations, by Question

Participant Survey Question Number and Original Text	Summary of Feedback	Recommendation
Directions	HRSA staff suggested rephrasing “You may skip questions that don’t apply to you.”	Change to: If you have been in Healthy Start before, please answer just about your experience for this enrollment period (April 2019 and after).
General – Scaled Questions	HRSA suggested removing the neutral option in questions with Likert scales.	Across all applicable question and respondents, the neutral option was only used once as the respondent had a truly mixed experience that resulted in neutrality. We recommend leaving the neutral option as it allow for an unbiased question/ response when the respondent is actually neutral on the subject.
3. How long have you been in the Healthy Start program? Select one only. [Those less than 6 months are screened out]	Respondents thought that starting the inclusion criteria at 6 months may omit too many participants.	Do not exclude any respondents.
5. When you signed up for Healthy Start during this enrollment period, were you? Select one only.	Several male respondents misread the options to this questions. HRSA suggested using inclusive language throughout the survey.	Change response option to: Non pregnant; Pregnant person; Postpartum- Not pregnant, but gave birth in the 3 months before I enrolled in Healthy Start; Postpartum – Not pregnant, but gave birth 3 months to 1 year before I enrolled in Healthy Start; A partner to a woman enrolled in Healthy Start or a father to child enrolled in Healthy Start; Other (Please Explain)
Did you get interpreter or translation services in your primary language from Healthy Start?	HRSA suggested that the term “Primary language” might not be clear to all respondents.	Move to section on additional information after language spoken and change to: 49. Did you get interpreter or translation services from Healthy Start? Yes; No, but I needed it; No, I bring a family member or friend to interpret for me; I don’t need these services because I’m fluent in English
7. Was the information and education that you received from Healthy Start helpful?	One respondent answered “No”, but there was nowhere for them to explain why.	Update the response option to: Yes; No. If no, please explain _____
13/17. If you didn’t apply for WIC/Medicaid, why not? (Check all that apply)	HRSA suggested adding skip patterns so that only people who did not apply for Medicaid should be answering this question.	Change to: Why didn’t you apply for WIC/Medicaid? (Check all that apply). Skip patterns were added.
18.a. Did Healthy Start staff refer you to any of the following services?	Several respondent expressed confusion when reading this question.	Rephrase question to: Did Healthy Start staff refer you to any of the following services outside Healthy Start?
19.a. Did Healthy Start staff refer you to any of the following services or programs?	Several respondent expressed confusion when reading this question.	Rephrase question to: Did Healthy Start staff refer you to any of the following services or programs outside Healthy Start?
26. What Healthy Start father/partner activities have you participated in? Check all that apply. Going to partner’s appointments; Going to child’s appointments; Case management; Support group(s); Health education sessions; Presentations; Community Action Network (CAN) meetings; CAN and other community events; Other (Specify) ____	Several respondent expressed confusion when reading this question and the response options.	Change question and options to: What activities have you participated in as a father/partner? Check all that apply. Going to partner’s healthcare appointments; Going to child’s healthcare appointments; Healthy Start case management; Healthy Start support group(s); Healthy Start health education sessions; Healthy Start presentations; Healthy Start Community Action Network (CAN) meetings; Healthy Start community
27.a. Did Healthy Start staff provide information on the following activities? Select “Yes” or “No” per row? c. How I feel about being a father	HRSA staff did not think option c made sense for the evaluation and suggested adding the option “Relationship building with the mother.”	Change option: option c was removed and the option “Relationship building with the mother or co-parent” was added.
27.b. Was the information provided by the Healthy Start staff on these activities helpful?	Respondent were unsure how to answer this question when they received information that did not apply to their situation.	Add a response option column: Not applicable.
29.a. Did Healthy Start staff refer and help you get any of the following services?	Respondents thought there were two question and were unsure how to answer both.	Remove “and help you get” from the question.
There are differences in pregnancy results for both the mother and the newborn among different racial and ethnic groups. For example, Black women have some of the highest rates of pregnancy-related illness and deaths, and infants have high rates of prematurity and deaths, compared to other groups. What do you think is causing these differences?	Several respondent found this question either to too general phrased, leading, or hard to answer. This question was skipped by three respondents as they did not feel they had the knowledge to answer.	Delete question
42. Please describe up to three experiences you have had with racism when getting healthcare.	Some respondents wanted context for this question. One respondent refused to answer under any circumstance as they did not want to relive the experiences.	Change to: In what ways has Healthy Start made it easier for you to receive healthcare services (such as prenatal care) or social services (such as housing or job assistance)? Please briefly describe up to three ways that Healthy Start has promoted health equity for you personally, or ways that the program has helped remove barriers to receiving healthcare or social services.
43. What more do you think Healthy Start can do to reduce the racial differences in poor pregnancy results in your community	This question was skipped by three respondents as they did not feel they had the knowledge to answer.	Change to: What more do you think Healthy Start can do to reduce the bias in health services and social services in your community?
	HRSA staff suggested including questions to this survey about the impact of COVID-19 on participants.	Add questions: 53. How difficult has it been to continue participating in your Healthy Start program? 54. How did the COVID pandemic affect your participation in Healthy Start and other services? Check all that apply. 55. How satisfied were you with the telehealth/virtual care you received during the COVID pandemic? Select one. 56. What were your greatest challenges during the pandemic?

Table 3. Network Survey Results and Recommendations, by Question

Network Survey Question Number and Original Text	Summary of Feedback	Recommendation
Not Applicable	Some respondents were involved in multiple organizations or programs within an organization, and they were unclear about which organization they were representing when completing the survey. CANs from the Virtual Networking Café also provided clarification on who could be considered as members (both individuals and organizations).	Add new questions (questions 1, 2.a., 2.b., 2.c.) at the beginning of the survey to confirm who the respondent represents in their CAN, and to prompt them to respond to survey questions from this perspective: 1. Do you serve on the Healthy Start CAN ([CAN NAME]) as an individual or as part of an organization? 2.a. [IF Q1 = I represent an organization] Which organization do you represent in the CAN? 2.b. Within your organization, do you represent a specific program, such as WIC, or other entity? 2.c. Within your organization, what is the name of the program, such as WIC, or other entity that you represent?
3.a. and 3.b. Which of the following best describes the organizations/groups that you represent?	Some respondents were involved in multiple organizations or programs within an organization, and they were unclear about which organization they were representing when completing the survey. CANs from the Virtual Networking Café also provided clarification on who could be considered as members (both individuals and organizations).	Clarify that the question is asking about representation in the CAN, and split up this question based on response options that are applicable if someone is representing themselves versus response options that are applicable if they are representing an organization. Remove all references to “group” throughout the survey since this is no longer applicable.
4. Which kinds of services does your organization/group provide?	Some respondents were not sure how to answer this question because their organizations did not provide direct services. Instead, they helped other organizations by providing referrals. Some respondents also wanted clarification on whether the question was asking about services for all clients or to Healthy Start participants specifically. One respondent asked whether “oral health” should be categorized as “other.”	Ask this question only if someone is representing an organization. Clarify that this question is asking about general services to all clients, and referrals should not be included. Add “oral health” as a response option. Revise the “domestic violence,” “employment,” and “substance abuse” categories for clarity.
5. and 6. How long have you and your organization/group been a member of the Healthy Start CAN?	Some respondents found this question to be confusing, and they were not sure how long they have been a member of the CAN versus their organization. One respondent was also not sure about her membership since they were more involved with Healthy Start in the past and were currently not as involved.	Separate this question to ask how long each respondent has been a member of the CAN, and if they are representing an organization, then ask how long their organization has been a member of the CAN. Include “Don’t know” in the question about their organization’s membership.
7. What roles have you and your organization/group had with the Healthy Start CAN?	One respondent did not think the meaning of “leadership” was clear, and another respondent was not sure what is considered as an organization or group’s role. CANs from the Virtual Networking Café also clarified that CANs are frequently led by chairs or co-chairs and organized by committees or workgroups, and some also have leadership/planning teams.	Ask about the respondent’s own role with their CAN. Revise the response options to ask about the following: CAN Chair or Co-Chair, Committee/Workgroup Chair or Co-Chair, Committee/Workgroup Member, Leadership/Planning Team, CAN Member, Other.
8. How would you rate your organization’s/group’s level of involvement with the Healthy Start CAN?	Upon internal review, we found that clarity is needed on which timeframe this question is asking about.	Specify “during the past 12 months” in the question, as later questions ask about this time period.
9.a. From the list below, please select the organizations/groups from the Healthy Start CAN with which your organization/group has collaborated during the past 12 months.	Some respondents asked how the list of organizations was generated and felt that some organizations were misnamed or were not included. One respondent selected their own organization. Some CANs from the Virtual Networking Café also noted that they have more than 40 members.	Allow the full list of CAN members (individuals or organizations) to be pre-populated, and ask respondents to select up to 20 to limit burden. Specify that the question is asking about collaboration on CAN activities, and add a programming note to exclude the respondent’s own organization.
9.b. On a scale from 1 to 5, (1=very little, 5=to a great extent), how much has your organization/group collaborated with each organization/group listed below?	Based on the changes to question 9.a., this question needed to be modified as well for flow and clarity. Upon internal review, we felt that the scale could be improved to ask about collaboration in a more tangible way.	Pre-populate this question with the people/organizations selected in 10.a. (up to 20), and specify that the question is asking about collaboration on CAN activities during the past 12 months. Revise scale to ask respondents to choose from the following: rarely, occasionally, frequently, very frequently.
10.a. and 10.b. Which of the following activities do you think are high priorities for your Healthy Start CAN? Please rank the three highest priorities with “1” being the highest priority. (ENTER TOP 3 RESPONSES ONLY)	Six respondents had trouble ranking the priorities in this question and did not realize that only 3 responses should be selected and ranked from 1-3. One respondent also asked for clarity about which grant one of the response options was referring to.	Split up the question into two parts. The first part will ask respondents to choose the top 3 priorities, and the second part will show just the three selected and will ask them to rank these 3. Specify that the response option is asking about the Healthy Start CAN.
11. The organizations listed below are those that you indicated having worked with over the last 12 months. For each organization/group, check the areas of CAN activities in which you ACTIVELY collaborate with each of them. Check “Not Applicable” for the organization if there has been no active collaboration around these activities. Check “Other” if you collaborated with the organization/group on areas other than the three areas listed.	One respondent asked whether this question is asking about referrals to other organizations for the specified services. While respondents generally did not seem to have difficulty with this question, upon internal review, we believe this question could be clearer that it is asking about collaborations through the CAN, and the categories could be more descriptive.	Specify that this question is asking about active collaborations through the CAN to improve services and outcomes for Maternal and Child Health populations, and revise the categories to the following: Health Care System Issues (e.g., access to culturally and linguistically appropriate comprehensive services that include medical, mental health, substance abuse, and enabling services), Programs and Services that Address Social Determinants of Health (e.g., employment, education/training, food assistance, housing), Promotion of Good Health Outcomes through Collaborations with Other Public Health Initiatives (e.g., Title V, Home Visiting).
13.a. What are the three main current goals of your Healthy Start CAN?	Two respondents did not know their CAN’s goals and were taken to question 14 (What do you think are the top barriers your CAN faces in achieving its goals?), which they felt they could not answer. One CAN from the Virtual Networking Café noted that their CAN has not fully defined their goals yet.	Add question before 3 main goals: 13.a. Are you aware of the current goals of your Healthy Start CAN? Allow respondents to skip question 14 if they do not know their goals or the CAN has not yet identified the goals. For 13.c about effectiveness in meeting goals: Prepopulate goals entered by those who know what the goals are in 13.b.
	HRSA staff suggested adding a question to allow respondents to list the organizations they think should be included in the CAN, based on their response to question 17.a. (The CAN includes all organizations that are needed to provide services to the Healthy Start community.).	Add a follow-up question (17) to ask respondents to name the organizations they think should be included in the CAN, and allow up to 3 responses to limit burden: 17. [IF Q16.i. = Strongly Disagree or Disagree] In the previous question, you indicated that the CAN does not include representatives from all of the service areas that Healthy Start participants need. Select the service areas that Healthy Start participants need that are not represented on the CAN. (CHECK ALL THAT APPLY)
Below is a definition of health equity. Is there anything that you would add to or change about this definition? Health equity is achieved when every person has the opportunity to attain his or her full health potential and no one is disadvantaged from achieving this potential because of social position or other socially determined circumstances. Health inequities are reflected in differences in length of life; quality of life; rates of disease, disability, and death; severity of disease; and access to treatment.	Seven respondents felt that the definition was too long, or they had to read the definition multiple times before they could answer the question. HRSA staff also suggested reminding the respondent what the “No” response pertains to.	Delete question and only provide a simplified definition of health equity: Health equity means that everyone has a fair and just opportunity to be as healthy as possible. This requires removing barriers/obstacles to health such as poverty, discrimination, and their consequences, including powerlessness and lack of access to good jobs with fair pay, quality education and housing, safe environments, and health care.
19. Using a scale from 1-5, (1=least likely cause, 5=most likely cause), to what extent do you think the following contribute to maternal and child health inequities in your community?	Respondents indicated that they did not have difficulty with using this scale, but per HRSA feedback, a 4-point scale was preferable. HRSA staff also suggested giving examples of racism for option on discrimination. Many respondents agreed that including examples would be helpful, and they agreed with the suggested examples.	Change the scale to the following 4-point scale: not at all a contributor, somewhat of a contributor, a moderate contributor, a significant contributor. Provide examples of racism: Discrimination: racism/bias (such as overt, perceived, structural/systemic, cultural, educational, employment, etc.)
22.a. and 25.b. Are there additional activities that you would like your CAN to conduct to further address health equity for mothers and their families in your community?	Three respondents skipped this question because their response was “No,” but there was not an option to select “No” as a response.	Allow respondents to select “Yes” or “No” after the question, and if the answer is “Yes,” then the respondent is asked to describe the additional activities.
23.a. and 23.b. Are there additional resources and support that you would like your CAN to get to enhance its health equity work?	Two respondents skipped this question because their response was “No,” but there was not an option to select “No” as a response.	Allow respondents to select “Yes” or “No” after the question, and if the answer is “Yes,” then the respondent is asked to describe the additional resources and support.
24. How much did your organization or group collaborate with other organizations/groups in the Healthy Start CAN to respond to COVID-19 by assisting the Healthy Start community in the following areas?	Respondents did not have difficulty answering this question, but we felt that the question could be clearer, as suggested by HRSA staff as well.	Change the question text to ask about collaborations with CAN members around activities to respondent to COVID-19 in their Healthy Start community.
25. Please list one thing that [CAN NAME] could do that would most enhance inter organizational collaboration.	While respondents did not have trouble with this question, we wanted to better define “inter-organizational.”	Instead of using the term “inter-organizational collaboration,” replace with “collaboration between CAN members.”
General	Some respondents were not sure about what their CAN is.	Include the name of the local CAN in more places throughout the survey.
General	Some respondents were not sure how to answer some questions but did not want to skip these questions. HRSA staff also asked whether the survey is programmed to require a missing response for each item to avoid missing data.	Add “Don’t know” as a response option to the following questions: 4, 10, 11, 14, 16, 18, 19, 20, 21, 24. While the survey and all questions are entirely optional, adding “Don’t know” as a response option should limit missing data.
General	One respondent forgot to select a response for “Other” in a matrix question even though they typed in a response.	Add the following programming note: “CHECK THAT A RESPONSE IS SELECTED FOR “Other” IF AN ANSWER IS SPECIFIED.” to the following matrix questions: 4, 19, 24.
General	One respondent used a tablet to complete the survey and had to scroll up to see the header row for the response options when completing some matrix questions.	Add the following programming note: “REPEAT HEADER ROW AS RESPONDENT SCROLLS DOWN.” to the following matrix questions: 4, 9.b., 11, 12, 16, 19, 20, 24.
General	HRSA staff asked whether certain response options in a matrix can be programmed as single-select options (e.g., “services not provided,” “not applicable,” etc.).	While SurveyMonkey does not have this programming capability, if the Westat survey framework is used, these response options can be programmed to be single-select. Add the following programming note: “PROGRAM [“Services Not Provided” AND “Not Applicable” AND “Don’t Know”] TO BE SINGLE-SELECT IF POSSIBLE.
General	HRSA staff suggested specifying the character limits in the free response questions.	Add character limits to the “other (explain)” fields and open-ended questions where applicable.
General	HRSA staff suggested using language that is inclusive of all genders and orientations.	In the introduction and applicable questions, revise “preconception, pregnant, and postpartum women” to “preconception, pregnant, and post-partum people,” and replace other references to “women” with “Healthy Start participants.”

Table 1. Stakeholder Interview Results and Recommendations, by Question

Stakeholder Interview Guide Question Number and Original Text	Summary of Feedback	Recommendation
Introduction and consent	HRSA staff suggested that some sentences are long and use complex language. They also noted that the interview guide seemed longer than the estimated 30-45 minutes.	Delete some text and use plain language, and change interview time to 45-60 minutes.
1. To begin, tell me how long you have worked with [INSERT HS PROGRAM NAME] and describe your current role. [INTERVIEWER – CHECK ROLE] - CAN member	The CAN members interviewed were unable to respond to questions about Healthy Start initiatives. They were not intimately familiar with Healthy Start program activities, even though they had been collaborators for over a decade.	Do not include CAN members for these interviews and delete the option “CAN member” from this question.
5. What are some of the activities that HS has implemented to advance health equity and reduce disparities in reproductive health and outcomes for the program’s staff, participants and community? d. What were some specific efforts? PROBES: Tell me more about efforts or programs addressing social determinants of health, such as related to racism; discrimination; healthcare access; education; workplace conditions; incarceration; poverty; employment; housing; transportation; access to food; or neighborhood crime and violence.	HRSA noted that leaving health equity questions until the end of such a long interview guide suggests that they are not high priority items. (This question was earlier #11 towards the end of the guide). Participants did not give in-depth and specific responses to this question and 5d, in particular did not flow well.	Move this question earlier in the interview. Revise 5d to: Tell me about efforts or programs addressing social determinants of health, such as related to healthcare access; education; incarceration; employment; housing; transportation; access to food; or neighborhood crime and violence. PROBE: How successful were these efforts? What challenges did you encounter in these efforts? How did you address those challenges?
8b. How would you describe how your CAN functions as a community network? PROBES: For example, in terms of low, medium, high functioning	HRSA staff suggested that respondents might not be comfortable to give an honest response to this probe.	In the introduction/consent section add: Whatever you say will be anonymous, as we will combine all the information from the different interviews for our report. Please be as frank as possible about Healthy Start because we want to hear both what works well and what does not so HRSA can improve the overall program.
9. What do you think are the major accomplishment of your HS program? PROBES: Tell me about outcomes, core services, specific initiatives (e.g., Fatherhood, Community Action Network or CAN).	Respondents felt they were discussing the same thing when responding to this question about major accomplishments (initially question 2) and successful activities/services and the key drivers of success (initially question 10).	Combine the two questions and revise to: What do you think are the major accomplishment of your HS program? PROBES: Tell me about outcomes, core services, specific initiatives (e.g., Fatherhood, Community Action Network or CAN). a. Who were the key players/ drivers of these accomplishments? For example, what were the roles or job functions of people who were responsible for the success of these activities?
11./Appendix A: Healthy Start Benchmarks	Most respondents were not able to give in-depth responses to the question about healthy start benchmarks due to not having enough information. Two respondents who gave in-depth responses went above 60 minutes overall interview time.	Include these questions only for the project directors as they are more likely to keep track of benchmarks and ask about key activities that their HS program has conducted to meet some of the benchmarks.
12. In thinking about some of the new national programs/policies, how have they impacted your community?	Some participants were not aware of how national policies affect their programs and mentioned their project directors as more knowledgeable. Some also pointed out that they were located in rural, low populated areas and did not have specific initiatives that address immigration.	Include this question for project directors only.
13. What are the other topic areas that are important for the Healthy Start program from your perspective that we have not discussed?	Respondents suggested a range of additional topics for discussion during the interviews pertaining to older children’s safety, a different approach to the fatherhood program, and environmental impact on service (e.g., hurricanes and tornadoes). Some also suggested including organizational issues such as leadership, staff interactions, and turnover.	We do not recommend including these additional topics as they are already addressed in either the NICHQ annual assessment or the Program Survey that will be administered to all grantees. However, we can analyze these responses to assess the topics that are important for program stakeholders.
Closing	HRSA staff suggested providing contact information to participants for additional feedback.	Include contact information for follow-up questions or comments at the end of the discussion: You can also reach us at HSEvalSupport@westat.com.