Non-Substantive Change Best Practices for COVID-19 Vaccination and Testing

Memo

To: OMB

From: Deborah Porterfield, U.S. Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (ASPE)

Date: March 27, 2024

Subject: OMB Control Number 0990-0421, Expires 2/28/2027

ASPE requests approval for non-substantive changes to site visit data collection for the Best Practices for COVID-19 Vaccination and Testing study, based on pilot interview experience. We request these changes to improve the quality of the data collected by our contractor.

1. Reduction in estimated burden

Summary of change: The contractor will no longer request 15 minutes of planning time from the primary staff person at each program selected for site visits. They will now request recommendations for additional program and partner staff respondents within the 60-minute interview time. This reduces burden for up to 57 lead program respondents by a total of 14.25 hours and $637 in burden costs.

Justification for change: Eliminating the extra 15 minutes is expected to support response rates because it is easier for respondents to commit to one hour than to more than one hour. Higher response rates will increase the quality of the data collected.

2. Increase in respondent payments

Summary of change: Instead of providing a $25 Visa gift card to community representative respondents who complete 30-minute interviews, we will increase this amount to $50. We will also provide $100 Visa gift cards to program and partner staff who complete 60-minute interviews.

Justification for change: The new payment amounts appropriately compensate respondents for their time and expertise. These amounts are also expected to boost response rates because they are more consistent with current standards for respondent compensation. Higher response rates will increase the quality of the data collected.

3. Improvements to outreach materials

Summary of change: We made two types of changes to outreach materials. First, we made minor updates in the text to offer interview scheduling options, to offer payments for all program respondents, and, for initial outreach emails, to CC (carbon copy) the person who recommended the potential respondent. We also added three new outreach email templates, one for the planning call outreach, and one each for reaching out to and following up with partner respondents.

Justification for change: Updates to existing materials reflect changes to our processes to improve response rates, which will increase the quality of the data collected. New outreach email templates are customized to meet the needs of specific respondent types, also with the goal of improving response rates and improving the quality of the data collected.

4. Changes to data collection instruments

Pilot interviews revealed the need to refine the flow of the interviews, clarify the intention of some questions, and improve the face validity of the questions. The revisions will support interviewers’ ability to collect rich data and identify nuances and novel findings. Table 1 (see next page) provides details on changes to instruments and justification for changes.

Table 1: Summary of changes to interview guides and justification

Original interview guide text	Changes to interview guide text	Summary of justification for change and benefit to the information collection
Changes to instructions to interviewers
All interview guide instructions: You do not need to read the content in parentheses; use that content as needed to help respondents if they ask for clarification.	All interview guide instructions: Collect detailed information on each topic. Use the questions and probes under each topic to help respondents generate details and information relevant to our project’s research questions.	Based on pilot experience that interviewers need instruction to ask all probes and follow up questions Expected to generate richer and more detailed data, thereby improving quality of data collected
Program and partner introductory talking points: I have you scheduled for [75 minutes for program leaders; 60 minutes for program staff and partners]. Are you available to talk for that amount of time?	Program and partner introductory talking points: I have you scheduled for 60 minutes. Are you available to talk for that amount of time?	Based on pilot experience that several interviewees were hesitant to provide 75 minutes and successfully conduct of shorter interviews Expected to reduce burden on respondents and increase response rates, thereby improving quality of data collected
Changes to clarify interview questions
Public health leader Q2b; program and partner Q3: How did you change those efforts to make them more effective over time? Program and partner Q2c: [If not already answered] How has [program] worked to improve COVID-19 testing or vaccination confidence? What about access? Awareness? Uptake or use?	Public health leader Q2b; program and partner Q2c: [If unanswered, and for each effort] When did this effort take place? (early in the pandemic before vaccines were available, after adult vaccines became available, after child vaccines became available, later) Program and partner Q2c-d: [If unanswered] When did those efforts take place? (early in the pandemic before vaccines were available, as adult vaccines became initially available, after adult vaccines became widely available, after child vaccines became available, later) Did the [program] change significantly over time? [If yes]	Added specificity on timeframes based on pilot experience that original questions were unclear with respect to timing Expected to generate more specific information to improve quality of data collected
Public health leader Q5 (similar changes made to program and partner Q14) What policies have supported improvements in COVID-19 vaccination and testing for people at disproportionate risk? [If unanswered] What federal or national policies, regulations, or funding have supported improvements in COVID-19 testing and vaccination for people at disproportionate risk? Did any policies present challenges? Why?	Public health leader Q5 (similar changes made to program and partner Q13) What policies have supported improvements in COVID-19 vaccination and testing for people at disproportionate risk? By policy, we mean a law, regulation, procedure, administrative action, incentive, or voluntary practice of government and other institutions. Policy is frequently reflected in resource allocation. [If unanswered] What federal or national policies, regulations, or funding have supported improvements in COVID-19 testing and vaccination for people at disproportionate risk? [Ask for names of policies or examples of policies in action so we can dig up specifics in analysis if needed] [If unanswered] What state or local policies, regulations, or funding have supported improvements in COVID-19 testing and vaccination for people at disproportionate risk? [Ask for names of policies or examples of policies in action so we can dig up specifics in analysis if needed] Did any policies present challenges? Why? [Ask for names of policies or examples of policies in action so we can dig up specifics in analysis if needed]	Added a definition of policy to the question based on pilot experience that respondents were not sure what we meant by policies Expected to generate more specific information to improve quality of data collected
Program and partner Q7: What are the views of the community on how well [program] works? How did you learn about their views? (Examples include survey, community advisory board, partnering with community leaders) What actions have you taken based on ideas from the community, if any?	Program and partner Q7: What are the views of the people or communities the [program] intended to serve on how well [program] works? How did you learn about their views? (Examples include survey, community advisory board, partnering with community leaders) What actions have you taken based on ideas from the people or communities [program] intended to serve, if any?	Added specificity about the communities that the program intended to serve, based on pilot experience that original question implied we were asking about communities generally Expected to generate more specific information to improve quality of data collected
N/A	Public health leader guide instructions: We’d like you to think about your perspective on the [state/tribal organization] rather than talk in detail about specific programs or strategies you worked on.	Added clarification about how the respondent should focus their answers during the interview, based on pilot experience that public health leaders sometimes talked in too much detail about programs Expected to generate more data about state/tribal organization overall, thereby improving the quality of data collected from public health leaders
Changes to questions to focus respondents’ answers
Public health leader guide Q6: What have been the roles of your partners in efforts to improving COVID-19 testing or vaccination? By partners, we mean organizations that contributed to the success of programs that work to improve COVID-19 testing and vaccinations, such as by helping design, plan, implement, or contribute resources. [If unanswered] Who have you partnered with? What have been their roles? [If unanswered] Were those partners you funded, or worked together with? Which of these were new partnerships for your COVID-19 testing/vaccination work? Why were new partnerships needed? What made the partnership(s) work well? What has been challenging about partnering? What lessons did you learn about how to overcome the challenges? Did you do anything to foster coordination or alignment among other testing and vaccination programs? Tell me about those efforts. What was their impact? Program and partner Q13a: Who were the different partners? What were their roles?	Public health leader guide Q6: Now I want to talk about partners [site] may have worked with. By partners, we mean organizations that contributed to the success of [site] programs that work to improve COVID-19 testing and vaccinations, such as by helping design, plan, implement, or contribute resources. Who were your most important or effective partners during the COVID-19 pandemic, and why? vaccination? [If unanswered] What were their roles? What made the partnership(s) work well? [If unanswered] Were those partners you funded, or worked together with? Which of these were new partnerships for your COVID-19 testing/vaccination work? Why were new partnerships needed? Program and partner Q12a: Who were [the program’s] most important or effective partners, and why?	Asked respondents to focus on most successful partnerships, rather than all partnerships Expected to generate information relevant to research questions to improve quality of data collected
Public health leader Q13a: Have you combined resources from different funding sources for these activities? If so, how?	Public health leader Q13a: Have you combined resources from different funding sources for these activities? If so, how did that go?	Based on pilot experience that the original question resulted in descriptions of the process, changed the question to focus on perceptions of how well that process went Expected to generate information relevant to research questions to improve quality of data collected
Program and partner Q1: What is/was your role in relation to [program]? How long have you had/did you have that role?	Program and partner Q1: I’d like to focus our discussion on [program’s] efforts related to [COVID-19 testing/vaccination.] What was your role in relation to COVID-19 [testing or vaccination] at [program] during the pandemic? How long have you did you have that role?	Based on vague responses in pilot interviews, changed question to ask about respondent’s role related to the program we are studying and to ask only about testing or vaccination Expected to generate information relevant to research questions to improve the quality of data collected
Community representative Q5: Are you aware of if [program] asked community members for ideas on how to design the program or make it better? [If yes] How did they do that? (Probe on community advisory boards, or similar entities)	Community representative Q5: Can you recall any aspects of [program] that didn’t work that well for [community identified in Q1], and that you think could have been, or could be, improved to better meet the needs of the community?	Based on pilot experience that this question was difficult for respondents to answer, we changed the question to be more direct and focused Expected to generate information relevant to research questions to improve the quality of data collected
Changes to the specific focus of the question
Program and partner Q18: [For program leaders only, if program leader mentioned a data source in response to Q7a] We are interested in looking at the data that programs collected to measure effectiveness, to help us understand the different ways that programs collected and used data for COVID-19 services. Previously, you mentioned [describe data from Q7a]. Would you be willing to share the data to inform part of our study? We can provide way to securely transfer the data and we would never share it with anyone outside of our research team without your permission. Alternatively, if you have any report or write up of your program data, are you able to share that with us?	Program and partner Q17: [For program leaders only, if program leader mentioned a data source in response to Q5b] We are interested in looking at any analyses or assessments you might have conducted to understand how well [program] worked. Previously, you mentioned [describe data from Q5b]. Would you be willing to share any results or analyses to inform part of our study, if those are available in the form of reports or presentations?	Pilot respondents expressed concerns about sharing data, therefore changed question to ask about publicly available program data or reports only Expected to increase the likelihood that respondents will be willing to share program data or analyses, thereby improving the quality of data collected
Community representative Q1: I’ll start by asking you about yourself. How would you describe your connection to [community]?	Community representative Q1: I’ll start by asking you about yourself. How would you describe your connection to the [program] and the [community/communities] that it intended to serve? How would you define that community – or the group of people the program intended to serve – that you’d like to focus our conversation on today? I’d like you to try to keep this community and group of people in mind when answering my questions today.	Based on pilot experience that respondents can represent a range of communities, changed the question to give respondents the opportunity to identify and define the community they would like to represent Expected to provide context for other questions in the interview, improving the research team’s ability to interpret the data and improving the quality of data collected
Deleted questions to save time in interviews
Public health leader Q6d: Which of these were new partnerships for your COVID-19 testing/vaccination work? Why were new partnerships needed?	None	Based on pilot experience, deleted this probe because we are interested in the most effective partnerships, not new ones Expected to save time for interview questions that are more relevant to our research questions, thereby improving the quality of data collected
Public health leader Q18: How did you come up with this definition of success?	None	Based on pilot experience, deleted this question because it did not generate new information Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of data collected
Public health leader Q18b: Which activities do you think were most successful or effective at improving COVID-19 services for specific populations?	None	During pilot interviews, this question resulted in information that duplicated another question Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Public health leader question 18c: [If relevant] Did the activity improve awareness of COVID-19 services? Confidence in the COVID-19 vaccine? Access to COVID-19 service? Use of COVID-19 services?	None	During pilot interviews, this question did not generate new information on the program’s outcomes, relative to other questions in the guide about outcomes Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q2a: [If not already answered] Did [program] focus on COVID-19 testing, vaccination, or both? If so, how?	None	Based on pilot experience, it is more efficient for interviewers to go into interviews knowing whether they will focus on testing or vaccination Expected to generate more detailed information about either testing or vaccination, rather than briefly covering both, thereby improving the quality of the data collected
Program and partner Q8: From your perspective, how has operating [program] gone? (Use Q10-11 to follow up on their response to this general question).	None	During pilot interviews this question generated information that duplicated responses to questions asking about barriers and facilitators Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q10a: [If unanswered] What about the planning process helped the program succeed? (Examples include identifying milestones, defining measures of success, clarifying roles and responsibilities, seeking community input, tailoring the program to be culturally relevant and align with community’s values.)	None	Deleted this question based on pilot experience that respondents could not recall enough information to answer Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Program Q10 c-d: [If unanswered] How did [program] adapt services to meet the needs of the people whom the program intended to serve? (For example, extending hours, allowing walk-in appointments, co-locating other social services, offering translation services, etc.) [If unanswered] How did you decide where to focus any program improvement efforts?	None	During pilot interviews these two probes generated similar information to that generated by another question Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q11: What was [program]’s experience with these resources?	None	Deleted this question based on pilot experience that respondents did not understand the question Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q3b: How have you prioritized what needs or barriers to services to address in [program]?	None	Deleted this question it did not generate new information during the pilot interviews, relative to other questions Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Community representative Q8: Are you aware of any other efforts to help people in the community stay up to date on routine vaccines during the pandemic, such as education about why that was important or any other support for getting routine vaccines?	N/A	Deleted this question based on pilot experience that it overlapped with Q7 Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Community representative Q3f: Did [program] change how often people got tested? How?	N/A	Deleted this question based on pilot experience that respondents could not answer this question Expected to save time for other interview questions that are relevant to our research questions, thereby improving the quality of the data collected
Changes to the wording or ordering of questions to improve flow and efficiency of interviews
Public health leader Q19: How has [state/tribal organization] assessed the effectiveness of the COVID-19 testing activities it provides? What about COVID-19 vaccination activities? What program metrics or outcomes have you tracked? (By outcomes, we mean the indicators of success or effectiveness. They could be quantitative or qualitative indicators). [If not already answered] Have you been able to assess outcomes like improvements in testing or vaccine awareness, confidence, access, or use? What types of resources or data have you needed for these assessments? [If not already answered] Does the data you use allow you to assess the success of specific initiatives?	Public health leader Q18: What were your [state’s/tribal organization’s] specific goals related to COVID-19 testing? What were your goals related to COVID-19 vaccination? [Interviewers: probe to understand if programs intended to improve awareness, confidence, access, or use of testing/vaccines in general] Did you meet those goals for testing? For vaccination? How do you know you met those goals for testing? For vaccination? What program metrics or outcomes did you track for testing? For vaccination? (By outcomes, we mean the indicators of success or effectiveness. They could be quantitative or qualitative indicators.) [if unanswered] Did the data allow you to assess the success of any specific initiatives?	Based on pilot experience that Q18 and Q19 covered similar topics, embedded original Q19 question into probes in Q18 Expected to improve efficiency and flow during the interview to save time for other topics relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q4: How have you designed communication efforts to address the values and beliefs of the people [program] intends to serve? [If relevant] How have you identified who or what is the right voice for communicating with people about [program]? Why did you think they would be an effective communicator for this community? Program and partner Q11: What resources were needed to operate [program]? (Examples of resources include funding, service site, staff, technology, cold storage, community members or leaders) How has [program] obtained the resources needed to support the program, including staffing? [If relevant] How did you identify bilingual staff or employ other language translation services?	Program and partner Q4: How did you design [program] to meet the needs of [populations of focus]? For example, making [program] culturally relevant, designing it to address specific barriers such as communication preferences and transportation availability. This could also include adapting services to meet the needs of specific groups, such as extending hours, co-locating with other social services, offering translation, etc. What information did [program] use to decide how to design or improve COVID-19 testing and vaccination efforts for specific communities?	Based on pilot experience that Q4 overlapped with another question, deleted the original Q4 and replaced with a question that combined the original Q4 and Q11 Expected to improve efficiency and flow during the interview to save time for other topics relevant to our research questions, thereby improving the quality of the data collected
Program and partner Q12: I’m also curious about the cost of [program]. Could you give me a ballpark estimate of how much it costs to run [program], overall or by year? What are the biggest expenses for [program]?	Program and partner Q11: I’m also curious about the cost of [program]. What were/are the biggest expenses for [program]?	Based on pilot experience that respondents could not answer this question (either because they did not know or could not remember), we replaced this question with one that respondents are more likely to be able to answer Expected to generate information relevant to our research questions, and do so more efficiently, saving time for other topics relevant to our research questions and improving the quality of the data collected
Program and partner Q17: How will COVID-19 [services] that [program] provides change in the future? How will [program] need to change going forward? Why? What resources will you need to continue? Probe to understand barriers and facilitators to continuing [program] in the long term.	Program and partner Q16: Do you plan to continue offering the COVID-19 [services] that [program] provides in the future? [If yes to Q18] What resources will you need to continue? [Probe to understand barriers and facilitators to continuing [program] in the long term.]	Based on pilot experience that these questions were overly time consuming, refocused on highest priority topics Expected to improve efficiency and flow during the interview to save time for other topics relevant to our research questions, thereby improving the quality of the data collected
New questions added
N/A	Program and partner Q19: To wrap up, I’d like you to think back on our conversation and your experience with the pandemic. What do you think is most important for me to get on the record to help other programs met the needs of people who are medically or socially at disproportionate risk of adverse outcomes, should another pandemic arise?	Added a final reflection question based on pilot experience that a wrap-up question generated new and insightful information Expected to generate new, insightful information that relevant to our research questions, thereby improving the quality of data collected
N/A	Program and partner Q20: [Read for all program respondents] Finally, would you be willing for [this program/organization] to be identified in a vignette or short profile about your efforts in our reports? If not, that’s completely OK.	Based on feedback from pilot interview respondents that they wanted credit for their work, added opportunity to be identified in project reports Expected to fulfill a principle of equitable evaluations to engage in reciprocal, mutually beneficial partnerships with respondents and to improve the quality of reports based on the data collected