0920-1027 NPIN Evaluation Moderator Guide - Rank and File

Form Approved

OMB No. 0920-1027

Expiration Date 07/31/2020

NPIN Evaluation Moderator Guide - PUBLIC HEALTH PROFESSIONAL RANK and FILE

Public reporting burden of this collection of information is estimated to average 2 hours per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; Attn: OMB-PRA (0920-1027)

Guide for Conducting Group Interviews

puBLIC HEALTH Rank and File Professionals

I. Welcome and Introduction

Thank you for coming today to our focus group on the National Prevention Information Network, better known as NPIN. I’m Mark Herring, and I’m from Schlesinger, a marketing research organization. The Centers for Disease Control and Prevention (CDC) is sponsoring this research.

The purpose of today’s focus group is to hear your views and opinions on NPIN in order to make it a stronger service and to better meet your prevention information needs. It’s been several years since NPIN was last evaluated, and we know that the landscape of health communication and health technology has changed considerably. The purpose of this effort is to evaluate the awareness, perceptions, and utilization of NPIN services from public health professionals like you who use the platform

Your insights are very important to us, and we appreciate your time today. Please be honest with your feedback. We expect that you will offer both compliments and criticism, and both are welcome. We want you to express both the positive and negative aspects in order to best shape NPIN for the future.

This focus group will last two hours.

Before we begin, I want to review a few ground rules for our discussion.

• Most importantly, there are no right or wrong answers. We want to know your opinions and what you think about the issues we will be discussing. All of your responses are confidential and none of your responses or feedback will be linked to any information that can identify you personally.

• You have probably noticed the microphones in the room. They are here because we are recording these sessions and they will be transcribed for analysis. Please speak loudly and clearly. I want to give you my full attention and not have to take a lot of notes. At the end of our discussion, I will be writing a report and will refer to the recording when doing so.

• Behind me is a one-way mirror. Some of the staff who support NPIN are observing this discussion so that they can hear your opinions directly from you and take notes to accurately capture your opinions. However, your identity and anything you personally say here will remain secure. Your name, address, and phone number, which only Schlesinger knows, will not be given to anyone, and no one will contact you after this interview is over.

• We have assigned you a number for reference. Be sure to only use participant numbers when addressing other members of the group. Please do not use any names.

• We ask that each participant respect each other’s privacy, as well. Please do not share what was said in today’s group with others after the group is over. Please be respectful of others’ opinions. Any participant who is disrespectful to others or disruptive to the group will be asked to leave the group.

• Please turn your cell phone to vibrate or silent mode.

• As indicated in your consent forms, today’s sessions will be recorded.

Do we have your permission to record today’s session? If so, please respond with “yes.” [MODERATOR WAITS FOR CONSENT] Thank you.

• Do you have any questions before we begin?

WARM UP

Before we begin our discussion, let’s learn a little bit about one another

• Please tell me your favorite hobby and a show you’re currently watching on Netflix, Hulu, Amazon Prime, etc., and what you like about. If you’re not a big TV person, tell us about a book you’re reading.

II. General Awareness of NPIN

• How did you first become aware of NPIN?

Probes:

• At a conference

• Email list

• Website search, such as Google

• NPIN social media

• Through training

• What would you say NPIN is known for?

Probe:

• What does NPIN do?

• What services does it provide?

• On the note pad in front of you, please write the one word or phrase that you think best describes NPIN. In a minute, I’m going to go around the table and ask each of you for the word you selected and a 15-second explanation for choosing that word.

[NOTE: Moderator elicits word and explanation from each respondent individually]

III. Overall NPIN Brand Perceptions

Now, let’s talk about NPIN, what that brand represents to you. I’d like you to write down three words or short phrases that describe positive qualities of NPIN. After that, I’d like you write down three words or phrases that describe negative qualities of NPIN. You will only have three minutes to complete both tasks, so please work quickly.

[NOTE: After respondents write answers, moderators write positive qualities and negative qualities on two different flip charts]

• Let’s discuss the positive and negative qualities you all put down.

• What made you write these positive qualities?

• What made you write these negative qualities?

I’d like to learn more about how you perceive NPIN. Using this worksheet, rate your perceptions of NPIN on the listed characteristics. Please complete this worksheet as honestly as possible.

[NOTE: Moderator distributes Perceptions of NPIN worksheet]

{NOTE: After respondents complete Perceptions worksheet, moderator asks the following questions]

• How many of you perceive NPIN as a primary source for [ask each disease as represented in the room-HIV/AIDs, STDs, viral hepatitis, TB, adolescent sexual health information –]?

• What is NPIN doing well, since you consider it a primary resource?

Probes:

• It’s the first place you go for prevention information

• NPIN keeps you consistently updated with the latest prevention information

• It’s a highly trusted source of prevention information

• For those of you who do not perceive NPIN as a primary source on this topic, why not?

Probes:

• There are other sources you go to first for prevention information

• You only go to NPIN when it comes up in a Google search

Now, let’s look at your responses at the bottom of this worksheet and let’s discuss each characteristic.

• When rating NPIN as being innovative, did any specific aspects of NPIN come to mind?

Probes:

• Its approach to providing prevention information

• Its use of communication, such as social media

• When rating NPIN as being technology-focused, did any specific aspects of NPIN come to mind?

Probes:

• NPIN’s technological quality

• NPIN digital prevention tools

• When rating NPIN as being a collaborative partner, did any specific aspects of NPIN come to mind?

Probe:

• A specific collaborative engagement involving NPIN

• When rating NPIN as being well-known, did any specific aspects of NPIN come to mind?

Probe:

• NPIN’s involvement in activities and events, such as conferences or trainings

• NPIN’s presence in areas such as digital media

• When rating NPIN as being staffed by experts, did any specific aspects of NPIN come to mind?

Probe:

• NPIN’s association with CDC

• NPIN staff

• Speakers at NPIN-sponsored trainings

NPIN Website

The main location to find NPIN resources is its website. [MODERATOR PULLS UP NPIN.CDC.GOV SITE]

• Who has visited the NPIN website before? [MODERATOR GETS A SHOW OF HANDS]

• Have you visited the site on your smartphone?

Probe:

• If so, when might you access the website through mobile instead of desktop?

• What are your positive impressions of the NPIN website?

Probes:

• Prevention information seems well organized

• Site has current information

• Site has information on disease topics relevant to my work

• It seems easy to use

• What are your negative impressions of the NPIN website?

Probes:

• Prevention information seems disorganized

• Information seems out of date

• Too much information/hard to navigate

• Think back to a time you visited the NPIN page. What made you go to the NPIN website instead of another?

Probes:

• Credible information due to relationship with CDC

• Regularly visiting the site

• Appeared high in Google search

• How has this website helped you in your prevention work?

Probes:

• Provides prevention resources, tools, and materials that are useful in my work

• Keeps me updated on the latest prevention news or science

• Connects me with other prevention colleagues/organizations

• What could be done to improve the NPIN Website?

IV. NPIN Services

In the next part of the discussion we are going to discuss the resources and services that NPIN has. As a reference, I’m going to give you a list of NPIN resources and services with their descriptions. We don’t expect you to have experience with everything we’re going to review.

Newsletters, Announcements, and Letters

I’d like us to start by reviewing some materials that are developed and distributed by NPIN. Each of these is sent to subscribers electronically.

Connections

[NOTE: Moderator distributes hard copy of Connections]

This is a recent copy of Connections. You can see that this is a bimonthly one-page newsletter. It’s delivered electronically.

• How many of you currently subscribe to the Connections newsletter? [MODERATOR GETS A SHOW OF HANDS]

• What do you like most about this newsletter?

• What type of information in such a newsletter is most helpful to you work?

• How does the content of this newsletter support your prevention work?

Probes:

• Provides timely update on prevention news

• Connects you to prevention resources you may not have been aware of

• Keeps you updated on new web content at CDC

• Let’s you know about relevant upcoming prevention-related events

• Are there any sections that are particularly helpful?

Probes:

• From the Director

• Upcoming events

• New from CDC

• What changes or improvements do you recommend to the Connections newsletter?

Probes:

• A different format

• Some other specific information

• Frequency of delivery

Find TB Resources

[NOTE: Moderator distributes hard copy of Find TB Resources Newsletter]

This is a recent copy of the Find TB Resources Newsletter. You can see that this is a monthly two-page newsletter.

• How many of you currently subscribe to the Find TB Resources newsletter? [MODERATOR GETS A SHOW OF HANDS]

• What do you like most about this newsletter?

• How does the content of this newsletter support your prevention work?

Probes:

• Provides timely updates on TB prevention news

• Connects you to TB prevention resources you may not have been aware of

• Keeps you updated on new web content at CDC’s Find TB Resources website

• Are there any sections that are particularly helpful?

Probes:

• Highlight of the Month

• Additional Resources

• What changes or improvements do you recommend to the Find TB Resources newsletter?

Probes:

• A different format

• Some other specific information

• Frequency of delivery

• Are there any specific needs or gaps that you have regarding TB prevention information that you think Find TB Resources could better address?

Funding Announcements

[NEED SAMPLE FUNDING ANNOUNCEMENT]

Funding announcements are regular email announcements with descriptions of and links to funding and grants information.

• Who receives funding announcements from NPIN? [MODERATOR GETS A SHOW OF HANDS]

• What do you like most about these announcements?

Probes:

• They come consistently to update you about funding

• They have funding opportunities you might not find elsewhere

• Can you think of an example of how these announcements have benefited your work?

Probe:

• Became aware of funding you may not have otherwise seen

• You applied for funding you saw on the announcement

• Is there anything you’d recommend be changed or improved about these funding announcements?

Dear Colleague Letters

Dear Colleague letters are email announcements sent out periodically to inform recipients on topics such as disease surveillance updates or health awareness days.

[NOTE: Moderator distributes hard copy of Dear Colleague Letter]

• Who has received the Dear Colleague letters? [MODERATOR GETS A SHOW OF HANDS]

• What do you like most about these announcements?

• What information in these types of email communications do you find most useful to your work?

• How have the Dear Colleague letters benefitted your work?

Probes:

• Became aware of a disease surveillance update, like an update in the HIV or STD rates

• Got links or resources for an awareness day

• Connected with a new prevention resource

• What improvements would you suggest for the Dear Colleague letters?

V. DATABASES

Now, we will discuss the NPIN databases. One is a database of organizations, a second is a database showing available materials (information and resources), a third lists possible funding sources, and the final one offers campaign resources.

Organizations Database [Moderator pulls up Organizations Database]

First, let’s talk about the NPIN Organizations Database. Public health professionals like you can use the Organizations Database to find information about public health and health organizations, including their locations and the types of services they provide.

• How many of you have accessed the Organizations Database? [MODERATOR GETS A SHOW OF HANDS]

• How did the organizations database benefit your work?

Probe:

• Did you locate an organization for a client in need of a service?

• Were you satisfied with your experience using the Organizations Database?

Probes:

• You find the organizations listed are relevant to your work

• Organization contact information is up to date

• Are there any specific needs or gaps that you feel the NPIN Organizations Database could address regarding your work?

• Has your organization’s information been uploaded to the Organizations Database?

Probe:

• If so, how did you find this experience?

• Straightforward? Difficult? User-friendly?

• What could make the Organizations Database better?

Probe:

• Better location on the website

Funding Database

Now, let’s discuss the Funding Database. In this database you can find information on private and government funding opportunities for community-based prevention programs.

• How many of you have accessed the Funding Database? [MODERATOR GETS A SHOW OF HANDS]

• How did the Funding Database benefit your work?

Probes:

• Gave you access to private and government funding opportunities you weren’t aware of

• Is a starting point for seeking financial support in my disease area

• Is an easy place to frequently look for possible funding opportunities

• What could make the Funding Database better?

Campaigns Database

The Campaigns Database is where NPIN partners like you can find disease campaign materials, like banners, graphics, and other campaign content from promotional campaigns like Act Against AIDS. Here is an example of the Campaigns Database. [SHOWS CAMPAIGNS DATABASE]

• How many of you have accessed the Campaigns Database? [MODERATOR GETS A SHOW OF HANDS]

• Do you find the database easy or difficult to search?

• In general, what types of campaign materials do you find to be most useful in supporting your prevention work?

Probe:

• Posters

• Banners

• Palm cards

• Toolkits

• What have you found in the Campaigns Database that benefited your work?

Probes:

• A banner

• A poster

• Graphics for your website

• When you found something useful in the Campaigns Database, how did it benefit your work?

Probes:

• Put it on your website

• Printed it out and disseminated it to your constituents in person

• Took campaign collateral to an event like a health fair

• You can also submit your own campaign materials to share on NPIN. If you have you ever submitted a campaign material, how did you find the process?

Probes:

• Straightforward? Difficult? User-friendly

• What could make the NPIN Campaigns Database better?

Probes:

• Different types of campaign materials

• Different organization scheme

Materials Database

The Materials Database is where NPIN partners like you can find information and resources like fact sheets, annual reports, and toolkits. These may be CDC-branded materials or materials that other prevention organizations have submitted. Here is what the Materials Database looks like. [MODERATOR SHOWS MATERIALS DATABASE]

• How many of you have accessed the NPIN Materials Database? [MODERATOR GETS A SHOW OF HANDS]

• In generals, what types of prevention materials do you find most useful to your work?

Probe

• A toolkit

• A fact sheet

• An annual report

• Widgets

• What tools or resources on NPIN have you found that benefited your work?

Probes [MODERATOR SHOULD GO THROUGH THIS ENTIRE LIST]:

• A toolkit

• A fact sheet

• An annual report

• Widgets

• Posters

• Infographics

• Banners

• Badges

• When you found a useful item in the Materials Database, how did it benefit your work?

Probe:

• Put it on your website

• Printed it out and disseminated it to your constituents?

• Emailed it to colleagues

• You can also submit materials to share on the NPIN Materials Database. If you have submitted a material to be uploaded to the database, how did you find the process?

Probe:

• Straightforward?

• Difficult?

• User-friendly?

• What could make the NPIN Materials Database better?

Probes:

• More instructions on how to locate items

• Different organization of the materials

• Updated listing of materials

• Ability to query by more factors (date, etc.)

• Are there other types of materials you’d like to see here?

Before we conclude this review of the databases, I’d like to ask just a couple of summary questions.

• Based on the discussion today, or your past experience, what’s the difference between the Campaigns Database and the Materials Database?

Probes:

• How do you differentiate these two databases in your mind?

• Should these two databases be combined, or is it best that they remain separate?

VI. Learning Opportunities

Webinars

Now, let’s discuss NPIN’s webinars. NPIN works with the various division in the Center to provide informative webinars on disease updates and surveillance for public health partners like you...

• How many of you have ever attended an NPIN webinar? [MODERATOR GETS A SHOW OF HANDS]

• How did you find out about the webinars?

Probe:

• Email from NPIN

• Invited as part of a grantee or program group

• What types of webinars do you find to be most useful to your work?

Probes:

• Disease related

• Data or surveillance updates

• Was the information you received through the NPIN webinar beneficial to your work?

• Any suggestions for improving NPIN’s webinars?

• If NPIN developed new webinars in the future, what topics would be of greatest interest to you?

• How should NPIN notify you if/when they develop new webinars?

Trainings

Next let’s talk about training programs. NPIN stores trainings on its website and provides and facilitates virtual trainings.

Let’s first discuss the trainings on the NPIN site. [Moderator pulls up the training section on NPIN website]

• How many of you have accessed the trainings available on the NPIN training site? [MODERATOR GETS A SHOW OF HANDS]

Probe:

• Which training or trainings did you access?

• What do you feel are the strengths of this training section of the NPIN website?

Probe:

• Relevance to your specific disease area

• Access to trainings from various organizations in one location

• What do you feel are the weaknesses of this training section of the NPIN website?

Probe:

• Finding relevant trainings

• Out of date

• How have you used the trainings available on the NPIN website in your work?

• How many of you have ever attended an NPIN training (virtual or in person)? [MODERATOR GETS A SHOW OF HANDS]

• Do you remember what topic the training covered?

• How did you learn this training was available?

Probe:

• Email, social media, talking with a colleague?

• What was your impression of the quality of the training?

• Do you feel that the training was effective?

Probes:

• Were the learning objectives achieved?

• Do you feel like you gained a new skill?

• How were you able to use the skills you learned to benefit your work?

Probes:

• To develop a material or communication

• To improve your social media efforts

• What incentives make you want to participate in NPIN trainings?

Probes:

• CHES credits

• Other types of CEUs

• Knowing more about the trainers and their credentials

• Knowing the training schedule further in advance

• What training topics would you like to see NPIN provide in the future?

Probes:

• Health communication

• Health literacy and education

• Social and digital media

• Partner engagement and collaboration

• Technology

• Using and communicating effectively with data

• Trainings related to your specific disease topic

VII. Other NPIN Websites

NPIN has multiple sites to support other aspects of prevention, including testing locators and websites to support health observance days. One is Get Tested, and another is the National HIV Testing Day website.

• How important is having websites like these that connect your constituents to local STD or HIV testing resources?

• How does having these types of resources help shape your prevention efforts?

Get Tested Website

Now let’s talk briefly about the Get Tested website.

• How many of you have visited the Get Tested website before? [MODERATOR GETS A SHOW OF HANDS]

• What is your general impression of the Get Tested site?

• What would you say are the strengths of this site?

Probes:

• Comprehensive list of testing locations

• Functionality: the way the website geo-locates testing sites

• Ease of use

• What would you say are the weaknesses of this site?

Probes:

• Finding a location is difficult

• Functionality: the way the website geo-locates testing sites needs improvement

• Hard to use

• How have you used the Get Tested site to support your work?

Probe:

• To help a client find a testing site

• To put the widget on your organization’s website or social media

• How could NPIN improve the Get Tested website?

National HIV Testing Day Site

As we all know, health observance and awareness events are useful to help bring attention to a particular health issue at various times throughout the year.

• Are health observance or awareness days very important to you as a program manager?

• Which health awareness days are most important for your team/organization?

Probe:

• NHTD

• STD Awareness Month

• NBHAAD

• Are there any observance days that need more support than others?

• How could NPIN best support your organization in conducting observance day activities?

One NPIN website that supports observance days is the National HIV Testing Day (NHTD) website. This is a page for prevention organizations and partners to promote their NHTD activities.

• How many of you have visited the NHTD website before? [MODERATOR GETS A SHOW OF HANDS]

• For those of you who have been on the site, how have you used the NHTD site to benefit your work?

Probe:

• Did you share the site link with constituents?

• Did you link to the site on your organization’s page to promote NHTD?

• Did you upload your organization to the NHTD site to share testing day activities for the observance?

• In your opinion, what are the strengths of the NHTD site?

Probes:

• Great resources for NHTD

• Functionality: the way the site identifies and geo-locates organizations with NHTD activities

• Great to see other organizations that are participating

• In your opinion, what are the weaknesses of this site?

Probes:

• Would like more content

• Functionality: the way the site identifies and geo-locates organizations with NHTD activities needs improvement

• How could NPIN improve the NHTD website?

Social Media Activities

NPIN also has a social media presence.

• How many of you use social media in your day-to-day work? [MODERATOR GETS A SHOW OF HANDS]

• Which social media channels do you use most in your work?

Probes:

• Twitter? FB? Instagram?

• How do you use social media in you work?

Probes:

• Send out prevention messages to constituents

• Connect with other prevention organizations

• Stay up to date on the latest prevention information

• Is it important to you that an organization like NPIN communicate to you through social media? Why or why not?

• Are there other HIV/AIDS, TB, STD, viral hepatitis, or adolescent sexual health focused social media accounts you follow?

Probes:

• Other relevant CDC accounts

• Local health departments or organizations

• Specific SMEs or leaders in these disease topics

• Federal organizations, like HIV.gov

• Which, if any, of the NPIN social media channels do you follow?

• @CDCNPIN on Twitter

• CDC NPIN on Facebook

• CDC NPIN on LinkedIn

• Where do you think it’s most important for NPIN to have a social media presence – Facebook, Twitter, Instagram, LinkedIn, or other?

• How have you engaged with NPIN social media?

Probes:

• Retweeted NPIN tweets

• Reposted Facebook messages

• Participated in an NPIN Twitter Chat

• Signed up for an NPIN Thunderclap

VIII. Summary Assessment

We’ve talked a lot about the specific resources that NPIN provides and I really appreciate all of the great feedback each of you has offered. We’re almost done.

We’ve discussed a lot of resources today—materials, newsletters, webinars, social media. Before we finish, I would like to know what else you need as public health professionals.

• Are there any gaps or needs in prevention resources that you can think of that you feel need attention?

IX. Wrap-Up

• In closing, do you have any last thoughts you’d like to share?

Excuse me for one moment while I see if the people observing have any questions that I have not asked. I will be right back.

[NOTE: Moderator steps out to check to see if there are any additional questions]

Thank you for your participation. Please leave all your worksheets and other materials on the table. We will be using your written feedback as we write our final report. Have a good day/evening.

14