Usability Testing to Inform Development of the
CDC Division of Sexually Transmitted Disease Prevention’s Website
Generic
Information Collection Request under
Formative Research and
Tool Development OMB #0920-0840
Supporting Statement B
CONTACT
Tiffani Phelps, MPH
Health Communication Specialist
Division of STD Prevention
Centers for Disease Control and Prevention
12 Corporate Square Blvd, Mailstop
H18-3
Atlanta, GA 30329
Phone: 404-639-4885
E-mail: itb0@cdc.gov
TABLE OF CONTENTS
This data collection does not involve statistical methods. 2
1. Respondent Universe and Sampling Methods 2
2. Procedures for the Collection of Information 5
3. Methods to Maximize Response Rates and Deal with No Response 8
4. Tests of Procedures or Methods to be Undertaken 9
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data. 9
There are three sets of participants to be recruited for this evaluation: adults disproportionately affected by STDs; healthcare providers who work on STD prevention, testing, and treatment; and public health professionals working on STD prevention and/or treatment. Adult populations disproportionately affected by STDs include three subgroups: younger adults (among adults, those ages 18-24 experience the highest rates of STDs), men who have sex with men (MSM), and pregnant women. The adult population will be recruited from across the United States using a panel provider for expediency and confidentiality given the sensitive nature of the topic. Panel providers have existing categorized lists of potential participants from which they will be able to select an appropriate sample for recruitment for 18 in-depth telephone interviews (IDIs).
Healthcare providers and public health professionals will be recruited from contacts and professional networks for 25 IDIs with healthcare providers, and 15 IDIs with public health professionals. For healthcare provider recruitment, these organizations will include the National Association of Community Health Centers, the National Medical Association, the Society for Adolescent Health and Medicine, National Coalition for Sexual Health, and the American Academy of Pediatrics. For public health professionals, these organizations will include the Association of State and Territorial Health Officials, National Association of County and City Health Officials, National Coalition of STD Directors, and National Public Health Information Coalition. The National Network of STD Prevention Training Centers and others from curated CDC lists also may be a source for participants who have visited the CDC DSTDP website.
All groups will be recruited to include a mixed education level and geographic mix as well.
Adult Populations Disproportionately Affected by STDs
The evaluation’s stated recruitment categories (age, MSM, pregnant women, and education) will be used to identify and recruit participants to achieve a balanced split across the adult subgroups (e.g., 6 young adults, 6 pregnant women, 6 MSM) with representation across different types of locations (urbanicity), education and racial/ethnic groups. Interviews will be conducted with 6 people from each subgroup with half of those recruited per group having an education level of high school or less and half with some college or more (see Table 1). All participants will have had sexual contact in the past year, be likely to use the internet to access health information and be able to access the CDC DSTDP website on a computer. Those recruited will include a mix of participants who are of different ages within ranges (for those 18-24 years old), race/ethnicity and types of communities (urban, suburban, and rural). Only one participant per household can be included in the evaluation.
Table 1: Adults by Type and Education Level
TYPE |
EDUCATION LEVEL |
|
|
High School or Less |
Some College or More |
18-24*
|
3 interviews |
3 interviews |
MSM |
3 interviews |
3 interviews |
Pregnant Women
|
3 interviews |
3 interviews |
*18 to 24-year-old groups will be skewed to over-represent younger participants.
Healthcare providers
To ensure that information is gathered from different types of providers, respondents will be segmented by their role (i.e. lead, support, and specialist providers). Lead providers are those who are primarily responsible for patient care and may include these roles: physicians, physician assistants, and nurse practitioners. Support providers are those who support patient care and may include these roles: nurses, health educators, or others who play a significant health education role. Specialists are those who deal exclusively with STD prevention, testing, or treatment. Table 2 presents the segmentation for healthcare providers. Participants will also be screened to ensure a mix of urbanicity of their location, type of work setting (i.e. public vs. private), age, gender, and familiarity with the CDC DSTDP site. We intend to recruit equal numbers of lead and support providers; however, we will screen for alternatives and collapse these categories if recruitment does not progress as expected.
To be included, providers will self-report that their work includes STD prevention, testing, or treatment, that they are likely to use the internet to access the type of content and information found on the CDC DSTDP website, and that they are able to access the CDC DSTDP website on a computer.
To limit the time required for any one healthcare provider to no more than 30 minutes, the full set of questions has been divided into two separate guides and recruited providers will be only asked to answer questions from one of the two healthcare provider guides. The first group of providers will include 15 individuals (6 lead providers, 6 support providers, and 3 STD specialists) and the second group will include 10 providers (4 lead providers, 4 support providers, and 2 STD specialists). The providers from each category will be selected to ensure as much diversity as possible (i.e. geographic, organizational affiliation).
Table 2: Segmentation for Healthcare Providers
ROLE AND SETTING |
NUMBER OF INTERVIEWS |
Lead Providers |
10 interviews
|
Support Providers
|
10 interviews
|
STD Specialists |
5 interviews |
Public health professionals
To ensure that information is gathered from different types of public health professionals, respondents will be segmented by their role (STD directors, public health information officers, and advocates) and whether they work at the state or local level (although advocates may include those working at the national level as well). “STD directors” are those in a leadership position within STD programs in state/territorial or local public health departments. “Public health information officers” are those in state/territorial or local public health departments who can implement public education campaigns and communication efforts that encourage STD prevention interventions. “Advocates” are those who work at national, state, or local non-profit organizations with STD prevention, testing, or treatment within their mission, such as community health centers, LGBT advocacy groups, etc. Table 3 presents the segmentation across all categories of public health professionals. Participants will also be screened to ensure a mix of urbanicity of their location, age, gender, and familiarity with the CDC DSTDP site. We intend to recruit equal number of state and local public health professionals; however, we will screen for alternatives and collapse these categories if recruitment does not progress as expected.
To be included, public health professionals will self-report that their work includes STD prevention, testing, or treatment, that they are likely to use the internet to access the type of content and information found on the CDC DSTDP website, and that they are able to access the CDC DSTDP website on a computer.
Table 3: Segmentation for Public Health Professionals
PUBLIC HEALTH ROLE |
STATE |
LOCAL |
STD Directors
|
3 interviews
|
3 interviews |
Public Health Information Officers |
3 interviews
|
3 interviews |
Advocates |
3 interviews
|
|
All data collection activities will be conducted by the appointed evaluation team. As is depicted in Tables 1-3 above, we propose to conduct 18 telephone in-depth interviews with members of adult populations disproportionately affected by STDs, 25 telephone in-depth interviews with healthcare providers, and 15 telephone in-depth interviews with public health professionals. Interviews with adults and public health professionals will gather information on why and how each of these three respondent groups need and use websites to gather information on STDs, feedback on the content and resources available on the CDC DSTDP website, and feedback on the architecture of the site and on the ease with which users are able to navigate through it. Respondents will be asked about their need for and use of CDC’s DSTDP website and other similar sites and also about the content and their feedback on the website.
As stated above, to limit the time required for any one healthcare provider to no more than 30 minutes, the full set of questions has been divided into two separate guides. One group of providers (15 people) will respond to questions about their need for and use of the CDC DTSTP website and will conduct a card sort activity to capture how they would categorize the website’s information. Another group (10 people) will focus on site exploration, specified tasks, and impressions of the site.
Information will be collected using standard open-ended questions along with two specific activities (scenarios and card sorting) that will allow the evaluation team to better understand the respondents’ interaction with the site as well as how they would expect information on the site to be organized. To understand their user experience, the evaluation team will provide scenarios to participants to learn how they use (or would use) the website and to receive feedback on content. Open-ended initial scenarios are used to encourage free exploration (e.g., “What would you do first if you arrived on this website?”) and some more general scenarios are used to see how individuals approach common tasks (“Please show me how you would find fact sheets about STDs to share with your patients”). If time permits, an optional set of scenarios will be included that are specifically selected for each of the different respondent groups and focus on gaining insight on the existing site architecture and determining which content is most valuable.
A card sort activity is included to explore how participants think about and mentally organize the type of information the CDC DSTDP site provides. Participants will be asked to sort online “cards” (topics) into groups that are most intuitive to them using an online card sorting tool (e.g., OptimalSort), which will provide the most efficient data collection and analysis process. Respondents will label the groupings of cards they’ve created.
Only individuals 18 years and older will be involved in the proposed evaluation. Written consent will be obtained for all participants when confirming their participation in the evaluation. Consent forms will be written in plain language. Details about participating in the evaluation will be described in the consent form and explained verbally. A copy of the consent form will be provided to all participants and include telephone numbers of the evaluation staff, who participants can call with questions or concerns. Only qualified personnel will consent participants. Before each interview, moderators will again inform participants that the session will be audio recorded for the purposes of accurate reporting. Participants will be asked to verbally consent to being recorded. If any interview participants do not want to be recorded, they will still be able to participate in the evaluation (and an audio recording will not be made of the interview).
The following data collection instruments will be used in this evaluation:
Eligibility Screeners for Adults (Attachment E1), Healthcare Providers (Attachment E2), and Public Health Professionals (Attachment E3): Eligibility screeners will be used to identify eligible respondents and take approximately 10 minutes to complete. If eligible, participants will be scheduled for an interview.
In-Depth Interview Guides: All interviews will be conducted by staff using semi-structured interview discussion guides. (Attachments – F1: Adult Interview Guide; F2: Healthcare Provider Interview Guide A; F3: Healthcare Provider Interview Guide B; F4: Public Health Professional Interview Guide). All audio-recordings will be transcribed, and staff will maintain all transcripts on password protected/encrypted computers.
Data Analysis
Interview notes will be transcribed and analyzed for the identification of recurrent themes following Crabtree and Miller’s 5 step “interpretive process.” Transcripts will be read to identify common themes, codes will be developed, and ~10% of data will be double coded and inter-rater reliability assessed. Coded text will be extracted and organized and read to identify emergent themes. A sequential, empirically-informed approach will be used to synthesize findings to develop recommendations for improving the CDC DSTDP website. Additionally, the recommendations will be informed by the analysis of the scenario and card-sort activities. Responses to navigation scenarios will be organized by navigation path followed and any additional comments will be transcribed. Responses for the card sort exercise will be analyzed using use an online card sorting tool, such as OptimalSort, to provide the most efficient analysis process.
Specific steps will be followed:
Step 1: Use qualitative data from interviews, scenario observations, and card sort activities to develop a summary of findings.
Step 2: Synthesize qualitative findings to develop a set of recommendations for improving the CDC DSTDP website.
Step 3: Present key findings and recommendations to DSTDP staff, obtain additional suggestions for refinement, and prepare a final report. The final data DSTDP will receive will be de-identified and aggregated.
Consent documents and data collection tools will be retained in password-protected computer files and will only be accessible to senior investigators or designated staff. All participants will be identified by ID numbers only. Participants’ names will not be included on materials. A master list of participant names linked with IDs will be kept in a password-protected computer file. All computer records will be protected by standard measures that limit data access to authorized personnel and will be identifiable only by participants’ IDs. Digitally recorded qualitative data will be stored on a secure password protected server following each interview. Once transcribed, the original audio file will be deleted. Transcription and data analysis will be done through secure password-protected server. All of these procedures will be adhered to in the proposed evaluation.
The following procedures will be used to maximize cooperation and to achieve the desired response rate:
All recruitment materials indicate the voluntary nature of participation in the evaluation and the value of providing feedback to improve the CDC DSTDP website.
Tokens of appreciation in the form of an online gift card, will be emailed to participants upon completion of their interviews. Adults will receive a $40 gift card, healthcare providers will receive a $100 gift card, and public health professionals will receive a $30 gift card.
There will be an adequate recruiting period before the start of fielding (as well as ongoing recruiting as needed during fielding period).
There will be an availability of sessions at time slots that are typically popular among healthcare providers and public health professionals—for example, early morning, evenings, and lunch.
The evaluation team includes experts with experience in conducting usability research with healthcare providers, public health professionals, and other audiences and research on sensitive topics.
Exhibit B5.1 below lists the team members who were consulted on the aspects of evaluation design and team members who will be collecting and analyzing the data. Please note: The CDC staff are primarily responsible for providing technical assistance in the design and implementation of this evaluation; assisting in the development of the protocol and data collection instruments for CDC Project Determination; working with the evaluation team to facilitate appropriate activities; and presenting findings at meetings and in publications. The data are primarily qualitative in nature and will be analyzed accordingly.
CDC staff will not be engaged in the direct collection of information. CDC staff will neither collect data from nor interact with participants. Data will be collected by members of the external evaluation team. No individual identifiers will be linkable to collected data, and no individually identifiable private information will be shared with or accessible by CDC staff.
Exhibit B5.1: Statistical Consultants
Team Member |
Organization |
|
Nikki Mayes |
CDC |
Grz3@cdc.gov |
Salina Smith |
CDC |
|
Tiffani |
CDC |
itb0@cdc.gov |
Joanna Barbour |
Trillium |
joanna.barbour@trilliuminfo.com |
Patrick Cook |
Trillium |
patrick.cook@trilliuminfo.com |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Bonds, Constance (CDC/OID/NCHHSTP) |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |