OMB Clearance Package
SUPPORTING STATEMENT
Part B
Evaluation of the National Guideline Clearinghouse™
August 12, 2010
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 5
3. Methods to Maximize Response Rates 6
The National Guideline Clearinghouse™ (NGC) Evaluation survey will be distributed to individuals involved in the provision of health care to patients; health care information delivery; health care education; health care research; guideline development, dissemination, evaluation, or implementation; health care policy makers; and/or health care purchasers. Individuals targeted for the focus groups and key informant interviews will also be composed of members involved in the above stated activities.
Survey: The online survey will be fielded via two means. First, the AHRQ Gov-Delivery E-mail Administrator will email an announcement that will include a link to the survey to subscribers of 21 AHRQ-sponsored Gov Delivery accounts listed below:
AHRQ's Electronic Newsletter
AHRQ's Events and Announcements
What’s New at AHRQ
AHRQ's Effective Health Care Spotlight
AHRQ's e-mail list for: Evidence-Based Practice
AHRQ's National Guideline Clearinghouse
AHRQ Prevention Program
AHRQ's Centers for Education & Research on Therapeutics
AHRQ's Health IT
AHRQ's Healthcare Innovations Exchange
AHRQ's Web M&M (Mortality and Morbidity)
AHRQ's Patient Safety Net
AHRQ's National Quality Measures Clearinghouse
AHRQ's e-mail list with focus: Long Term Care
AHRQ's e-mail list with focus: Mental Health
AHRQ's Practice Based Research Networks (PBRNs)
AHRQ's e-mail list with focus: Child and Adolescent Health
AHRQ's e-mail list with focus: Disabilities
AHRQ's e-mail list with focus: Minority Health
AHRQ's e-mail list with focus: Women’s Health
AHRQ's Research Activities Online Newsletter
Currently, there are approximately 150,000 individuals registered through this service who have registered to receive e-mail alerts regarding the happenings, new products, services, and other health-related information from the Agency.
If individuals are registered for more than one of the e-mail accounts listed above, the Gov Delivery system is able to identify duplicates to ensure that those individuals will only receive one email announcement.
Second, we will complement our survey respondent sample by partnering with two Medical Associations and the National Network of Medical Libraries (of the National Library of Medicine) to disseminate the Web-based survey via e-mail to their respective members (who are relevant stakeholders). The medical associations specifically include the American Medical Association (AMA; ~250,000 members) and America’s Health Insurance Plans (AHIP; ~1,200 members), AHRQ’s original partnering organizations in the development of the NGC.
Both approaches to fielding the Web-based survey are designed to target both current NGC users and non-users from NGC's intended stakeholder groups (e.g., physicians, nurses, and other health professionals, health care providers, health plans, integrated delivery systems, purchasers, researchers, medical informaticians, etc.).
No sampling strategy will be employed, as AHRQ seeks to obtain feedback from as many registered individuals as possible. We will be conducting a nonrandom convenience sample survey, meaning our sample will consist of individuals who are easiest to recruit (from the selected health-service related e-mail subscription lists) and willing to complete our survey. The survey will not constitute a valid scientific survey because it will employ a non-probability convenience sample and the analytical objectives of this evaluation are descriptive in nature. Best practice and extant literature explain that determining acceptable response rates for convenience sample surveys is extremely difficult and almost impossible (Kelley, Clark, Brown & Sitzia, 2003)4. However, research has shown that, for special populations that regularly use the Internet (which is our target NGC audience), the Web has been found to be a useful means of conducting research (Couper, Traugott, and Lamias, 2001)5.
In addition, we will conduct focus groups and key informant interviews to obtain qualitative information that will be used to elaborate on the information gathered from the survey questionnaire, and to target specific stakeholder groups. We will conduct a total of 11 focus groups (in order to obtain information from the varied stakeholders of interest in this evaluation), and 30 key informant interviews. The individual focus groups will be stakeholder-specific and will comprise individuals from four of the seven stakeholder groups (i.e., guideline developers [2 focus groups], medical librarians [1 focus group], informatics specialists [1 focus group], and clinicians/students [7 focus groups]). We have prepared a Focus Group Discussion Guide (See Attachment D) which contains a core set of questions to be asked of all participants across all focus groups. We have also included additional, stakeholder-specific questions at the end of this guide which will only be presented to individuals who comprise a given stakeholder group. For example, focus groups representing the “guideline developer” stakeholder group will include an additional set of questions which pertain to NGC’s influence on their guideline development, dissemination, and implementation efforts. We plan to conduct approximately seven focus groups with the clinician/medical student stakeholder group. The primary reason being that this group is diverse in nature consisting of physicians, nurses, other health care professionals, as well as medical, nursing, and pharmacy students. This key stakeholder group is also large in number relative to the other NGC stakeholder groups. The seven focus groups are intended to capture the variety of responses from of this particular group as well as the large number of individuals making up this group.
Key informant interviews will also be used to obtain additional data from the various stakeholder groups. We have prepared a Key Informant Interview Discussion Guide (See Attachment E) which contains a core set of questions to be asked of all individuals interviewed. We have also included additional, stakeholder-specific questions at the end of this guide which will only be presented to individuals who comprise a given stakeholder group. For example, a key informant representing the “researcher” stakeholder groups will include an additional set of questions that pertain to how the availability and use of NGC has influenced their data collection activities and their approaches to identifying guidelines needed in their research efforts.
We will use conferences attended by NGC stakeholders, our team’s contacts with multiple medical societies, and our consultant databases of health services researchers, physicians, and other clinicians as a mechanism for recruiting focus group participants and key informant interview participants. We will also use the information obtained in our environmental scan and suggestions from the evaluation's Participant Evaluation Team (PET) to identify potential participants. The objective is to select a range of focus group and key informant interview participants that represent the NGC stakeholder groups.
The survey will be administered to all individuals who are currently registered to the AHRQ GovDelivery accounts listed above and to members of AMA, AHIP, and NNLM using the Web-based survey. The invitational correspondence to various email subscribers are presented as Attachment C. The correspondence will contain a hyperlink to easily access the online survey.
The Web-based survey will be accessible to our target audience 24 hours a day for a total of 4 weeks. Upon entrance into the survey, respondents will view an introduction page that explains the survey objectives and stresses the importance of participation. Following the access page will be a page describing specific instructions on how to complete the survey. Respondents will be able to easily respond to the survey items by clicking on pre-coded options for closed-ended items and typing in “boxes” for open-ended items. The survey also contains a “skip logic” pattern that allows respondents to only complete relevant questions.
During the data collection period, invited respondents will be contacted via email reminding them of the opportunity to participate and the importance of their feedback regarding the NGC. The reminder email notice sent via email will provide the hyperlink to access the survey, the estimated time (in minutes) it will take to complete the survey, and the impending deadline for submission of their responses
Following data collection, questionnaire responses will be compiled and assessed formally for data quality to produce a finalized database for statistical analyses. Incomplete response data poses a substantial threat to confident interpretation and generalization of the study results. We will exclude surveys where respondents answered fewer than 25% of the total number of questions.
For the focus groups and key informant interviews guides have been developed to help the moderator in guiding the focus groups and key informant interviews and will not be read verbatim. Not all questions apply to all focus group participants and key informants. Each group/participant will have a core set of questions followed by a specific set of questions relating to the NGC stakeholder group which they represent. Each NGC stakeholder group may offer unique feedback regarding the NGC and this is the reason for the conducting focus groups/key informant interviews with representatives from multiple NGC stakeholder groups. In this way, we can maximize the breadth of data to be used to derive NGC’s impact on guideline development, use, and implementation.
Focus groups will be conducted in-person at AFYA’s offices in Laurel, MD and/or at one or more professional conference for the stakeholder groups. Key informant interviews will be conducted in-person or by phone when an in-person interview is not possible. An experienced AFYA or Lewin focus group moderator/interviewer will conduct all focus groups and key informant interviews. The focus group will last approximately 90 minutes and the key informant interviews will last approximately 60 minutes. Focus groups and key informant interviews will be digitally recorded and transcribed deleting any use of the participants’ names. Focus group and interview transcripts and any additional notes will be stored on secure computer servers at AFYA.
The text files for the transcripts will be used to construct a database in a qualitative analysis software package. The software will also be used for coding and analysis. A preliminary code list will be defined but it will be revised based on the actual review of the data after identifying any additional common themes. The revised coding scheme will be used to code the appropriate text fragments in each of the transcripts. Coding will be performed by two research associates. Intercoder agreement will be tested by double coding an initial set of interviews and once 80% agreement has been reached, coding will proceed. Analysis will be based on text analysis and analysis of the results of queries using the software that will identify text fragments with particular codes of interest. A memo for each of the major code or topic of interest will be prepared and these will be used to draft the summary document.
For the NGC Evaluation Survey the expected response rate is 10% - 68% based on estimates in the literature for comparable e-mail-initiated, online surveys (RAND, 2002)6. The response rate will depend on the overall willingness of the targeted individuals who receive AHRQ- and clinical guideline-related information to respond to the short survey. To improve the response rate the survey will only be sent to individuals who have agreed to receive information from the identified dissemination methods. As is commonly the case with all surveys, the acceptable response rate is often not achieved initially. To effectively bolster this response rate, the AFYA/Lewin Team will use time-staggered notices via email of the opportunity to participate in the survey. More specifically, the survey announcement will be sent out a total of three times over a 4-week implementation period. The initial announcement will be followed by two additional announcements (at the midpoint, and 1 week prior to the end of the survey).
Inviting language will be used in the e-mail notice that is distributed to potential respondents to encourage their participation. Providing information which lays out the salience of issues for which the evaluation concerns has been found to have a strong positive correlation with response rates of e-mail and Web-based surveys (Sheehan & McMillan, 1999).7 The language to be used in the E-mail notification of the survey is included in Attachment C.
In addition, as noted above, e-mail invitations will only be sent to individuals who have subscribed to receive information from the respective sources.
Second, we will partner with medical associations and the National Network of Medical Libraries (of the National Library of Medicine; ~1000 subscribers) to disseminate the survey to relevant stakeholders. The medical associations specifically include the American Medical Association (AMA; ~250,000) and America’s Health Insurance Plans (AHIP; ~1,200 subscribers), AHRQ’s original partnering organizations in the development of the NGC.
To increase participation in the focus group and key informant interviews, we will utilize our PET members to help recruit participants. We will also hold the focus group and key informant interviews at locations and times that are most convenient for the majority of participants.
We will perform a comparison of early or initial survey respondents to responses obtained from individuals who respond late (following the last e-mail notification; late respondents) on key survey metrics. Studies8,9 have shown that late-respondents, or those who respond after several attempts, tend to have some similarities with individuals who do not respond (non-respondents). Any differences between these subsets of survey respondents will provide a measure of the potential non-response bias. In addition, because we are using a mixed-methods approach (i.e., survey, focus groups, and interviews) to collect data for the relevant metrics, we will also compare responses obtained from the focus groups and key informant interviews to the survey responses to assess the degree to which the survey data may reflect non-response bias.
In terms of assessing non-response bias as a function of the different stakeholder groups we intend to target, unfortunately, we do not have descriptive information in our email lists that would let us identify the stakeholder group or groups that an individual email is associated with. We do know, however, that certain lists are predominantly associated with particular stakeholders (e.g., the AMA list will be largely made up of physicians and non-NGC users; the NNLM list will have a preponderance of medical librarians). We may be able to detect response bias by looking at differences in the response rates by the list that the e-mail was drawn from and then triangulate patterns to identify if non-response is more or less likely to be associated with different types of member organizations (those representing providers, researchers, non-users, etc).
The last question of the survey asks users if they have already completed the survey within the previous 4 weeks. Answers to this question are intended to allow us to gauge the degree to which multiple response bias affects the data.
In order to test Web-based survey procedures, AFYA will conduct a pre-testing of the Web-based survey with a sub-sample of no more than nine potential respondents, including PET members. Our PET members consist of experts representing multiple NGC stakeholder groups. During the pre-testing, we will ask the volunteer to “think aloud” as he or she answers each question. In doing so, the research analyst is able to examine the thought processes of the respondent as he or she hears, interprets, and decides on an answer. The results of the pre-testing will be used to refine the survey prior to field-testing. In the event that fine tuning is required, OMB will be notified in a memorandum with a copy of the final version of the Web-based survey.
We will also test the data capture procedures to ensure Web-enabled survey captures and renders correctly. Two members of our project team will do this by manually completing 10 surveys (on hard copy), in parallel with our online data entry component and compare the outputs to ensure that all data were captured correctly.
AFYA, Inc. will serve as the primary consultants for statistical aspects of the design and analysis of the Web-based survey. Dr. Ajay Bhardwaj, Division Director of AFYA’s Analytical and Technical Services Division is the primary point of contact for statistical design and analyses. He can be reached at abhardwaj@afyainc.com or 301-957-3040.
4 Kelley, K., Clark, B., Brown, V., & Sitzia, J. (2003). Good practice in the conduct and reporting of survey research. International Journal for Quality in Health Care, 15, 261-266.
5 Couper, M., Traugott, M., Lamias, M., (2001). Web survey design and administration. Public Opinion Quarterly, 65, 230-253.
6 Schonlau, M., Fricker, RD., & Elliott, MN. (2002). Conducting Research Surveys via E-mail and the Web. RAND Distribution Services. Pg. 142 (ISBN/EAN: 0-8330-3110-4)
7 Sheehan, K. B., & McMillan, S. J. (1999). Response variation in e-mail surveys: An exploration. Journal of Advertising Research, 39 (4), 45-54.
8 Lahaut VM, Jansen HA, van de Mheen D, Garretsen HF, Verdurmen JE, van Dijk A. Estimating non-response bias in a survey on alcohol consumption: comparison of response waves. Alcohol. 2003 Mar-Apr;38(2):128-34
9 Bose, J. (2001) Nonresponse bias analyses at the national center for Education statistics. Proceedings of Statistics, Canada Symposium 2001: Achieving Data Quality in a Statistical Agency: a Methodological Perspective. Pgs: 8
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | Jenice James |
| Last Modified By | william.carroll |
| File Modified | 2010-08-13 |
| File Created | 2010-08-13 |