SUPPORTING STATEMENT
Part B
Evaluation of the Educating
the Educator (EtE) Project Train-the-Trainer
Workshop and
Shared Decisionmaking (SDM) Tools to Support the Dissemination of
Patient-Centered Outcomes Research (PCOR)
November 29, 2013
Agency for Healthcare Research and Quality (AHRQ)
Table of contents
B. Collection of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 4
The respondent universe for the evaluation of the AHRQ Educating the Educator (EtE) train-the-trainer workshop, and the new tools developed for this project, will be comprised of the following:
Individual health care professionals who participate in the EtE workshop training
Individuals who access the new tools available on the AHRQ Web site
For the EtE workshop training, no sampling strategy will be employed, as AHRQ seeks to obtain feedback from everyone who has participated in the training program (i.e., a census survey).
For the new tools survey, no systematic sampling strategy will be employed. Rather, we will be conducting a non-random convenience sample survey, with respondents consisting of individuals who visit the new tools landing page on the AHRQ Web site during the time the survey is fielded, and who are willing to complete the brief survey.
1.a. Respondents Associated with EtE Training
There are two categories of individuals who participate in the AHRQ EtE workshop training:
Primary trainees—Individuals who participate in the AHRQ-sponsored, in-person, train-the-trainer workshop
Secondary trainees—Individuals who receive training secondarily from primary trainees
Primary Trainee Sample Frame
Primary trainees are individuals who register for and participate in the AHRQ-sponsored EtE train-the-trainer workshop. AHRQ’s contractor, AFYA, Inc., will conduct 10 in-person train-the-trainer workshop sessions each year for 3 years. A total of 25 to 50 participants will participate in each train-the-trainer workshop for a maximum of 500 participants per year, or 1,500 participants over 3 years. Primary trainees will reflect individuals from a variety of health care disciplines, including physicians, physician assistants, nurse practitioners, nurses, pharmacists, certified health education specialists, diabetes educators, and others who have an interest in the training content and in training other health care professionals, most likely from their own practice settings or organizations.
All EtE train-the-trainer workshop participants (primary trainees) will be asked to complete the pre- and post-training surveys. Because receipt of continuing education (CE) credits associated with this workshop training will be contingent upon the completion of the immediate pre- and post-training surveys, we anticipate that all primary trainees will complete these two surveys. Primary trainees also will be asked to complete 6-month and 12-month post-training surveys as part of the evaluation. Given that completion of these surveys is not required for CE credits, we expect that there will be some drop-out from the evaluation at these time points. However, because primary trainees will have participated in training in order to train others, we anticipate that their interest in continued participation will be relatively high. We expect that at least 80 percent (~1,200) of primary trainees will complete the 6-month post-training survey. We expect that an additional 80 percent of the 6-month respondents (~960) will complete the 12-month post-training survey.
Secondary Trainee Sample Frame
As with primary trainees, secondary trainees will reflect individuals from a variety of health care disciplines, including physicians, physician assistants, nurse practitioners, nurses, pharmacists, certified health education specialists, diabetes educators, and others who have received EtE training secondarily from a primary trainee, most likely someone from their own practice settings or organization.
It is difficult to estimate how many secondary trainees will be available to participate in the training evaluation. We have assumed that each primary trainee will administer the EtE workshop training content to at least 4 other individuals on average, for a total of 6,000 secondary training participants over 3 years.
Secondary trainees who received training from primary trainees also will be able to go online and complete an enduring educational activity associated with the training in order to claim CE credits for undergoing the EtE training. A one-time annual survey will be administered to all secondary participants who claim CE credit for their participation in the training. We anticipate that approximately one-half of these individuals (~3000) will agree to take the one-time annual evaluation survey over the 3-year period of this project.
1.b. Respondents Associated With the New Tools
As noted above, the respondent universe for the new tools survey will be comprised of individuals who visit the new tools landing page on the AHRQ Web site where the new tools developed for this project will be hosted. The quarterly survey will be available online for a period of 2 weeks with each quarterly administration, thus the respondent universe for this survey will be limited to those individuals visiting the AHRQ new tools landing page during this timeframe.
Based on AHRQ’s historical Web site usage statistics data (specifically ‘product views’) for comparable Web site pages (~1,200 page views over a 2-week time frame), we estimate that approximately 100 individuals (8%) will opt to complete the brief survey with each quarterly fielding, for a total of 1,200 participants over the 3-year evaluation period. This sample will reflect a convenience sample (i.e., those who choose to participate and are willing to complete our survey). This survey will not constitute a valid scientific survey because it will employ a non-probability convenience sample and the analytical objectives of this part of the evaluation are descriptive in nature (e.g., to learn about the types of health care professionals who are using the new tools, how they are using the tools, what features they like or dislike with new tools, and if new tools are considered to be useful among individuals who have not participated in the EtE training).
The pre- and post-training surveys will be administered as paper and pencil surveys immediately before and following the in-person train-the-trainer workshop sessions. Data will be electronically entered into a database by project staff.
The 6-month and 12-post-training survey of primary trainees will be administered electronically as Web-based surveys. The 6-month and 12-month surveys will be accessible to primary training participants 24 hours a day for a total of 30 days. Upon entrance into the electronic survey, respondents will view an introduction page that explains the survey objectives and stresses the importance of participation. Respondents will be able to easily respond to the survey items by clicking on pre-coded options for closed-ended items and typing in "boxes" for any open-ended items.
AHRQ contractor staff will coordinate and manage sending out an e-mail invitation with a link to a Web-based survey for participants to complete on or about their 6-month and 12-month post-training time points. Invited respondents who have not responded to the survey within 3 days of the initial invitation will be contacted via e-mail reminding them of the opportunity to participate and the importance of their feedback for the EtE evaluation. The reminder notice sent via e-mail will provide the hyperlink to access the survey, the estimated time (in minutes) it will take to complete the survey, and the deadline for submitting responses.
The annual secondary trainee survey will likewise be administered electronically as a Web-based survey. AHRQ contractor staff will coordinate and manage sending out an e-mail invitation with a link to the one-time annually administered Web-based survey for those claiming their CE credits for participating in a secondary training session.
The quarterly new tool survey will likewise be administered as a Web-based survey. AHRQ contractor staff will enable a link to the survey instrument on the AHRQ Web page where new tools are posted. This Web-based survey will be accessible to AHRQ Web site users (specifically those individuals who access the Web page where new tools are available) 24 hours a day for a total of 2 weeks each quarter. Upon entrance into the survey, respondents will view an introduction page that explains the survey objectives and stresses the importance of participation. Following the access page will be a page describing specific instructions on how to complete the survey. Respondents will be able to easily respond to the survey items by clicking on pre-coded options for closed-ended items and typing in “boxes” for open-ended items.
The project team, led by The Lewin Group (a subcontractor to AFYA, Inc., responsible for implementing the evaluation component of the AHRQ EtE project in consultation with AFYA staff), will be responsible for sending out electronic requests by e-mail to EtE training participants to complete the evaluation instruments. Lewin staff also will be responsible for collecting, analyzing, and summarizing the data. Lewin has extensive experience conducting evaluations and surveys; consequently, facilities and procedures have been developed to protect stored data. All Lewin project staff who are in contact with human subjects undergo training about maintaining the confidentiality of information. Any databases created by Lewin will be stored in protected files, with only project staff having access.
Following data collection, survey responses will be compiled and assessed formally for data quality to produce a finalized database for statistical analyses. Incomplete response data poses a substantial threat to confident interpretation and generalization of the study results. We will exclude surveys where respondents answered fewer than 25 percent of the total number of questions.
We will employ a number of strategies to maximize response rates, including keeping all evaluation instruments short and simple. Additionally, primary trainees who participate in the train-the-trainer workshops will be given time at both the beginning and the end of the workshop sessions to complete the instruments. Moreover, completion of these assessments will be required for collecting CE credits for participating.
E-mail requests, which include a hyperlink to the Web-based survey, will be sent to both primary trainees (for the 6-month and the 12-month survey) and secondary trainees to complete the Web-based evaluation surveys. Strategies that will be used to enhance the response likelihood include the following:
E-mail requests will be individualized by respondent name.
The request will include friendly and inviting language and include an estimate of the short time required to complete the survey.
These e-mail requests will convey the importance of the evaluation activities to the success of the project and to AHRQ, and will indicate that the participants’ responses will not be identified to any Government Agency.
Web-based survey respondents also will be provided with a “resume” capability that allows them to break off the session mid-survey and then return to the survey at a later time to complete it without losing previously entered data.
Reminder e-mail notices will be sent to non-responders beginning 3 days after the initial invitation. A total of 3 reminders e-mails (1 week apart), which include a hyperlink to the Web-based survey, will be sent to non-responders.
Additionally, among primary trainees who have not responded to the survey at 6 months and 12 months, after three reminder notices have been sent, a project staff member will initiate one telephone communication per non-responder, encouraging them to participate in the AHRQ Web-based survey of primary training participants. If at any point, an invited respondent refuses to participate, they will not be contacted again through any means.
For the EtE project, AHRQ’s contractor AFYA also will be implementing a learning network and providing ongoing technical assistance to primary workshop participants who request it. These technical support activities also will be used to encourage primary participants to respond to the evaluation data requests so that AHRQ can make improvements to the program to meet their needs when using PCOR resources and when training other secondary participants.
For the EtE training survey of secondary trainees, we will perform a comparison of early or initial survey respondents to responses obtained from individuals who respond late (following the last e-mail notification; late respondents) on key survey metrics. Studies1 have shown that late-respondents, or those who respond after several attempts, tend to have some similarities with individuals who do not respond (non-respondents). Any differences between these subsets of survey respondents will provide a measure of the potential non-response bias.
In addition, among primary trainees, we will assess whether or not there are differences at baseline (both pre- and immediate post-training surveys) between individuals who respond at 6-months and 12-months to those who don’t respond at these time points. Differences in baseline measures between those who do and don’t respond at 6 months and 12 months may reflect non-response bias and will shed light on potential differences between these groups of trainees.
The evaluation instruments have been reviewed extensively by the research team. Additionally, AFYA will conduct a pre-testing of each survey with a sub-sample of two or three health care professionals to assess the clarity of questions. During the pre-testing, we will ask the volunteers to “think aloud” as he or she answers each question. In doing so, the research analyst is able to examine the thought processes of the respondent as he or she hears, interprets, and decides on an answer. The results of the pre-testing will be used to refine the survey prior to implementation. In the event that fine tuning is required, OMB will be notified in a memorandum with a copy of the final version of the Web-based survey.
We also will test the data capture procedures to ensure that the Web-enabled surveys capture and render data correctly. Two members of our project team will do this by manually completing 10 surveys (on hard copy), in parallel with our online data entry component and compare the outputs to ensure that all data were captured correctly.
The Lewin Group, as a subcontractor to AFYA Inc., will serve as the primary consultant for statistical aspects of the design and analysis for the evaluation of the EtE training program for AHRQ. Lewin has conducted numerous comparable studies and analyses, comprehensive evaluations, and technical assistance for Federal Government agencies. Lewin has partnered with Federal, State, and local governments, and foundations to evaluate and implement key health initiatives.
The Project Director and Evaluation Task Leader at AFYA Inc., is Michelle Bieber-Tregear, Ph.D. (telephone: 484-868-0866; e-mail: mtregear@afyinc.com). The Evaluation Lead at The Lewin Group is Christine Jones, M.S., M.P.H. (telephone: 703-269-5743; e-mail: christine.jones@lewin.com).
1 Lahaut VM, Jansen HA, van de Mheen D, Garretsen HF, Verdurmen JE, van Dijk A. Estimating non-response bias in a survey on alcohol consumption: comparison of response waves. Alcohol. 2003 Mar-Apr;38(2):128-34
Bose, J. (2001) Nonresponse bias analyses at the national center for Education statistics. Proceedings of Statistics, Canada Symposium 2001: Achieving Data Quality in a Statistical Agency: a Methodological Perspective. Pgs: 8
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT |
| Author | Jenice James |
| Last Modified By | DHHS |
| File Modified | 2014-02-06 |
| File Created | 2014-02-06 |