AHRQ OMB Part B_5-18-20

SUPPORTING STATEMENT

Part B

Evaluation of Learning Health Systems K12 Training Program

Version: May 18, 2020

Agency for Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collections of Information Employing Statistical Methods 1

1. Respondent Universe and Sampling Methods 1

2. Information Collection Procedures 2

3. Methods to Maximize Response Rates 2

4. Test of Procedures 4

5. Statistical Consultants 5

List of Appendices 6

2. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS

The purpose of this evaluation is to assess the overall achievement of the LHS K12 training program’s objectives, outcomes, and impact, as well as the program’s value to its stakeholders. The information collected through this data collection will allow AHRQ to improve the LHS K12 program and identify whether results correspond to intentional changes in program strategy and implementation.

Respondents include scholars, their health system advisors, and program directors who receive and participate in the grant awards. Therefore, this study does not require a nationally representative sample or implement any sampling strategy. The evaluation will include two types of data collection: semi-structured interviews with scholars who are close to completing the LHS K12 training program, their health system advisors, and program directors of each of the 11 institutions; and surveys of health system advisors.

1. Respondent Universe and Sampling Methods

Table 1 provides the respondent universe size, anticipated response rates, and estimated number of respondents (total and annually) for each data collection activity. The total estimated number of respondents¹ is 439 individuals. This total includes 123 scholars interviewed (approximately 41 scholars each year), 116 health system advisors interviewed (approximately 39 health system advisors each year), 10 program directors (approximately 4 program directors each year), and 190 health system advisors surveyed (approximately 63 health system advisors each year). As indicated in the table, the expected response rate varies by data collection activity, with a higher response rate estimated for respondents who are more engaged with the program (scholars and program directors) and a slightly lower response rate estimated for respondents who are less directly involved with the program (health system advisors).

Table 1. AHRQ’s Expected Response Rates

Information Collection Activity	Respondent Universe Size	Expected Response Rate	Total Estimated Number of Respondents	Annual Estimated Number of Respondents
1. Scholar Interviews	137	90%	123	41
2. Health System Advisor Interviews	137	85%	116	39
3. Program Director Interviews	11	90%	10	4
4. Health System Advisor Surveys	237	80%	190	63
Total			439	147

2. Information Collection Procedures

2.a Statistical methodology for stratification and sample selection

This study is not nationally representative; therefore, concern regarding the statistical methodology for both stratification and sample selection is not applicable.

2.b Estimation procedure and degree of accuracy needed for the purpose described in the justification

This is not a nationally representative study; therefore, concern regarding the estimation procedure is not applicable.

2.c Unusual problems requiring specialized sampling procedures

AHRQ does not anticipate unusual problems that require the usage of specialized sampling procedures.

2.d Any use of periodic (less frequent than annual) data collection cycles to reduce burden

AHRQ does not plan to use data collection cycles that are less frequent than annual.

3. Methods to Maximize Response Rates

General data collection procedures are described below, followed by methods used to maximize response rates.

Interview procedures. Similar data collection procedures will be used for the semi-structured telephone interviews across the three different respondent types mentioned previously. Scholars, health system advisors, and program directors will be invited to participate in an interview by email (Appendices A2, B2, and C2, respectively) at least 1 month in advance of the data collection period. This email will include information about the purpose of the study, as well as anticipated frequently asked questions (Appendices A3, B3, and C3), and will ask the respondent to determine their availability during a range of dates. If the respondent does not reply to the email within 1 week, the study team will send a reminder email (Appendices A4, B4, and C4). If the respondent does not reply within another week, the study team will attempt to recruit the participant by telephone (Appendices A5, B5, and C5). Shortly after a response is received, the scholars, health system advisors, and program directors will be contacted by email to schedule the interview (Appendices A6, B6, and C6, respectively); however, if requested or more appropriate, the email may be substituted with a phone call. Once the interview is scheduled, each respondent will receive a confirmation email including the date, time, and call-in information for the interview (Appendices A7, B7, and C7). Each respondent will receive an email (Appendices A8, B8, and C8) within 3 days and a phone call (Appendices A9, B9, and C9) within 1 day of their interview as a reminder and to provide the respondent the opportunity to reschedule, if needed. After completing an interview, the respondent will receive a thank you email to express AHRQ’s appreciation (Appendices A10, B10, and C10).

Survey procedures. Health system advisors selected for the survey (i.e., all who advise an enrolled scholar) (Appendix D1) will be notified of the data collection by email (Appendix D2). The email will invite them to participate in the survey, explain the purpose of the survey, and include information necessary to complete the web survey. The survey will provide a toll-free number to contact the study team at any point during data collection.

If the health system advisor does not complete the web survey within 1 week, the health system advisor may be contacted throughout the duration of the data collection period (up to three times by email [Appendix D3] and three times by telephone [Appendix D4]) to remind and encourage respondents to complete the survey. The study team will read a prepared script to the respondent and, in addition to encouraging participation, will verify the email address to ensure the web survey is being delivered to the appropriate person. To maximize response rates, non-working/incorrect email addresses and telephone numbers will be monitored, and new information will be located, if possible. Once the health system advisor completes the survey, a thank-you email will be sent to express AHRQ’s appreciation (Appendix D5).

Data Collection Management. The Contractor will closely monitor field data collection efforts. Contact attempts will be recorded and made available to guide interviewers’ timing of contact attempts. Supervisors and interviewers will meet regularly to discuss work plans and alternative approaches for contacting and gaining cooperation of individual cases, as needed. If the Contractor determines that data are missing or responses are unclear, a member of the data collection team may contact the respondent. The Contractor will oversee the management of data including the coding (as appropriate) and analysis of the data.

Achieving the specified response rates involves using procedures described below to secure participation once the potential respondents have been identified. Research shows that preliminary notification, multiple follow-ups with respondents, personalization techniques, proof of sponsorship or endorsement, and shorter questionnaires increase response rates.² In the event an 80 percent response rate is not obtained, a nonresponse bias analysis will be conducted, and data will be weighted.

Below, we describe the recruitment procedures designed to maximize the number of scholars, health system advisors, and program directors who participate in telephone interviews.

• The emailed letters inviting scholars, health system advisors, and program directors to participate in the interviews will be carefully developed to emphasize the importance of this study and how the information will help AHRQ.

• A toll-free number and email address will be provided to respondents. They will be encouraged to call if they have questions about the study.

• Follow-up attempts will be made, as needed, for scholars, health system advisors, and program directors who do not respond. The primary purpose of these calls will be to encourage them to participate in an interview.

• Call-scheduling procedures that are designed to call numbers at different times of the day and different days of the week will be used to improve response rates.

• A training for telephone interviewers will be conducted. The training, specific to this study, will include an overview of the project, a review of the research questions the study will address, a primer on interviewing practices and procedures, and techniques for encouraging respondent candor.

Below, we describe the recruitment procedures designed to maximize the number of health system advisors who respond to the survey.

• The introductory email will be carefully developed to establish the legitimacy of the study, the importance of the study, and the connection of the study to larger AHRQ goals. These procedures help increase respondent participation by increasing the perceived benefits of the data collection and by providing advance notice.

• Emailed follow-up attempts will be made with all health system advisors who do not complete the survey by the due date cited. The primary purpose of the contact attempts will be to urge health system advisors to complete the survey as soon as possible.

• A toll-free number and email address for the study will be provided, so respondents can receive assistance with the survey.

• Call-scheduling procedures that are designed to call numbers at different times of the day and different days of the week will be used to improve response rates.

4. Test of Procedures

The Contractor conducted usability testing of the Health System Advisor Survey (Appendix D1) and related email communication materials (Appendices D2, D3, and D5) in April 2020 with eight members of their research team. The survey was tested using multiple modes (computer, cellular telephone [androids and iPhones], and tablet iPads, Samsung tablets, and Kindle Fires]) and across various browsers (Chrome, Safari, Edge, Firefox, Internet Explorer, and Safari). Test participants were assigned to one of two groups, one that would be sent the reminder email and one that would not. Test participants also tested the “close and continue at a later time” function. Minor changes were made to the survey materials including streamlining response options and clarifying the wording of one question.

5. Statistical Consultants

The Contractor, 2M Research, will serve as the primary consultant for statistical aspects of the design and analysis of the study. Dr. Regina James, 2M Research’s project director, is the primary point of contact for statistical design and analyses. She can be reached at rjames@2mresearch.com or 703-721-8359.

List of Appendices³

Appendix A1. Scholar Interview Guide

Appendix A2. Scholar Invitation Email

Appendix A3. Scholar Study Description/FAQs

Appendix A4. Scholar Invitation Reminder Email

Appendix A5. Scholar Telephone Recruitment Script

Appendix A6. Scholar Scheduling Email

Appendix A7. Scholar Confirmation Email

Appendix A8. Scholar Reminder Email

Appendix A9. Scholar Reminder Telephone Script

Appendix A10. Scholar Thank-You Email

Appendix B1. HSA Interview Guide

Appendix B2. HSA Invitation Email

Appendix B3. HSA Study Description/FAQs

Appendix B4. HSA Invitation Reminder Email

Appendix B5. HSA Telephone Recruitment Script

Appendix B6. HSA Scheduling Email

Appendix B7. HSA Confirmation Email

Appendix B8. HSA Reminder Email

Appendix B9. HSA Reminder Telephone Script

Appendix B10. HSA Thank-You Email

Appendix C1. PD Interview Guide

Appendix C2. PD Invitation Email

Appendix C3. PD Study Description/FAQs

Appendix C4. PD Invitation Reminder Email

Appendix C5. PD Telephone Recruitment Script

Appendix C6. PD Scheduling Email

Appendix C7. PD Confirmation Email

Appendix C8. PD Reminder Email

Appendix C9. PD Reminder Telephone Script

Appendix C10. PD Thank-You Email

Appendix D1. HSA Survey

Appendix D2. HSA Survey Invitation Email

Appendix D3. HSA Survey Reminder Email

Appendix D4. HSA Survey Reminder Telephone Script

Appendix D5. HSA Survey Thank-You Email

1 Note, this is not the unique number of respondents. It is assumed that the same health system advisor will respond to the survey twice (once at the beginning of a scholar’s tenure and once close to when the scholar is expected to complete the LHS K12 program [in approximately 2 years]) during the data collection period.

2 Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014). Internet, phone, mail, and mixed mode surveys: The tailored design method (4th ed.). John Wiley & Sons Inc.

3 Appendices are grouped by data collection activity.