Supporting Statement A

AHRQ’s mission. As described in its 1999 reauthorizing legislation, Congress directed the Agency for Healthcare Research and Quality (AHRQ) to enhance the quality, appropriateness, and effectiveness of health services, as well as access to such services, by establishing a broad base of scientific research and promoting clinical and health systems practice improvements.ⁱ The legislation also directed AHRQ to “conduct and support research, evaluations, and training, support demonstration projects, research networks, and multidisciplinary centers, provide technical assistance, and disseminate information on healthcare and on systems for the delivery of such care, including activities with respect to health statistics, surveys, database development, and epidemiology.”ⁱⁱ

Furthermore, AHRQ shall conduct and support research “to provide objective clinical information to healthcare practitioners and other clinicians of healthcare goods or services; identify the causes of preventable healthcare errors and patient injury in healthcare delivery; develop, demonstrate, and evaluate strategies for reducing errors and improving patient safety; and disseminate such effective strategies throughout the healthcare industry”.ⁱⁱⁱ

In the U.S., at least 5 percent of adults seeking outpatient care experience a diagnostic error.^iv Diagnostic errors are also expensive, accounting for approximately 10 percent of all medical costs, or $250 billion per year in the U.S.^v They also represent the highest proportion of total malpractice insurance payouts (i.e., 35 percent).^vi Costs also can vary greatly by condition; for example, the annual cost of misdiagnosis of stroke alone is an estimated $1 billion annually.^vii

In 2013, with AHRQ sponsorship, the Institute of Medicine, National Academies of Sciences, Engineering, and Medicine (NASEM) appointed a Committee on Diagnostic Error in Health Care “to synthesize what is known about diagnostic error as a quality of care challenge and to propose recommendations for improving diagnosis.”^viii The Committee define diagnostic error as “the failure to (a) establish an accurate and timely explanation of the patient’s health problem(s) or (b) communicate that explanation to the patient.” The resulting and seminal 2015 NASEM report, Improving Diagnosis in Health Care, notes that “most people will experience at least one diagnostic error in their lifetime, sometimes with devastating consequences” and calls improving diagnostic processes a “moral, professional, and public health imperative.”

In follow-up to the 2015 NASEM report, the National Quality Forum (NQF) convened a multi-stakeholder expert committee to develop a conceptual framework for measuring diagnostic quality and safety, identify gaps in measurement, and suggest priorities for measure development. NQF’s Committee reached consensus around key priority areas for measure development: timeliness of test result follow-up, patient access to information, diagnostic quality improvement activities, handoffs, timeliness and accuracy of diagnosis, communication with patients, and diagnostic workload.^ix

Also in 2015, the National Patient Safety Foundation and American Board of Medical Specialties co-sponsored a Summit on Certification and Diagnostic Accuracy, to identify challenges and recommendations for improving medical diagnoses. The Summit report indicated that the majority of diagnostic errors occur in ambulatory settings where there is less technical support. Resulting strategies from the Summit fell into three domains: education and training, assessment and improvement, and practice environment and culture. In particular, the panel discussed the need to enhance cultural assessment tools, increase clinician training and feedback mechanisms, and better empower patients in the diagnostic process.^x

Shortly thereafter, AHRQ held a 2016 Research Summit on Improving Diagnosis in Health Care. This Summit’s goals were to: 1) learn from the insights and experiences of participants; 2) examine definitions, physician and patient perspectives, the use of data and measurement, the role of health information technology, and the impact of organizational factors on the diagnostic process; and 3) identify future directions for research.^xi Speakers recognized that the field of diagnostic safety is in its infancy and identified critical evidence gaps. Summit participants helped define AHRQ’s opportunities to advance the field that included developing culture surveys and other tools relevant to diagnostic safety.

There has been little focus to-date on the role of patient safety culture and diagnostic error, which represents a significant gap. AHRQ has a unique opportunity to help remedy this gap by expanding its existing and highly successful Surveys on Patient Safety Culture™ (SOPS™) program to include a new supplemental item set on diagnostic error that can optionally be administered at the end of the Medical Office SOPS Survey. Similar to Westat’s work on previous SOPS survey supplemental item sets, our goal will be to develop a reliable, public-use item set to augment the Medical Office SOPS survey. The evidence shows that diagnostic errors occur more frequently in ambulatory care; therefore, developing the item set for the outpatient setting directs AHRQ’s resources to areas where the evidence indicates a gap, and the potential for improvement is significant. Therefore, we are requesting a supplemental item set so that providers of ambulatory care can efficiently assess the culture of their organization with respect to diagnostic error. Moreover, because there is overlap in the content of the SOPS Medical Office survey and the Diagnostic Safety supplemental items, we want to limit the duplication of questions on the two instruments. For this reason, Westat proposes that the limited scope of a supplemental item set (i.e., 12-15 items across 3-4 composite measures) is more appropriate than developing a separate, full-length survey.

The supplemental item set will assist medical offices in assessing factors contributing to diagnostic errors, help them identify strengths and areas for improvement to efficiently target resources to improve diagnostic processes, reduce errors and malpractice claims/costs, and improve patient safety and patient outcomes.

This research has the following objectives:

Conduct cognitive interviews with individual respondents to test the items in the Diagnostic Safety supplemental item set for the Medical Office Survey on Patient Safety Culture. Cognitive testing will be conducted in English.

Conduct pilot test data collection in 150 medical offices and based on an analysis of results, determine which survey items to retain and refine the questionnaire accordingly.

Engage a Technical Expert Panel (TEP) to review pilot results and finalize the supplemental item set.

Make the supplemental item set and related administration guidance documents publicly available.

To achieve these objectives, we propose the following activities:

Cognitive interviews – The purpose of these interviews is to understand the cognitive processes respondents engage in when answering each draft item on the survey, which will help refine the survey instrument. These interviews will be conducted with a mix of medical office personnel, including clinicians, nurses, and other types of staff (including administrative and clinical support staff).

Cognitive interviews have already been conducted with 9 respondents to inform development of the current draft Diagnostic Safety supplemental item set presented in Attachment A. Up to three rounds of cognitive interviews will be conducted by telephone for up to 30 respondents (ten respondents each round). The cognitive interview guide found in Attachment B will be used for all rounds.

Feedback obtained from the first round of interviews for the draft supplemental item set will be used to refine the items. The results of Round 1 testing, along with the proposed revisions, will be reviewed with a Technical Expert Panel before proceeding with Rounds 2 and 3 of testing. In total, up to 30 cognitive interviews will be conducted to refine the draft supplemental items for pilot testing.

Pilot test – We will plan one data collection effort of the draft supplemental items aimed at assessing the psychometric properties of the items and composites. We will assess the variability, reliability, factor structure and construct validity of the draft supplemental items and composites, allowing for their further refinement (see Part A, Section 16 for analysis plan description). The draft supplemental items following the SOPS Medical Office survey (see Attachment C) will be administered to approximately 2,500 clinicians and staff from 150 medical offices to facilitate analysis of the data. A medical office point of contact (POC) will be recruited to publicize the pilot test of the supplemental items and assemble a list of sample clinicians and staff. We have found that, on average, one point of contact will represent approximately 10 medical offices; therefore, we estimate that we will be working with approximately 15 medical office points of contact. Instructions for the POCs are included in Attachment C1, and Exhibit 2 includes a burden estimate for the POCs’ time in assisting with the pilot test. Clinicians and staff will receive notification of the supplemental item set and reminders via email. The supplemental item set will be administered via web. The draft pilot test notification and the weekly follow-up reminder notice are included in Attachment D.

Technical Expert Panel (TEP) feedback – A TEP has been assembled to provide input to guide the development of the Diagnostic Safety supplemental item set. Upon completion of the pilot test, results will be reviewed with TEP members to finalize the supplemental items. The TEP is discussed in more detail in Section 8b. This TEP activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.

Dissemination activities – The final supplemental items will be made publicly available through the AHRQ website. This dissemination activity does not impose a burden on the public and is therefore not included in the burden estimates in Section 12.

This work is being conducted by AHRQ through its contractor, Westat, pursuant to AHRQ’s statutory authority to conduct and support research on healthcare and on systems for the delivery of such care, including activities with respect to the quality, effectiveness, efficiency, appropriateness and value of healthcare services and with respect to quality measurement and improvement. 42 U.S.C. 299a(a)(1) and (2).

2. How, by Whom, and for What Purpose Information Will Be Used

The responses from the cognitive testing of the draft Diagnostic Safety supplemental items will be used by project staff to test and improve the items and composites. Following cognitive testing results, we will revise the survey for pilot testing. Further, the information collected in the pilot test data collection effort will be used to test and improve draft supplemental items. Psychometric analysis will be conducted on the supplemental items data to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items and composites. Because the items are being developed to measure specific aspects or composites of patient safety culture in the medical office setting, the factor structure of the items will be evaluated through confirmatory factor analysis. On the basis of the data analyses, items or composites may be dropped to create the final supplemental item set.

Medical offices participating in the pilot test data collection effort will receive a report of their medical office-specific results. This feedback report serves as an incentive for participation, and saves the medical offices time and effort to analyze their own results.

The final supplemental item set will be made available to the public for use in medical offices to assess how their organizational culture supports the diagnostic process, accurate diagnoses, and communication around diagnoses. The supplemental items can be used by medical offices to identify areas for improvement related to their diagnostic safety culture. Researchers are also likely to use the supplemental items to assess the impact of diagnostic safety improvement initiatives.

3. Use of Improved Information Technology

The pilot test data collection will be conducted using a web survey. The majority of medical offices that voluntarily submitted their patient safety culture survey data to the 2018 SOPS Medical Office Database administered web surveys (86 percent)^xii. In addition to reducing the burden associated with survey administration (printing and tracking paper surveys), a web-based survey will offer increased security of responses and eliminate data entry expense.

4. Efforts to Avoid Duplication

Information on an organization’s culture in the context of diagnostic safety has not been systematically or rigorously collected and is not available through any other sources based on the literature scan we conducted. During this scan, no surveys were found on the culture of diagnostic safety in medical offices.

5. Involvement of Small Businesses

Some of the medical offices participating in this pilot test will be designated as small businesses. The data collection instruments and procedures are designed to minimize burden on individual medical office staff respondents.

6. Consequences if Information Collected Less Frequently

This effort is a one-time data collection.

7. Special Circumstances

The data collection efforts will be consistent with the guidelines at 5 CFR 1320.5(d)(2).

8. Federal Register Notice and Outside Consultations

8.a. Federal Register Notice

As required by 5 CFR 1320.8(d), notice was published in the Federal Register April 28^th, 2017 on Page 19727 for 60 days for 60 days (see Attachment E).

8.b. Outside Consultations

To guide the development of the Diagnostic Safety supplemental items, a Technical Expert Panel (TEP) has been assembled. The TEP reviewed drafts of the supplemental items and will also review feedback from the cognitive interviews and assist in finalizing the supplemental items. The TEP includes 5 clinician members with experience in diagnostic safety (see Attachment F).

9. Payments/Gifts to Respondents

Cognitive Interview Respondents. To successfully recruit 30 cognitive interview participants, it is appropriate to offer a cash incentive. For 1-hour cognitive interviews, we propose a $150 cash remuneration for an estimated 14 clinicians (8 physicians, 3 physician assistants, 3 nurse practitioners), a $100 cash remuneration for an estimated 4 registered nurses, and a $75 cash remuneration for an estimated 12 other medical office support staff (4 medical assistants, 4 secretaries, 4 office managers). The amount for incentives totals $3,400.

The survey research literature uniformly demonstrates that incentives are an effective means of communicating the importance of the study to the respondent. In a meta-analysis, Mercer, Caporaso, Cantor, and Townsend (2015)^xiii show that incentives follow a dose-response model – the greater the incentive, the greater the level of respondent participation to a point. This is equally true for the incentives offered in cognitive testing. The amounts proposed are based on the average hourly rates for the respondents, coupled with the time to complete the cognitive interviews (see Exhibit 2). Based on both our research and experience, Westat asserts that not providing incentives equal to at least the average hourly wage for respondents will significantly hinder respondent recruitment.

Because of the time pressures in a clinical setting, cognitive testing with clinical staff is particularly challenging. To the best of our knowledge, all cognitive testing of questionnaires with clinical staff offer an incentive. Usually, it is a monetary incentive, often called an honorarium. Examples of studies where the clinical staff was provided a monetary incentive for testing a questionnaire include: Shaw, Talley, Beebe, and Rockwood (2001)^xiv; Cho, Johnson, and VanGeest (2013)^xv; McLeod, Klabunde, Willis, and Stark (2013)^xvi; and Salinas (2014)^xvii.

Pilot Test Respondents. No incentive is proposed for organizations or individuals participating in the pilot test. Medical offices will receive customized feedback reports that compares their results with the aggregated results from the other participating medical offices. We think this report provides sufficient value for offices to participate.

10. Assurance of Confidentiality

Individuals and organizations will be assured limitation on use of certain information under Section 944(c) of the Public Health Service Act, 42 USC 299c-3(c). This law requires that information collected for research conducted or supported by AHRQ that identifies individuals or establishments be used only for the purpose for which it was supplied.

11. Questions of a Sensitive Nature

We do not consider survey questions related to safety culture are particularly sensitive; however, if during cognitive testing we discover any sensitivities, we will modify or delete these questions accordingly.

12. Estimates of Annualized Burden Hours and Costs

Exhibit 1 shows the estimated annualized burden hours for the participants’ time to take part in this research. Cognitive interviews for the supplemental items will be conducted with 30 individuals (approximately 8 physicians, 3 physician assistants, 3 nurse practitioners, 4 registered nurses, 4 medical assistants, 4 administrative or clerical staff, and 4 medical office managers) and require approximately one hour to complete.

For the pilot test, the supplemental items will be administered to about 2,500 individuals from 150 medical offices and require 20 minutes to complete. Assuming a response rate of 60 percent, this data collection effort will yield a total of 1,500 completed questionnaires. We expect an estimated 15 POCs, each representing an average of 10 individual medical offices, will complete the medical office information form survey (completion is estimated to take about 3 minutes).

We estimate the total annualized burden is 538 hours.

Exhibit 2 shows the estimated annualized cost burden associated with the participants’ time to take part in this research. The total cost burden is estimated to be $26,298.

Exhibit 1. Estimated annualized burden hours

Form Name/Activity	Number of respondents/POCs	Number of responses per respondent	Hours per response	Total burden hours
Cognitive interviews	30	1	1	30
Medical office information survey	15	10	3/60	8
Pilot test	1500	1	20/60	500
Total	1545	na	na	538

Exhibit 2. Estimated annualized cost burden

Form Name/Activity	Total burden hours	Average hourly wage rate*	Total cost burden
Cognitive interviews^a	30	$54.16	$1,625
Medical office information survey^b	8	$60.07	$481
Pilot test^c	500	$48.38	$24,192
Total	538	na	$26,298

^a Based on the weighted average wages for 8 Family and General Practitioners (29-1062; $,102.73), 3 Physician Assistants (29-1071; $50.01), 3 Nurse Practitioners (29-1171; $50.83), 4 Registered Nurses (29-1141; $32.16), 4 Medical Assistants (31-9092; $16.14), 4 Medical Secretaries (43-6013; $16.71), 4 General and Operational Managers (11-1021; $60.07) in the medical office setting;

^b Based on the average wages for General and Operational Managers (11-1021; $60.07) in the medical office setting;

^c Based on the weighted average wages for 300 Family and General Practitioners (29-1062; $102.73), 150 Physician Assistants (29-1071; $50.01), 150 Nurse Practitioners (29-1171; $50.83), 200 Registered Nurses (29-1141; $32.16), 300 Medical Assistants (31-9092; $16.14), 200 Medical Secretaries (43-6013; $16.71), 200 General and Operational Managers (11-1021; $60.07) in the medical office setting;

* National Occupational Employment and Wage Estimates in the United States, May 2017, “U.S. Department of Labor, Bureau of Labor Statistics” (available at http://www.bls.gov/oes/current/naics4_621100.htm [for medical office setting]

13. Estimates of Annualized Respondent Capital and Maintenance Costs

Capital and maintenance costs include the purchase of equipment, computers or computer software or services, or storage facilities for records, as a result of complying with this data collection. There are no direct costs to respondents other than their time to participate in the study.

14. Estimates of Annualized Cost to the Government

Exhibit 3 shows the estimated total and annualized cost for this project. Although data collection will last for less than one year, the entire project will occur over 3 years. The total cost of the data collection activities includes $3,400 in incentives to the cognitive interview respondents. The total cost for this project is approximately $433,500, and the annualized cost is estimated at $144,500.

Exhibit 3. Estimated Total and Annualized Cost

Cost Component	Total Cost	Annualized Cost
Project Development	$24,772	$8,257.25
Data Collection Activities	$123,857	$41,285.50
Data Processing and Analysis	$92,894	$30,964.50
Publication of Results	$37,157	$12,385.75
Project Management	$30,965	$10,321.50
Overhead	$123,857	$41,285.50
Total	$433,500	$144,500

Exhibit 4. Estimated Annual cost to AHRQ for Project Oversite

AHRQ Position

% Time

Annualized Cost

Project Lead-GS 15/05

$7,811

Program Analyst-GS 13/05

$5,619

Total

$13,430

https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2019/DCB.pdf

15. Change in Burden

This data collection effort is a new activity.

16. Time Schedule, Publication and Analysis Plan

As soon as OMB approval is received, pilot test activities will begin. The estimated time schedule to conduct these activities is shown below:

Up to three rounds of cognitive interviews (4 months)
Pilot test data collection (5 months)
Data analysis, feedback report production, and development of technical reports (6 months)
Final Diagnostic Safety supplemental item set and development of toolkit materials (3 months)

The final version of the Diagnostic Safety supplemental item set, technical reports, and accompanying toolkit materials will be made publicly available on the AHRQ website.

This section describes the specific analyses that we will conduct on the pilot test data.

Pilot Test Psychometric Analysis. Psychometric analysis will be conducted to examine item nonresponse, item response variability, factor structure, reliability, and construct validity of the items. Because the supplemental items are being developed to measure specific aspects or composites of patient safety culture, the factor structure of the supplemental items will be evaluated through confirmatory factor analysis. On the basis of the data analyses, items or composites may be dropped.

Descriptive Statistics

The means, standard deviations, and response frequencies for the supplemental items will be examined to ensure that respondents and medical offices exhibit adequate response variability on the supplemental items. In addition, items will be examined to ensure that there are low rates of missing data (i.e. lower than 20 percent missing response per item). Poorly functioning items will be identified.

Confirmatory Factor Analysis

A confirmatory factor analysis will be conducted to initially examine whether groups of items intended to measure a specific patient safety composite are interrelated, ignoring the nesting of respondent data within medical offices. Factor loadings for each item in an a priori composite will be considered as having an adequate contribution to a particular composite or factor if the strength of the item's relationship to that factor (i.e., its factor loading), is 0.40 or greater.

We will also examine overall model fit indices using standard fit statistics: the chi-square, comparative fit index (CFI), and the standardized root mean square residual (SRMR). For chi-square statistics, lower and non-significant chi-squares indicate good fit. The factor structure is determined to adequately fit the data if the CFI is at least 0.90. A value of zero for the SRMR indicates perfect fit, but a value less than 0.08 is considered a good fit.

Intraclass Correlations (ICCs) and Design Effects

Intraclass correlations (ICCs) will be computed for each composite. ICC's determine if substantial variation exists between groups compared to variation within groups. ICCs above 0.05 or 5 percent indicate that the between group variance is greater than expected by chance and imply that nesting in groups does have an effect on the responses of individuals.

Given that ICCs are likely to be influenced when there are many groups with few individuals within the groups (or when there are few groups with many individuals within the groups), we will also examine design effects, which take into account within-group sample size. A design effect of 2 or more implies that group membership or nesting of individuals within groups does have an effect on the responses of the individuals and therefore multilevel modelling should be conducted to account for the multilevel nature of the data.

Reliability Analysis

Reliability analyses will then be performed on the composites to examine whether individuals responded consistently to the items within each composite. Internal consistency reliability will be calculated using Cronbach's alpha. The minimum criterion for acceptable reliability is an alpha of at least 0.70.

Intercorrelations

Intercorrelations among the supplemental item set’s patient safety composites will also be examined. Intercorrelations will be explored at the individual and medical office levels of analysis. While the composites should be correlated since they measure aspects of the patient safety culture, the intercorrelations should not be extremely high (0.80 or higher) because very high intercorrelations indicate that the composites may not be unique enough to be considered separate constructs or measures. While there is no standard criterion for acceptable levels of dimension intercorrelations and construct validity, in general, such correlations should be less than 0.80 for the composites to be considered unique and to avoid problems with multicollinearity.

The above analyses will be used to determine which, if any, items and composites are functioning poorly and remove them from the survey to derive a final set of items and composites with good psychometric properties and reduce the overall length of the final supplemental item set. The Technical Expert Panel will be informed of the data analysis results and asked to advise Westat on which items to retain or drop (when the psychometric results do not provide enough guidance and decisions can be made on the content value of the items).

The final supplemental item set will be made publicly available on the SOPS pages of the AHRQ website for use by medical offices and researchers.

17. Exemption for Display of Expiration Date

No exemption is requested.

List of Attachments

Attachment A: Draft Diagnostic Safety Supplemental Item Set

Attachment B: Draft Diagnostic Safety Cognitive Interview Guide

Attachment C: Pilot test

Attachment C1: Medical Office Point of Contact (POC) Instructions

Attachment D: Survey Invitation and Reminder Notices

Attachment E: Federal Register Notice

Attachment F: Technical Expert Panel Members

i Healthcare Research and Quality Act of 1999. Available at https://www.ahrq.gov/policymakers/hrqa99a.html. Last accessed 2/26/2019.

ii See Section 902, (a) (8) of the Healthcare Research and Quality Act of 1999. Available at https://www.ahrq.gov/policymakers/hrqa99a.html. Last accessed 2/26/2019.

iii See Section 912, (b) (2) (A) (ii) (I) and (iii) (II) and (c) (1) (2) and (3) of the Healthcare Research and Quality Act of 1999. Available at http://www.ahrq.gov/policymakers/hrqa99b.html. Last accessed 2/26/2019.

ivSingh H, Meyer AND, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. 2014;23(9):727-731.

v Feldman L. Managing the Cost of Diagnosis. Managed Care 2009:43-5. Available at http://simulconsult.com/company/ManagingCostDiagnosis.pdf. Last accessed: 2/26/2019.

vi Saber Tehrani AS, Lee H, Mathews SC, Shore A, Makary MA, Pronovost P, et al. 25-Year summary of U.S. malpractice claims for diagnostic errors 1986-2010: An analysis from the National Practitioner Data Bank. BMJ Qual Saf 2013;22:672–80. Available at http://qualitysafety.bmj.com/content/22/8/672.short. Last accessed 2/26/2019.

vii Newman-Toker DE, McDonald KM, Meltzer DO. How much diagnostic safety can we afford, and how should we decide? A health economics perspective. BMJ Qual Saf 2013;22(Suppl 2):ii11–20. Available at http://qualitysafety.bmj.com/content/22/Suppl_2/ii11.short. Last accessed 2/26/2019.

viii Improving Diagnosis in Health Care. Author: Committee on Diagnostic Error in Health Care. Contributors: Erin P. Balogh, Bryan T. Miller, and John R. Ball, Editors; Committee on Diagnostic Error in Health Care; Board on Health Care Services; Institute of Medicine; National Academies of Sciences, Engineering, and Medicine. Published by National Academies Press, 2015. Available at http://nap.edu/21794. Last accessed 2/26/2019.

ix Improving Diagnostic Quality and Safety: Final Report. September 19, 2017. This report is funded by the Department of Health and Human Services under contract HHSM-500-2012-00009I, Task Order HHSM-500-T0026. Available at: http://www.qualityforum.org/Publications/2017/09/‌Improving_Diagnostic_Quality_and_Safety_Final_Report.aspx. Last accessed 2/26/2019.

x Summit on Certification and Diagnostic Accuracy Report: Insights and Opportunities. American Board of Medical Specialties. Chicago, IL. Published February 24, 2016. (The meeting, cosponsored by the ABMS and the National Patient Safety Foundation, was held in Washington, DC in December 2015.) Available at http://www.abms.org/media/100055/npsfsafetysummitsummarydigital.pdf. Last accessed 2/26/2019.

xi Henriksen K, Dymek C, Harrison MI, Brady PJ, Arnold SB. Challenges and opportunities from the Agency for Healthcare Research and Quality (AHRQ) research summit on improving diagnosis: a proceedings review. Published Online: 2017, May 23. Available at https://doi.org/10.1515/dx-2017-0016. Last accessed 2/26/2019

xii SOPS Medical Office Database. Content last reviewed May 2018. Agency for Healthcare Research and Quality, Rockville, MD. http://www.ahrq.gov/sops/databases/medical-office/index.html. Last accessed 2/26/2019.

xiii Mercer A., Caporaso, A., Cantor, D., and Townsend, R. (2015). How much gets you how much? Monetary incentives and response rates in household surveys. Public Opinion Quarterly, 79(1), 105-129.

xiv Shaw, M., Talley, N., Beebe, T., and Rockvood, T. (2001). Initial validation of a diagnostic questionnaire for gastroesophageal reflux disease, Am J Gastroenterology, 96(1):52-7

xv Cho, Y., Johnson, T., Vangeest, J. (2013). Enhancing surveys of health care professionals – a meta-analysis of techniques to improve response, Eval Health Prof, 36(3): 382-407

xvi McLeod, C., Klabunde, C., Willis, G., Stark, D. (2013). Health Care Provider Surveys in the United States, 2000-2010: A Review, Eval Health Prof, 36: 106-126

xvii Salinas, G. D. (2014). Trends in physician preferences for and use of sources of medical information in response to questions arising at the point of care: 2009–2013. Journal of Continuing Education in the Health Professions, 34(S1), S11-S16

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	0000-00-00

Supporting Statement A

Supporting Statement Part A - Pilot Test of SOPS Dx Safety - 4-9-2019 (003).docx

Questionnaire and Data Collection Testing, Evaluation, and Research for the Agency for Healthcare Research and Quality