201507-0935-003_Supporting Statement Part B -- MOSOPS Comparative Database 08-19-2015

SUPPORTING STATEMENT

Part B

Medical Office Survey on Patient Safety Culture Comparative Database

July 1, 2015

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collection of Information Employing Statistical Methods 2

1. Respondent universe and sampling methods 2

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 6

4. Tests of Procedures 7

5. Statistical Consultants 7

B. Collection of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The AHRQ Medical Office Survey on Patient Safety Culture (Medical Office SOPS) Comparative Database serves as a central U.S. repository for data from the survey and AHRQ houses the largest database of the survey’s results. However, the database is comprised of data that are voluntarily submitted by medical offices that have administered the survey, and is not a statistically selected sample, nor is it a representative sample of all U.S. medical offices. The Medical Office SOPS was pilot tested in 2007 (OMB #0935-0132; approved on July 5, 2007). The original Medical Office SOPS Comparative Database was approved on June 12, 2012 (OMB #0935-0196; expiration date 6/30/2015). Comparative results from 935 medical offices homes that participated in the 2014 database are publicly available on the AHRQ Web site at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/medical-office/2014/index.html.

The first Medical Office Survey on Patient Safety Culture Comparative Database report was published in 2012. In 2014, AHRQ published a second report. Comparative results from 935 medical offices homes that participated in the 2014 database are available on the AHRQ Web site at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/medical-office/2014/index.html.

Quantifying the universe of medical offices is not straightforward. Medical offices are typically characterized as either those with 1 or 2 physicians or group medical practices consisting of 3 or more physicians. According to the U.S. Census Bureau 2007 Economic Census (2007 NAICS code 6211 “Offices of physicians”), there were 220,131 physicians’ offices in the U.S. (http://factfinder2.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ECN_2007_US_62SSSZ7&prodType=table). These offices consist of those with only 1 or 2 physicians as well as offices from medical groups with 3 or more physicians. Because the AHRQ Medical Office SOPS is a survey, it is recommended that it only be implemented in group medical offices with 3 or more providers and staff to protect the confidentiality of medical office staff respondents when survey results are summarized. Smaller offices with only 1 or 2 physicians are advised to use the survey as a discussion tool rather than administer it as a survey. Therefore, the population most likely to submit data to the Medical Office SOPS Comparative Database is group medical offices.

A 2005 Health Affairs article examining electronic health record adoption in group medical practices, whose lead author is from the Medical Group Management Association states: “The total number of U.S. group practices is not known, but we estimate it to be somewhat larger than the 34,490 practices we identified, perhaps in the range of 40,000–50,000” (David Gans, John Kralewski, Terry Hammons, and Bryan Dowd, “Medical Groups’ Adoption Of Electronic Health Records And Information Systems,” Health Affairs, 2005, Vol 24 (5), pp. 1323-1333).

The most relevant and thorough source of data on the population of medical group practices in the U.S. is the American Medical Association’s (AMA) 1999 edition of “Medical Group Practices in the U.S.: A Survey of Practice Characteristics.” This report is the only and most recent source that reports information about the characteristics of medical group practices in the U.S., with groups defined as those with 3 or more physicians. A total of 34,066 eligible medical groups were identified and a census of them was conducted by the AMA in 1996. The problem with comparing group practices to individual medical offices is that a single group practice can comprise several different medical office locations. The AMA report only includes data from the parent or primary location of group practices, and therefore is also an underestimate of the actual number of group medical offices.

A more recent report from the National Center for Health Statistics in 2008 (http://www.cdc.gov/nchs/data/series/sr_13/sr13_166.pdf) presents estimates of the number and characteristics of medical practices with which physicians are associated. These data, from the 2005-2006 National Ambulatory Medical Care Surveys (NAMCS), are physician-based rather than office-based, and do not allow direct comparisons with the Medical Office SOPS database medical offices. However, we present the NAMCS geographic region data in Table 1 below. Finally, the NAMCS report estimates that during 2005-2006 there were 163,700 medical practices in the United States, which is considerably lower than the 220,131 physicians’ offices in the U.S. Census Bureau 2007 Economic Census.

Table 1 shows the geographic distribution of the AHRQ Medical Office SOPS database medical offices compared to the distribution of offices of physicians based on the 2007 U.S. Economic Census (http://factfinder2.census.gov/faces/tableservices/jsf/pages/productview.xhtml?pid=ECN_2007_US_62SSSZ7&prodType=table) and the NAMCS estimates of the number of office-based medical practices in 2005-2006. The AHRQ database over represents medical offices in the South and Midwest regions and underrepresents medical offices in the Northeast and West.

Table 1. Distribution of Offices of Physicians in AHRQ Database Medical Offices (2014), U.S. Economic Census, Offices of Physicians (2007), and NAMCS Office-Based Medical Practices (2005-2006) by Region

Census Region	AHRQ MO SOPS Database Medical Offices (2014)		U.S. Economic Census, Offices of Physicians (2007)		NAMCS Office-Based Medical Practices (2005-2006)
	Number	Percent	Number	Percent	Number	Percent
South	479	51%	84,424	38%	60,700	37%
Northeast	60	6%	44,605	20%	36,300	22%
Midwest	370	40%	38,951	18%	30,100	18%
West	26	3%	52,151	24%	36,600	22%
TOTAL	935	100%	220,131	100%	163,700	100%

* Note: Column percent totals may not add to exactly 100% because of rounding.

Because there is not a recent and comprehensive source of data describing the population of group medical offices in the U.S., we do not present comparisons of the database medical offices to any other population statistics. Only descriptive statistics about the database medical offices are provided. Participants represent a spectrum of characteristics.

Medical offices that submit data to the database receive a free, customized feedback report that displays their medical office’s results against the database. Medical offices that do not submit data to the database can still compare their results using the Comparative Database report. The report provides instructions on how to calculate percent positive scores and compare them against the database. As part of a toolkit of support materials for the Medical Office SOPS survey, medical offices can also use a Microsoft® Excel-based Data Entry and Analysis Tool that is an Excel file with macros that will automatically produce graphs and charts of a medical office’s results once data are entered into a data sheet. Many medical offices use this tool to produce their results.

In the overall database report, medical offices are provided with a detailed description and explanation of the statistics that are presented and given examples and guidance on how to compare their medical office’s results against the comparative results from the database.

Most medical offices simply compare their percent positive scores against the database percentiles and averages and do not attempt any statistical comparisons. To help medical offices simplify comparisons and provide conservative guidance on what level of difference would be considered meaningful, the report recommends a 5 percentage point difference in scores as a rule of thumb to determine whether its scores can reasonably be considered higher or lower than the database scores.

2. Information Collection Procedures

Information collection for the AHRQ Medical Office Survey on Patient Safety Culture Comparative Database occurs in a periodic data collection cycle every other year. The next submission period is October 2015. Information collection procedures for submitting and processing data are shown in Figure 1.

Figure 1. Medical Office SOPS Comparative Database Data Submission

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s patient safety and electronic newsletters target approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. An example of email announcements calling for data submission is shown in Attachment D, Email # 1 and # 3. Through these efforts, U.S. medical offices are made aware of and invited to submit their survey data to the database.

As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting medical offices through a dedicated email address (DatabasesOnSafetyCulture@westat.com) and toll-free phone number (1-888-324-9790).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as medical offices and medical groups to register and submit data. Registration takes about 3 minutes to complete and asks for contact and other basic information (see Attachment A). After registering, if registrants are deemed eligible to submit data, an automated email is sent to authenticate the account and update the user (see Attachment D, Email # 2).

Once users are registered and have a password, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the SOPS database is posted and can be reviewed.

Step 3: Enter Medical Office Information and Upload Questionnaire. At this step, users provide information about each of their medical offices, such as point of contact, method of survey administration, overall response rate, and other medical office characteristics (e.g., specialty, number of staff and providers, and ownership) (see Attachment C). They also upload their survey questionnaire that they administered to enable us to determine whether any changes were made to the survey (see Attachment G, Figure 1).

Step 4: Submit Data Use Agreement (DUA). To protect the confidentiality of all participating medical offices, a duly authorized representative from the medical office must sign a Data Use Agreement (DUA) (see Attachment B). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the medical office’s data will be handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are reported, and that the medical office will not be identified by name. Data are not included in the database without this signed data use agreement. Users fax and/or mail a copy of the signed agreement.

Step 5: Upload Data File(s). At this step, users are asked to upload their individual-level survey data for each medical office (see Attachment G, Figure 2). Data submitted through the secure data submission Web site and are encrypted to ensure secure, confidential transmission of the survey data. Data are accepted in Microsoft Excel® format since this is the format preferred by medical offices. Users must upload one data file per medical office. If a user has multiple offices within a medical group, users can upload one data file that identifies all of the offices in their group. The data file specifications (see Attachment E) are provided to data submitters to ensure that users submit standardized and consistent data in the way variables are named, coded, and formatted.

Once a data file is uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. A data quality report is then produced and made available to the participant. This report displays item frequencies and flags out-of-range values and incorrectly reverse-coded items. If there are no problems with the data, an acknowledgement of data upload and acceptance will be granted during the user session. If data are improperly coded, the user is informed that the data file failed during the user session by having a message post on the screen. Users are expected to fix any errors and resubmit their data file(s) for processing. Once there are no problems, the user is informed of the acceptance of data during the user session with an online message of acceptance.

Step 6: Approve Data Submission. Once all of the information required for submission is submitted and approved, an email is sent to the medical office contact indicating that their data have received final acceptance.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist medical offices with the SOPS surveys. The Medical Office SOPS has a Survey User’s Guide that gives users guidance and tips about survey administration on the following topics: planning; selecting a sample; determining their data collection method; data collection procedures (including a section on Web surveys); and analyzing data and producing reports (at http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/medical-office/index.html). The Survey User’s Guide also gives medical offices tips about how to increase response rates through publicity efforts, top management support, use of incentives, and following all steps of proper data collection protocols. Of the medical offices that voluntarily submitted their data for the 2014 Medical Office Survey on Patient Safety Culture Comparative Database, the average response rate for was 64% across 935 medical offices.

The Surveys on Patient Safety Culture User Network promotes the database in a number of ways:

1. The GovDelivery listserv of subscribers to the AHRQ Medical Office Survey on Patient Safety Culture;

2. National partners that have national reach to medical offices that include primary care and specialty practices;

3. Users that have contacted the Surveys on Patient Safety Culture technical assistance helpline about the medical office survey;

4. Other outlets such as Webcasts and conferences

As noted earlier in this document under Information Collection Procedures, Step 1 – Call for Data Submission, announcements about the opening of data submission go out through various publicity sources as a way to boost medical office participation in the database. AHRQ’s electronic newsletter target approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. The GovDelivery listserv is not intended to identify the universe of medical offices; rather, it is a promotional tool to reach potential users who have voluntarily signed up to receive news on the safety culture surveys, including this information on database submission. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

The 2012 Medical Office Survey on Patient Safety Culture Comparative Database had 934 medical offices and the 2014 database had 935 medical offices. These are considered very successful submission cycles especially given that the medical office survey is not required by any accrediting organization.

4. Tests of Procedures

Input and Feedback for the Development of the SOPS Database Submission System. Because the Surveys on Patient Safety Culture are public-use instruments, the SOPS program has generally modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Database that has been in operation for many years. SOPS staff consulted with CAHPS Database staff and programmers to determine best practices for data submission. This information, as well as feedback obtained during the provision of technical assistance each year the database has been running, has been used to improve the SOPS online data submission system and process over time.

5. Statistical Consultants

Joann Sorra, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

joannsorra@westat.com

Naomi Dyer Yount, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

naomiyount@westat.com

2