201507-0935-001_Supporting Statement Part B -- NHSOPS Comp DB_12-4-14_2015-08-05 Comments_8-7-15 Westat response

SUPPORTING STATEMENT

Part B

Agency for Healthcare Research and Quality’s (AHRQ)

Nursing Home Survey on Patient Safety Culture Comparative Database

December 4, 2014

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collection of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 7

4. Tests of Procedures 7

5. Statistical Consultants 8

B. Collection of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The AHRQ Nursing Home Survey on Patient Safety Culture (Nursing Home SOPS) Comparative Database serves as a central U.S. repository for data on the survey and AHRQ houses the largest database of the survey’s results. However, the database is comprised of data that are voluntarily submitted by nursing homes that have administered the survey, and is not a statistically selected sample, nor is it a representative sample of all U.S. nursing homes. The Nursing Home SOPS was pilot tested in 2007 (OMB #0935-0132; approved on July 5, 2007). The original Nursing Home SOPS Comparative Database was approved on June 8, 2012 (OMB #0935-0195; expiration date 6/30/2015). Comparative results from 263 nursing homes that participated in the 2014 database are publicly available on the AHRQ Web site at: http://www.ahrq.gov/professionals/quality-patient-safety/patientsafetyculture/nursing-home/2014/nhsurv14-ptI.pdf.

As of 2014, there are estimated to be 15,648 nursing homes in the United States (CMS Nursing Home Compare data were obtained from Nursing Home Compare, available at:

https://data.medicare.gov/data/nursing-home-compare (accessed May 7, 2014). The distribution of nursing homes by type in the Centers for Medicare and Medicaid Services (CMS) Nursing Home Compare database and the Nursing Home SOPS Comparative Database is presented in Table 1.

Table 1. Nursing Homes in the CMS Nursing Home Compare Database and the 2014 Nursing Home Survey on Patient Safety Culture (NH SOPS) Comparative Database by Ownership

Type of Ownership	CMS Nursing Home Compare Nursing Homes		NH SOPS Comparative Database Nursing Homes
	Number	Percent	Number	Percent
For profit	10,873	69%	113	43%
Non-profit	3,838	25%	134	51%
Government	937	6%	16	6%
Total	15,648	100%	263	100%

Nursing homes that submit data to the database receive a free, customized feedback report that displays their nursing home’s results against the database. Nursing homes that do not submit data to the database can still compare their results using the Comparative Database report. The report provides instructions on how to calculate percent positive scores to enable nursing homes to calculate their own scores to compare their results against the database. As part of a toolkit of support materials for the Nursing Home SOPS survey, nursing homes can also use a Microsoft^© Excel-based Data Entry and Analysis Tool that is an Excel file with macros that will automatically produce graphs and charts of a nursing home’s results once data are entered into a data sheet. Many nursing homes use this tool to produce their results.

In the overall database report, nursing homes are provided with a detailed description and explanation of the statistics that are presented and given examples and guidance on how to compare their nursing home’s results against the comparative results from the database.

Most nursing homes simply compare their percent positive scores against the database percentiles and averages and do not attempt any statistical comparisons. To help nursing homes simplify comparisons and provide conservative guidance on what level of difference would be considered meaningful, the report recommends a 5 percentage point difference in scores as a rule of thumb to determine whether its scores can reasonably be considered higher or lower than the database scores.

2. Information Collection Procedures

Information collection for the AHRQ Nursing Home Survey on Patient Safety Culture Comparative Database occurs in a periodic data collection cycle every other year. The next submission period is April 2016. Information collection procedures for submitting and processing data are shown in Figure 1.

Figure 1. Nursing Home SOPS Comparative Database Data Submission

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s electronic newsletter targets approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. An example of email announcements calling for data submission is shown in Attachment D, Email # 1 and #3. Through these efforts, U.S. nursing homes are made aware of and invited to submit their survey data to the database.

As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting nursing homes through a dedicated email address (DatabasesOnSafetyCulture@westat.com)) and toll-free phone number (1-888-324-9790).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as nursing homes and nursing home systems to register and submit data. Registration takes about 3 minutes to complete and asks for contact and other basic information (see Attachment A). After registering, if registrants are deemed eligible to submit data, an automated email is sent to authenticate the account and update the user password (see Attachment D, Email #2).

Once users are registered and have a password, they can enter the main page menu of the Web site. Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the SOPS database is posted and can be reviewed.

Step 3: Enter Facility Information and Upload Questionnaire. At this step, users provide information about each of their facilities, such as point of contact, method of survey administration, overall response rate, and other facility characteristics (e.g., bed size, ownership, and geographic region) (see Attachment C). They also upload their survey questionnaire that they administered to enable us to determine whether any changes were made to the survey (Attachment G, Figure 1).

Step 4: Submit Data Use Agreement (DUA). To protect the confidentiality of all participating nursing homes, a duly authorized representative from the nursing home must sign a data use agreement (DUA) (Attachment B). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the nursing home’s data will be handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are reported, and that the nursing home will not be identified by name. Data are not included in the database without this signed data use agreement. Users can fax and/or mail a copy of the signed agreement.

Step 5: Upload Data File(s). At this step, users are asked to upload their individual-level survey data for each nursing home (see Attachment G, Figure 2). Data submitted through the secure data submission Web site are encrypted to ensure secure, confidential transmission of the survey data. Data are accepted in Microsoft Excel^® format since this is the format preferred by nursing homes. Users must upload one data file per nursing home. If a user has multiple nursing homes within a system, users can upload one data file that identifies all of the nursing homes in their system. The data file specifications (Attachment E) are provided to data submitters to ensure that users submit standardized and consistent data in the way variables are named, coded, and formatted.

Once a data file is uploaded, a separate load program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. A data quality report is then produced and made available to the participant. This report displays item frequencies and flags out-of-range values and incorrectly reverse-coded items. If there are no problems with the data, an acknowledgement of data upload and acceptance will be granted during the user session. If data are improperly coded, the user is informed that the data file failed during the user session by having a message post on the screen. Users are expected to fix any errors and resubmit their data file(s) for processing. Once there are no problems, the user is informed of the acceptance of data during the user session with an online message of acceptance.

Step 6: Approve Data Submission. Once all of the information required for submission has been submitted and approved, an email is sent to the facility contact indicating that their data have received final acceptance.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist nursing homes with the SOPS surveys. The Nursing Home SOPS has a Survey User’s Guide that gives users guidance and tips about survey administration on the following topics: planning; selecting a sample; determining their data collection method; data collection procedures, with a section on Web surveys; and analyzing data and producing reports (at http://www.ahrq.gov/qual/nhsurvey08/nhguide.htm). The Survey User’s Guide also gives nursing homes tips about how to increase response rates through publicity efforts, top management support, use of incentives, and following all steps of proper data collection protocols.

As noted earlier in this document under Information Collection Procedures, Step 1 – Call for Data Submission, announcements about the opening of data submission go out through various publicity sources as a way to boost nursing home participation in the database. AHRQ’s electronic newsletter targets approximately 50,000 subscribers. In addition, the AHRQ Surveys on Patient Safety Culture listserv targets approximately 22,000 subscribers. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the SOPS Database Submission System. Because the Surveys on Patient Safety Culture are public-use instruments, the SOPS program has generally modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) Database that has been in operation for many years. SOPS staff consulted with CAHPS Database staff and programmers to determine best practices for data submission. This information, as well as feedback obtained during the provision of technical assistance each year the database has been running, has been used to improve the SOPS online data submission system and process over time.

5. Statistical Consultants

Joann Sorra, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

joannsorra@westat.com

Naomi Dyer Yount, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

naomiyount@westat.com

2