Supporting Statement Part B -- NHSOPS Comparative Database 5-15-2012

SUPPORTING STATEMENT

Part B

Agency for Healthcare Research and Quality’s (AHRQ)

Nursing Home Survey on Patient Safety Culture Comparative Database

January 6, 2012

Agency of Healthcare Research and Quality (AHRQ)

Table of Contents

B. Collection of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 6

3. Methods to Maximize Response Rates 9

4. Tests of Procedures 9

5. Statistical Consultants 10

B. Collection of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The AHRQ Nursing Home Survey on Patient Safety Culture (Nursing Home SOPS) Comparative Database serves as a central U.S. repository for data on the survey, the largest database of the survey’s results. However, the database comprises data that are voluntarily submitted by nursing homes that have administered the survey and is not a statistically selected sample, nor is it a representative sample of all U.S. nursing homes. Only nursing homes that voluntarily administer the survey and are willing to submit their data for inclusion in the database are represented. The Nursing Home SOPS, toolkit materials, and the comparative database are all made available in the public domain along with technical assistance provided by AHRQ through its contractor at no charge to nursing homes, to facilitate the use of these materials for nursing home patient safety and quality improvement.

Currently, comparative results from 226 nursing homes are presented in the Nursing Home Survey on Patient Safety Culture: 2011 User Comparative Database Report available at http://www.ahrq.gov/qual/nhsurvey11/nhsurv111.pdf. Similar to the Hospital Survey on Patient Safety Culture Comparative Database Report (http://www.ahrq.gov/qual/hospsurvey10/), the first Nursing Home SOPS Comparative Database Report includes a section entitled “Data Limitations” that outlines the limitations of the data and makes it clear that the data are not a statistically selected sample of U.S. nursing homes.

Universe of nursing homes and representativeness of the data. There are estimated to be around 15,900 nursing homes in the United States (https://www.cms.gov/CertificationandComplianc/Downloads/nursinghomedatacompendium_508.pdf; accessed December 29, 2011). In an effort to compare the 226 database nursing homes in the 2011 Comparative Database Report to the population of nursing homes, AHRQ presents statistics comparing the characteristics of the 226 database nursing homes (for example, bed size, ownership, and region) against characteristics of U.S. nursing homes based on those in the Centers for Medicare and Medicaid Services (CMS) Nursing Home Compare Database (http://www.medicare.gov/Download/DownloadDB.asp?%20language=Englishversion=default; About the Nursing Home accessed February 17, 2011). This database is used as the comparison because it is the best source of data on nursing home characteristics and includes more nursing homes than any other dataset.

Statistics From the 2011 Nursing Home SOPS Comparative Database Report.

Tables B-1 to B-6 compare the 226 nursing homes from the 2011 Comparative Database Report to nursing homes in the Nursing Home Compare Database on several characteristics. The tables show that the 226 database nursing homes are fairly consistent with the nursing homes in the Nursing Home Compare Database, on characteristics such as the number of beds, ownership, Census region, and others.

Note: Column percent totals in the tables may not sum to exactly 100% due to rounding.

Table B-1. Distribution of Nursing Homes in CMS and AHRQ Databases by Bed Size, 2011

Bed Size	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
1-49 beds	2,048	13%	26	12%
50-99 beds	5,732	37%	78	35%
100-199 beds	6,939	44%	100	44%
200 or more beds	972	6%	22	10%
TOTAL	15,691	100%	226	100%

Table B-2. Distribution of Nursing Homes in CMS and AHRQ Databases by Type of Ownership, 2011

Type of Ownership	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
For profit	10,763	69%	109	48%
Non profit/Government	4,928	31%	117	52%
TOTAL	15,691	100%	226	100%

Table B-3. Distribution of Nursing Homes in CMS and AHRQ Databases by Region, 2011

Census Region	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
South	5,398	34%	86	38%
Northeast	2,685	17%	27	12%
Midwest	5,170	33%	64	28%
West	2,438	16%	49	22%
TOTAL	15,691	100%	226	100%

Table B-4. Distribution of Nursing Homes in CMS and AHRQ Databases by Ownership by Multi-Facility Organization, 2011

Ownership by Multi-Facility Organization	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
Owned or leased by Multi-Facility Organization	8,569	55%	147	66%
Not owned or leased by Multi-Facility Organization	7,122	45%	72	34%
TOTAL	15,691	100%	219	100%

Note: Data missing for seven NH SOPS database nursing homes

Table B-5. Distribution of Nursing Homes in CMS and AHRQ Databases by Resident and/or Family Councils, 2011

Resident and/or Family Councils	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
Both Resident and Family Councils	5,104	33%	80	37%
Resident Council only	9,971	64%	133	61%
Family Council only	52	<1%	2	1%
None	564	4%	4	2%
TOTAL	15,691	100%	219	100%

Note: Data missing for seven NH SOPS database nursing homes

Table B-6. Distribution of Nursing Homes in CMS and AHRQ Databases by Participation in Quality Indicator Survey,* 2011

Participation in Quality Indicator Survey	CMS Nursing Home Compare Database Nursing Homes		AHRQ NH SOPS Database Nursing Homes
	Number	Percent	Number	Percent
Yes	2,979	19%	65	30%
No	12,712	81%	154	70%
TOTAL	15,691	100%	219	100%

Note: Data missing for seven NH SOPS database nursing homes

* The Quality Indicator Survey (QIS) is a computer assisted long term care survey used by the Centers for Medicare and Medicaid Services (CMS) and selected State Survey Agencies to determine if Medicare and Medicaid certified nursing homes are meeting federal requirements. The QIS is a two-staged process whereby surveyors systematically review specific nursing home requirements and objectively investigate any regulatory areas that are triggered. More information on the QIS is available at: http://www.cms.gov/SurveyCertificationGenInfo/downloads/SCLetter08-21.pdf.

Comparative results and explanation of how results are calculated. The 2011 Nursing Home SOPS Comparative Database Report presents average percent positive scores for each of the 12 patient safety culture composites and for the survey’s 42 items (plus two additional questions on respondents’ willingness to recommend their nursing home and an overall rating on resident safety). The average percent positive scores are calculated by averaging composite-level percent positive scores across all nursing homes in the database, as well as averaging item-level percent positive scores across nursing homes. Since the percent positive is displayed as an overall average, scores from each nursing home are weighted equally in their contribution to the calculation of the average. An alternative method would be to report a straight percent of positive response across all respondents, but this method would give greater weight to respondents from larger nursing homes since they account for more than twice as many responses as those from smaller nursing homes.

Percentages are presented rather than mean scores because nursing home administrators have indicated that percentages are more easily understood and interpreted (for example, indicating that 75% of staff responded positively to an item rather than reporting that the mean score on the item was 4.00 out of 5). In addition, the minimum and maximum percent positive scores are presented along with percentile scores for the 10^th, 25^th, 50^th, 75^th, and 90^th percentiles to present information about the distribution of scores across database nursing homes.

2. Information Collection Procedures

Information collection for the AHRQ Nursing Home Survey on Patient Safety Culture Comparative Database occurs in a periodic data collection cycle each year from January to February. Information collection procedures for submitting and processing data are shown in Figure 1.

Figure 1. Nursing Home SOPS Comparative Database Data Submission

Step 1: Call for Data Submission. Announcements about the opening of data submission go out through various publicity sources. AHRQ’s patient safety and electronic newsletters target approximately 50,000 subscribers. In addition, email announcements are sent to approximately 1,600 survey users who have at some point requested technical assistance or who have used the nursing home survey before. An example of an email announcement calling for data submission is shown in Attachment I, Email # 1. Reminder emails are sent one and two weeks after the initial email announcing the call for data submission (see Attachment I, email #2). In addition, the AHRQ Web site has public information about the yearly timeline and instructions for data submission (http://www.ahrq.gov/qual/nhsurvey10/nhsubinfo.htm). Through these efforts, U.S. nursing homes are made aware of and invited to submit their survey data to the database.

As the administrator of the database and under contract with AHRQ, Westat provides free technical assistance to submitting nursing homes through a dedicated email address (DatabasesOnSafetyCulture@ahrq.hhs.gov –routes to Westat) and toll-free phone number (1-888-852-8277).

Step 2: Registration for Potential Participants. A secure data submission Web site allows interested parties such as nursing homes and nursing home systems to register and submit data (see Attachment J, Figures 1 and 2). Potential participants complete Eligibility and Registration forms (see Attachment B), which takes about 3 minutes that ask for contact and other basic information. If registrants are deemed eligible to submit data, two separate, automated emails are sent to provide them with a username and password and information needed in the next steps of the data submission process (see Attachment I, Emails #3 and #4).

Once users have a username and password, they can enter the main page menu of the Web site (Attachment J). Information about eligibility requirements, data use agreements, and data file specifications regarding how to prepare their data for inclusion in the SOPS database is posted and can be reviewed.

Step 3: Enter Facility Information and Upload Questionnaire. Here, users provide information about each of their facilities, such as point of contact, method of survey administration, overall response rate, and other facility characteristics (bed size, ownership, and geographic region) (see Attachment D: Nursing Home Site Information Form in Supporting Statement Part A). They also upload their survey questionnaire that they administered to enable us to determine whether any changes were made to the survey (Attachment L).

Step 4: Submit Data Use Agreement (DUA). To protect the confidentiality of all participating nursing homes, a duly authorized representative from the nursing home must sign a data use agreement (Attachment C). The DUA language was reviewed and approved by AHRQ’s general counsel. The DUA states that the nursing home’s data are handled in a secure manner using necessary administrative, technical and physical safeguards to limit access to it and maintain its confidentiality. In addition, the DUA states the data are used for the purposes of the database, that only aggregated results are reported, and that the nursing home is not identified by name. Data are not included in the database without this signed data use agreement. Users fax, email, and/or mail a signed copy of the DUA.

Step 5: Upload Data File(s). Here users are asked to upload their individual-level survey data for each nursing home. Data are submitted through a secure data submission Web site and are encrypted to ensure secure, confidential transmission of the survey data. Data are accepted in Microsoft Excel® format since this is the format preferred by nursing homes. If a user has multiple nursing homes within a system, users can upload one data file that identifies all of the nursing homes in their system. The data file specifications (Attachment M) are provided to data submitters to ensure that users submit standardized and consistent data in the way variables are named, coded, and formatted.

Once a data file is uploaded, a separate program developed in Visual Basic (VB) reads the submitted files and loads them into the SQL database that stores the data. A data quality report is produced and made available to the participant. This report displays item frequencies and flags out-of-range values and incorrectly coded items. If there are no problems with the data, an acknowledgement of data upload is granted via an automatic email. If data are improperly coded, an automatic email informs the participant of the problem. Users are expected to fix any errors and resubmit.

Step 6: Approve Data Submission. Once all of the information required for submission is submitted and approved, an email is sent to the facility contact indicating that their data have received final acceptance.

3. Methods to Maximize Response Rates

AHRQ makes a number of toolkit materials available to assist nursing homes with the SOPS surveys. The Nursing Home SOPS has a Survey User’s Guide that gives users guidance and tips about survey administration on the following topics: planning; selecting a sample; determining their data collection method; data collection procedures, with a section on Web surveys; and analyzing data and producing reports (at http://www.ahrq.gov/qual/nhsurvey08/nhguide.htm). The Survey User’s Guide also gives nursing homes tips about how to increase response rates through publicity efforts, top management support, use of incentives, and following all steps of proper data collection protocols.

As noted earlier in this document under Information Collection Procedures, announcements about the opening of data submission go out through various publicity sources as a way to boost nursing home participation in the database. AHRQ’s patient safety and electronic newsletters target approximately 50,000 subscribers. In addition, email announcements are sent to approximately 1,600 survey users who have at some point requested technical assistance or who have used the survey before. AHRQ, through its contractor Westat, provides free technical assistance to users through a dedicated email box and toll-free phone number. In addition, reminders are sent to database registrants to remind them of the deadline for data submission.

4. Tests of Procedures

Input and Feedback for the Development of the SOPS Database Submission System. Because the Surveys on Patient Safety Culture are public-use instruments, the SOPS program has generally modeled its data submission processes after those utilized by the Consumer Assessment of Healthcare Providers and Systems (CAHPS) database that has been in operation for many years. SOPS staff consulted with CAHPS database staff and programmers to determine best practices for data submission. This information, as well as feedback obtained during the provision of technical assistance each year, has been used to develop the SOPS online data submission system and processes.

5. Statistical Consultants

Joann Sorra, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

joannsorra@westat.com

Naomi Dyer, PhD

Westat

1600 Research Blvd.

Rockville, MD 20850

naomidyer@westat.com

2