CMS-10097.Supp Stmt Part B - Collection of Information-CLEAN-11-10-09

The target population for the 2010 survey consists of all active providers served by Medicare contractors across the country and Puerto Rico who submitted 50 or more Medicare claims in a one-year span. Each year CMS selects an initial sample of approximately 30,000-35,000 providers designed to yield no more than 25,000 completed surveys. Respondent completion rates, eligibility rates, and the ability to locate providers determine the final, achieved sample size. Table 1, Medicare Provider Initial Sample Allocation for National Implementation, provides an example of how the initially allocated sample was distributed by provider type in the 2009 MCPSS sample. A similar allocation can be expected in 2010 (the sample will be selected closer to the administration of the survey; the exact allocation is not available for this OMB submission).

Table 1. Medicare Provider Initial Sample Allocation for National Implementation – Example from the 2009 MCPSS

Provider Types	Sample Size
Hospitals	3,298
Skilled Nursing Facility	5,252
Other Part A Providers	5,369
Home Health Agencies	1,594
Hospice Facilities	831
Physicians	7,895
Licensed Practitioners	3,469
Other Part B Providers	2,068
DME Suppliers*	2,496
Total	32,272

* DME Suppliers includes physicians who submitted claims for durable medical equipment or supplies.

C-2 Procedures for Collecting Information

C-2.1 Study Sample

In order to make valid comparison between contractors, the study sample is designed to obtain an equal number of completed questionnaires, 400, from each contractor. For those contractors with a provider population size of 400 or smaller, all the providers will be selected with certainty. The maximum percent error for estimates of percentages obtained from a simple random sample yielding 400 completed questionnaires will not exceed 5 percent, 95 percent of the time. For example, if 50 percent of providers responded as satisfied with the service they received, CMS can be 95 percent confident that between 45 percent and 55 percent of the providers are satisfied with the service. The percent error is the largest for the 50 percent proportion and decreases as proportion moves further away from the 50 percent / 50 percent split. For example, for an 80 percent / 20 percent split, the error is 4 percent. Thus, 400 completed questionnaires should provide adequate precision for contractor-level estimates.

Sample sizes smaller than 400 were considered. Sample sizes smaller than 400 will not only provide smaller precision, they will also require more oversampling for smaller provider types. For example, a sample size of 300 will provide an error not exceeding 5.8 percent, which is not substantially higher than 5 percent; however, it will require more extensive and higher oversampling rates in smaller provider types. This oversampling can further reduce the precision of the contractor level estimates.

The sample size of 400 is allocated proportionately to states and provider types within each contractor. For contractors with multiple service areas, the providers will be first stratified by service area, and then within service area by provider type. The proportional allocation provides a representative sample of providers for contractors across the service areas and provider types and minimizes the variance of the contractor-level estimates.

The proportional allocation could result in small sample sizes in several relatively smaller provider types and states. Currently, some provider types are oversampled to yield a minimum of 30 completed questionnaires (based on CMS data needs during a given year, oversampling at the state level is also done). Thirty responses are adequate to conduct statistical tests to detect valid differences between provider types within or across the contractors, or within or across states.

Figure 1. Power by Sample Size

C-2.2 Survey Materials

The supplemental survey materials associated with the proposed 2010 survey instrument will follow a similar design and format as those used in the prior administrations of the national MCPSS. A description of materials follows.

Cover Letters

Sampled providers are mailed/e-mailed a survey notification package, i.e., an invitation to participate in the survey, which can include up to two cover letters, one on CMS letterhead and another from the relevant contractor. The letters explain the purpose of the study and the need for the data, include a confidentiality clause, provide the unique Provider ID and password to access the Internet survey, and contain contact information (e.g., a toll free phone number, a fax number, and an e-mail address) so that respondents can ask questions, request assistance, or request a paper questionnaire.

Survey Tool

The 2010 survey tool has a revised satisfaction scale. As initially designed, the survey utilized a 6-point scale. The scale had text labels for only the two end points, with a ‘1’ labeled as “Not at all satisfied”, and a ‘6’ labeled as “Completely satisfied”. In the past CMS experienced difficulties in communicating survey results to stakeholders based on this scale.

The new proposed scale is a 5-point, fully-labeled Likert scale with a neutral category:

• Very Satisfied

• Satisfied

• Neither Satisfied nor Dissatisfied

• Dissatisfied

• Very Dissatisfied

The change to a fully anchored scale will allow CMS to communicate a well defined message about the performance of the Medicare contractors. We will be able to report percent satisfied (the percent of respondents with a survey mean of “Satisfied” or “Very Satisfied”). In addition, the literature indicates that a fully labeled scale has higher reliability than those that are only partially labeled (Christian, Dillman and Smyth, 2006).

In the Provider Outreach and Education section of the 2010 survey CMS plans to include up to three additional questions on emerging topics that are of interest to CMS leadership and/or to providers. Because of the time lapse between submittal of the OMB package and the emerging issues, the topics are yet to be determined as of the submission date of this OMB package. The additional questions would be closed ended (5-point scale), would not be sensitive in nature, and would provide insight on the success of outreach and education efforts for specific topics. The estimated time of response for these three questions is included in the calculation of the total survey burden appearing in section B, Table 1, Time Burden per Survey Section. The Summary of Instrument Revisions, located in Section 4 of this submission, details the changes from the current OMB-approved instrument (2009) with the proposed 2010 instrument.

C-2.3 Data Collection

The data collection steps for the 2010 administration are as follows:

Mail/e-mail invitations to complete the Internet survey (survey notification package) to randomized sample of providers
Screener call to non-responders and providers for which CMS lacks a reliable address.
Callee may choose to complete the survey during the screening call or complete the survey at a later time via the Internet or hardcopy.
Mail/e-mail survey notification package to the address identified during the sample cleaning/screening process
Send a reminder/thank-you postcard/e-mail
Start non-response follow-up to remaining non-respondents
Close data collection in late May

Providers are encouraged to complete the survey over the secure website. Although, the survey notification package includes instructions for accessing and completing the survey on-line, it also provides the respondent with the option to print a copy of the questionnaire from the website and return it by mail or fax (enabling respondents to respond using their preferred delivery method). In addition, all providers are given the option to request a paper copy of the questionnaire (rather than downloading it from the Internet) and then submitting their responses via mail or fax.

The following media have been established to allow respondents to communicate with CMS during data collection:

Toll free telephone number: The survey contractor maintains a toll-free telephone number to receive calls from respondents concerning any issues they have regarding the survey.
E-Mail Box: The survey contractor maintains a study e-mail box. This has been a popular feature and can facilitate communication regarding alternative ways respondents want to submit survey responses.
Fax Number: A fax number is available for respondents who wish to respond via this method. The fax machine, to which inquiries or responses are sent, is located in a secure location and only authorized project staff has access to retrieve these documents.

C-2.4 Processing Returned Surveys

Two criteria are used for processing returned surveys:

The submission must contain the pre-coded identification number.
All core items must be complete.

C-2.5 Calculating Satisfaction Scores

Moving to a fully-labeled 5-point scale will allow us to calculate the percent satisfied. For the 5-point survey instrument, the weighted sum for the numerator will be the number of item responses of “Satisfied” and “Very Satisfied” and the denominator will include these along with responses of “Neither Satisfied nor Dissatisfied,” “Dissatisfied” and “Very Dissatisfied.”

C-2.6 Contractor Reports

The Medicare contractors have been pleased with the content and level of detail provided in the final contractor reports. Contractors have indicated that the reports, particularly the item level results and respondent comments, are useful in identifying the services that need improvement. Several contractors have also stated that the satisfaction scores confirmed what they already thought and/or knew to be problem service areas. In addition, contractors have agreed that the timeframe for receiving these documents (i.e., July) was especially helpful because it helped them prepare for the upcoming fiscal year.

The survey results from the national implementation are available to all contractors via a secure, interactive Web-based system. Contractors can access the following information via the on-line reports:

Contractor specific scores at the contractor level, provider level, and business function level as well as those levels crossed by state or jurisdiction,
Item level weighted frequencies,
Verbatim and coded comments (these comments are sanitized and do not contain any identifiers).

To help identify problem areas, contractors can view both scores and frequencies by certain provider attributes such as state, provider type, and size.

Per OMB’s request, the 2010 final public report will include standard errors for all reported scores. The summary scores at all levels include cell sizes as well as standard errors. Since providers may have answered some, but not all of the sections, or only some of the questions for a particular section, the cell size for calculating the scores can vary across sections of the survey. A cell size is presented with each score so contractors know how many providers responded to that section; this provides an indication of the stability of the score. If only a few providers answered the question, then the survey estimate could fluctuate considerably if a different set of providers were surveyed. The larger the number of providers who respond to an item, the more confident we are that the survey estimate is close to the answer we would find had we not selected a sample, but instead surveyed all providers. The standard errors are intended to help the contractor determine how close the contractor score is to the average contractor score. If too few providers, (fewer than 30), answered any given survey section then the results are suppressed to reduce the chance of a contractor identifying a specific provider.

The reports will include information on key drivers of satisfaction within each business function. They will also include information on which business functions are key drivers of overall satisfaction. This information can help contractors focus their performance improvement efforts.

C-3 Methods to Maximize Response Rates and Deal with Non-response

The 2009 MCPSS concluded with a final response rate of 70 percent. CMS has explored many issues related to increasing the saliency of the study among the provider community and using non-response follow-up strategies to maximize response rates.

The OMB target response rate for the national implementation is 80 percent. Any year that the MCPSS falls below the OMB target of 80 percent, CMS will explore the option of conducting a non-response bias analysis. See C-3.3 for a detailed description of the proposed non-response bias analysis.

In the MCPSS, non-response is ignored in the final dataset – data are not imputed. Scores are based on aggregate data – the numerator of a section score is based on the aggregated responses to all items in the survey section, divided by the number of valid responses in that section. Responses of “Don’t Know” or “Not Applicable” are excluded from both the numerator and denominator.

Before the survey is closed, CMS reviews the core questions for the Internet and paper surveys. If the core item was left entirely blank, then an interviewer will attempt to re-contact the respondent to ask about any missing core items. The goal is to make sure that respondents were given the opportunity to answer all core questions.

C-3.1 Promoting the Survey Project to Increase Saliency

CMS is taking an aggressive approach to achieving the OMB response rate goal of 80 percent. It is essential to create awareness and understanding of the value and importance of the survey within provider communities in order to motivate participation in the survey. To achieve high saliency for the study, the level of outreach activity between October and January is aggressive. We use selective outreach campaigns between January and March to low responding groups and also conduct follow-up outreach activities when results are available in July and August. The CMS has implemented an annual public relations campaign to generate broad coverage of the MCPSS initiative through a variety of channels:

The healthcare print and Web-based trade media serving financial and business managers employed by Medicare providers and FFS contractors.
Contractor-based communications channels such as list-servs, conferences and meetings, newsletters, etc.
Professional organizations that serve the provider community.
CMS- based channels of communications to both the providers and contractors.

C-3.2 Follow-up with Non-respondents

CMS uses telephone, e-mail, and mail as modes of follow-up with non-respondents.

C-3.3 Non-response bias analysis

If response rates fall below 80 percent, CMS will conduct a non-response bias analysis. The purpose of this analysis is to determine if the non-respondents are significantly different from the respondents. This may include an analysis of sample frame variables, including contractor, provider type, number of claims, dollar value of claims, size of facility (bed size and or number of patient days), specialty type (in the case of physicians, licensed practitioners, and medical equipment providers), ownership type (for hospitals and skilled nursing homes). CMS has submitted to OMB results from non-response bias analyses for prior administrations.

In the event that the response rate falls below 60 percent, CMS will create a sub-sample of non-respondents to conduct a more detailed non-response bias study. The sub-sample will include providers who refused as well as and those who were contacted. This non-response study would include a an abbreviated follow-up survey to the sub-sample. The six to seven minute survey survey would likely include only the claims processing section, the overall satisfaction question, and a question on why the respondent initially refused or did not respond.. We will then compare the satisfaction scores of the initial respondents to the sub-sample respondents by contractor type to determine if there is a significant difference. If a significant difference is found, estimates can be adjusted for non-response bias through weighting. The protocol for the sub-sample survey will be as follows:

Mail/e-mail invitation to complete the abbreviated survey including revised cover letters from CMS and contractors
One week later, a reminder/thank-you postcard
One week later, telephone interviews, with up to nine additional callbacks

C-3.4 Non-response adjustment

In spite of the best practices, virtually all surveys experience non-response. One consequence of non-response is the potential for bias in the survey estimates, making them larger or smaller than the true statistic for all providers. The extent to which those that do reply differ in their satisfaction from those that do not reply affects the extent of bias. When response rates vary among subgroups, such as provider types, there is an even greater potential for bias in survey estimates.

We will adjust the sampling weights to remove potential bias on satisfaction caused by not obtaining responses from all sampled providers. If response propensity is independent of the satisfaction, then no bias would arise. Therefore, the objective is, using the known characteristics of the sampled providers, to form non-response adjustment cells so that the response propensity within each cell is independent of satisfaction. To the extent that this was achieved, the estimates of satisfaction obtained using the sampling weights that are adjusted for non-response within these cells, will have smaller potential bias. There are several alternative methods of forming the cells to achieve this result. In forming the cells, we will attempt to minimize the variation in response propensity within the cells.

We plan to use Chi-Square Automatic Interaction Detector (CHAID) software to guide us in forming the cells. CHAID uses an AID type of algorithm. CHAID partitions data into homogenous subsets with respect to response propensity. To accomplish this, it first merges values of the predictors, which are statistically homogeneous with respect to response propensity, and maintains all other heterogeneous values. It then selects the most significant predictor, with the smallest p-value, as the best predictor of response propensity and thus forms the first branch in the decision tree. It continues applying the same process within the subgroups (nodes) defined by the "best" predictor chosen in the preceding step. This process continues until no significant predictor is found or a specified (about 20) minimum node size is reached. The procedure is stepwise and creates a hierarchical tree-like structure.

The data on the relevant characteristics of the providers will be available from the sampling frames for both respondents and non-respondents. These characteristics include provider type, number of claims (both volume and dollar value) and Metropolitan Statistical Area(MSA)/non-MSA status for all providers, number of beds for hospitals and skilled nursing facilities, total patient days for hospitals, ownership type of the facility, physician/non-physician specialty and age, and specialty for DME MACs.

Although non-response adjustment should reduce bias, it can also increase the variance of estimates. Small adjustment classes and/or low response rates (or large non-response adjustment factors) may increase the variance substantially and give rise to unstable estimates. In order to prevent an excessive increase in variance and thereby an adverse effect on the mean square error of the estimates, we will limit the size of the classes to a minimum and avoid large adjustment factors.

C-4. Individuals Consulted

Organization	Name	Contact Information
CMS	Rich Cuchna	410.786.7239/Rich.Cuchna@cms.hhs.gov
	Rochelle Fiato	410.786.6853/Rochelle.Fiato@cms.hhs.gov
	Robin Fritter	410.786.7485/Robin.Fritter@cms.hhs.gov
	Tressa Mundell, PO	410-786-9176 / Teresa.Mundell@cms.hhs.gov
	Geraldine Nicholson	410.786.6967/ Geraldine.Nicholson@cms.hhs.gov
	Carlene Randolph	410.786.4008/Carlene.Randolph@cms.hhs.gov
	Colette Shatto	410.786.6932/ Colette.Shatto@cms.hhs.gov

Westat	David Cantor	301.294.2080/ DavidCantor@westat.com
	Sherm Edwards	301.294.3993/ ShermEdwards@westat.com
	Pamela Giambo	240-453-2981/ PamelaGiambo@westat.com
	Huseyin Goksel	301.251.4395/ HuseyinGoksel@westat.com
	Vasudha Narayanan	301.294.3808/ VasudhaNarayanan@westat.com

File Type	application/msword
File Title	SUPPORTING STATEMENT
Author	CMS
Last Modified By	CMS
File Modified	2009-11-10
File Created	2009-11-10