Form CMS-10204 - Instruments (Appendices B,C,D,E)

This request seeks Office of Management and Budget’s (OMB) approval of (1) collection of enrollment data by demonstration sites and (2) face-to-face interviews with Medicare beneficiaries (not to exceed 45 minutes in length). These data collection and interviews are to be completed during Phase I of the Evaluation of the Medical Adult Day-Care Services Demonstration (Contract Number 500-00-0038/5). Additional information collection is planned for Phase II of this evaluation. Because results from Phase I data collection activities will be used to finalize plans for Phase II, OMB approval for Phase II collection activities will be sought at a later date.

Section 703 of the Medicare Prescription Drug, Improvement and Modernization Act of 2003 (MMA) (Pub. L. 108–173) authorizes a three-year demonstration to assess the clinical and cost-effectiveness of providing medical adult day-care services as a substitute for a portion of home health services that would otherwise be provided in the beneficiary’s home. Under this authority, the Centers for Medicare & Medicaid Services (CMS), through its Office of Research, Development and Information (ORDI), is conducting the Medical Adult Day-Care Services Demonstration. Five Medicare certified home health agencies were selected by CMS through a competitive process to participate in the demonstration. These five demonstration sites are Aurora Visiting Nurse Association (Milwaukee, Wisconsin), Doctor’s Care Home Health (McAllen, Texas), Landmark Home Health Care Services (Allison Park, Pennsylvania), Metropolitan Jewish Health System (Brooklyn, New York) and Neighborly Care Network (St. Petersburg, Florida).

CMS/ORDI is also conducting an evaluation of this demonstration, as mandated under Section 703(h) of MMA. The evaluation aims to assess both the costs and the benefits of delivering home health services in the adult day-care setting. The evaluation will examine the achievements as well as the difficulties inherent in demonstration implementation, and will include the following components:

Phase I (October 2005 through March 2007)

Case studies to assess the implementation of the Demonstration;
Interviews with small samples of beneficiaries who receive services at participating home health agencies (10 per site at each of the 5 sites, for a total of 50 interviews, see Appendix E);
Descriptive analyses of beneficiary characteristics, and services provided by the Demonstration sites to enrolled beneficiaries and non-participants, using information on beneficiaries collected by demonstration sites at the start of each home health episode;

Phase II (April 2007 through September 2009)

Selection of control patients matched to beneficiaries participating in the Demonstration, using demographic and medical diagnosis characteristics;
Analysis of the use and cost of home health services among beneficiaries receiving services at the Demonstration sites and matched comparison patients;
Telephone interviews with a large sample of beneficiaries who receive services at the participating home health agencies to assess patient experience with the Demonstration (180 interviews at each of the 5 sites, for a total of 900 interviews; see Appendix D); and
Synthesis of learning from the case studies and descriptive and statistical analyses to assess the possible implementation effects of the Demonstration, including potential areas for improvement.

This request to OMB seeks approval of (1) the collection of and reporting of beneficiary enrollment data collection by the demonstration sites and (2) beneficiary face-to-face interviews to be conducted by Brandeis university personnel during site visits to Demonstration agencies, as part of the development of case studies as specified by the RTOP. Because interview specifications for the Phase II telephone interviews with beneficiaries will be developed based on the results of the Phase I beneficiary interviews, approval for these interviews will be sought in a future OMB submission. This request pursues no further discussion of information collection specific to Phase II.

A. Justification

1. Legal and Administrative Justification

The legal justification for conducting the Demonstration is provided by Section 703 of the Medicare Prescription Drug, Improvement and Modernization Act of 2003 (Pub. L. 108–173), which authorizes a three-year demonstration “to permit a home health agency, directly or under arrangements with a medical adult day-care facility, to provide medical adult day-care services as a substitute for a portion of home health services that would otherwise be provided in the beneficiary’s home.” [see Appendix A for Legal Basis].

The administrative justification is provided by CMS’s Request for Task Order Proposal (RTOP) No. CMS-05-031/ERD entitled, “Evaluation of the Medical Adult Day Services Demonstration.” This RTOP requires that a case study be conducted at each demonstration site, including “interviews with key informants (HHA administrators, providers, patients, caregivers, advocacy groups, etc.).” The RTOP specifically authorized the inclusion of surveys among the data collection activities, designed to provide “sound and scientifically valid answers” to a number of research questions in the following areas [see Appendix B for Evaluation Research Questions Specified by RTOP No. CMS-05-031/ERD].

Operational issues
Characteristics of participants and non-participants
Impact on use of services
Impact on quality and outcomes
Satisfaction
Market impacts
Cost impacts
Provider impacts
Overall impact and next steps

2. Use of Information

The users of the information will be the Brandeis University team that is evaluating the Demonstration and CMS.

Data Collection by Demonstration Sites: The collection of enrollment data by the Demonstration sites consists of (a) beneficiary identification number; (b) gender; (c) home health care episode start date; (d) information indicating whether the beneficiary was offered participation in the Demonstration; (e) the reason for not offering participation to the beneficiary; (f) whether the beneficiary declined to participate; and (g) whether or not the beneficiary is a recent user of adult day care services (within the 14 days prior to their home health episode). This information will be used by Brandeis to identify the universe for sample selection for the subsequent interview data collection. In addition, these data will allow Brandeis to construct comparison groups (excluded, accepted, refused), which will lead to an understanding of how demonstration participants differed from beneficiaries who refuse participation and beneficiaries who are not offered participation.

Interviews with Beneficiaries: Topics covered by the beneficiary interviews will include beneficiaries’ need for personal and household assistive services, use of home health services, satisfaction with home health and adult day services under the Demonstration, current and prior Medicaid enrollment status, out of pocket costs and living arrangements. Face-to-face interviews will be conducted using a semi-structured approach, guided by interview protocols [see Appendix E for Beneficiary Interview Guides]. If beneficiaries agree, we will tape record the interviews. If agreement to tape record is not granted, we will utilize written notes to record the interviews. These data will allow the researchers to examine use, effectiveness, and satisfaction of Medicare beneficiaries with the home health and adult day services they receive under the Demonstration in relation to their demographic and clinical characteristics. The results will help CMS to understand the user’s experience with home health services in the medical adult day setting and with this Medicare Demonstration. Information gained from these interviews will be used to design the survey instrument for the Phase II beneficiary surveys. Data will be analyzed in summary form only. Data on individual beneficiaries will be kept confidential.

3. Collection of Information

Enrollment data will be collected at the Demonstration sites by home health agency staff. The method of collection, manual or electronic, will be selected at the discretion of the individual site.

Beneficiary interview data will be collected by face-to-face interviews with respondents. We do not plan to use automated or other electronic techniques. Please see Appendix C for a complete description of the interview design and methodology employed.

4. Duplication of Similar Information

The Demonstration, the enrollment data collection, and the interviews being conducted as part of its evaluation do not duplicate any prior efforts by CMS or other organizations.

5. Impacts on Small Businesses

Not applicable. This data collection does not have a significant economic impact on a substantial number of small businesses or other small entities.

6. Less Frequent Data Collection

The enrollment data collection will be performed ongoing during the Demonstration by the sites. Data will be submitted to Brandeis on a monthly basis.

The beneficiary interviews will be administered only once during site visits to each of the Demonstration sites and will be the only means by which CMS and the evaluation team will obtain data from users of home health services in medical adult day care facilities during Phase I. RTOP No. CMS-05-031/ERD specifies that “a case study should be conducted at each demonstration site. These case studies should include reviews of relevant documents and interviews with key informants (HHA administrators, providers, patients, caregivers, advocacy groups, etc.).”

7. Special Circumstances

There are no special circumstances as specified in General Instructions for Supporting Statement for Paperwork Reduction Act Submissions, Specific Instruction A.7.

8. Federal Register Notice/Outside Consultation

There has been no prior publication of the enrollment or site visit data collection in the Federal Register. Inquiries to CMS and the Visiting Nurse Association of Boston did not reveal any similar data from the users of home health and medical adult day services. Similarly, search of the home health and medical adult day literature has not revealed any reports of prior site visit interviews.

9. Payments/Gifts to Respondents

Not applicable. No payment or incentive is to be offered to interview respondents.

10. Confidentiality

All Freedom of Information Act (FOIA) and the Federal Privacy Act requirements will be fully met. The Demonstration sites will provide enrollment and contact data to Brandeis to support selection of the sample for the face-to-face beneficiary interviews. Enrollment data will be transmitted for Brandeis’ use via secure file transfer. Beneficiary contact information is provided to Brandeis as evaluators on behalf of CMS, under the authority stipulated by a data use agreement between CMS and Brandeis [see Appendix F for Data Use Agreement for Use of CMS Beneficiary Data]. The plan for identification and solicitation of Medicare beneficiaries to participate, and protection of beneficiaries’ privacy, has been approved by the CMS Privacy Board [see Appendix G for CMS Privacy Board Notice of Approval and Appendix H for Plan Approved by CMS Privacy Board for Beneficiary Privacy Protections].

A letter will be sent to potential respondents, drafted and approved by the CMS Privacy Officer [see Appendix I for Introductory Letter]. This letter explains the purpose of the study and invites participation. It emphasizes the voluntary nature of the study, and the right of the beneficiary to refuse to participate with no consequences to their Medicare benefits. Brandeis will follow up this letter with a phone call to beneficiaries to see if the beneficiary would like to participate in the evaluation and, if so, to schedule a time to meet to conduct the interview in the beneficiary’s home. During this call, Brandeis will review a scripted verbal informed consent with beneficiaries to review the purpose of the study, and its voluntary and confidential nature [see Appendix J for Verbal Informed Consent Scripts]. At this time, Brandeis will assure beneficiaries that any information they provide during the interview will remain confidential [see Appendix J].

At the time of the interview, the purpose of the study and its voluntary nature will again be explained, and the beneficiary’s decision to continue with the interview and answer questions will be taken to imply informed consent.

In addition to approval by the CMS Privacy Board, the proposal for protection of beneficiary’s privacy, the introductory letter, the verbal informed consent script and the semi-structured interview protocols have been approved by the Committee for Protection of Human Subjects of Brandeis University [see Appendix K for Brandeis University IRB Approval Notification].

The privacy of sampled beneficiaries and respondents will be strictly maintained and the confidentiality of the information obtained will be closely guarded. All information revealed during the interviews will be confidential. No identifying information will be released, and names, addresses and other identifying information will be removed from the final dataset. Results of analysis will be reported only in aggregate, with no identifying information provided.

Raw data and data files will physically reside in locked offices at the Schneider Institute in the Heller School for Social Policy and Management at Brandeis University, with password protection for computers, hard drives, folders and individual files. Any backup data sets stored on external media will remain in locked office space at the Schneider Institute. Access to raw data, data files and backup data sets will be restricted to signatories to the Data Use Agreement between CMS and Brandeis University [see Appendix F for Data Use Agreement for Use of CMS Beneficiary Data]. The Project Director and Brandeis University have responsibility for assuring confidentiality of the data. All data files will be destroyed or returned to CMS at the end of the evaluation [see Appendix H for Plan Approved by CMS Privacy Board for Beneficiary Privacy Protections].

11. Sensitive Questions

There will be no sensitive information collected during any interviews of beneficiaries. There are no sensitive questions contained in the site interview protocols. The semi-structured nature of the interview process allows for adequate flexibility to adapt questions and explanations to the needs of the respondent. As a further step, respondents are told that they are free not to answer any question that they find troubling, or for any other reason.

12. Burden Estimate (Hours & Wages)

We estimate that the enrollment data collection will add little burden to the process of beneficiary intake that is already in place at the Demonstration sites. The information regarding the beneficiary’s prior usage of adult day services will be collected by means of an additional question incorporated into the initial assessments that are routinely conducted in beneficiaries’ homes, as required by Medicare. Demonstration site staff will record and prepare the information for transmission to Brandeis. Although the number of beneficiaries subject to enrollment data collection will vary by agency, we estimate that, on average, each site will spend no more than one hour per week collecting and transmitting Demonstration enrollment data, for a total of 260 burden-hours for each year of the Demonstration. (One hour per week X 52 weeks per year X 5 sites = 260 hours per year.) Using $14.68 per hour based on the Bureau of Labor Statistics 2004 overall U.S. average for a financial administrative support specialist, we estimate the total wage burden for enrollment data collection to equal $3,817 per year each of the three years of the Demonstration, for a total wage burden of $11,451.

We estimate that the beneficiary face-to-face interviews will take 45 minutes, on average, to complete. Burden-hours for beneficiary face-to-face interviews are estimated at 37.5 hours (50 beneficiaries interviewed for an average of 45 minutes per interview). Using $10.40 per hour as the average salary for seniors (based on figures used by CMS for burden estimation in a 2003 OMB submission of Medicare enrollees being conducted by Brandeis University), we estimate the total wage burden to equal $390.

13. Capital Costs

There are no capital costs.

14. Annualized Costs to the Federal Government

The cost to the federal government of conducting the data collection is specified in the contract with Brandeis to be $167,431 for 1,687 person-hours. Seventy-five percent of these costs and hours are associated with Task 3b, “Site visits and phone interviews”, and 25% are associated with Task 3c "Evaluation data from demonstration sites" in Brandeis' final budget for RFP No. CMS-05-031/ERD.

15. Program Changes

This is a new program.

16. Publication and Tabulation Dates

Neither CMS nor Brandeis, as the evaluator, plans to publish collected data in a form that will, in any way, permit identification of individual beneficiaries. Analysis of aggregate results will assist CMS in deciding policy regarding expansion of coverage for medical adult day services under the home health benefit for Medicare beneficiaries. Selected aggregate data may be used in professional publications.

Timeline for the Interviews Data Collection and Report to Congress
Task Description	Date
Site visits and interviews	December 2006 – Feb 2009
Final Case Study to CMS	June 2007
Interim Report to CMS	March 2008
Final Report to CMS	April 2009
Final Summation for Report to Congress	September 2009

17. Expiration Date for OMB Approval of Information Collection

CMS plans to display the expiration date.

18. Exceptions to “Certification for Paperwork Reduction Act Submissions” Statement

Not applicable. There are no exceptions to the certification statement.

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

The two types of data being collected involve different sampling issues. First, there are no sampling issues for the collection of enrollment data, since all beneficiaries who begin home health episodes and are eligible for the demonstration will be asked the question about whether they previously participated in adult day care.

Second, there is no randomization of site selection for the case studies, since the Demonstration design selected five study sites, and the RTOP calls for case studies of each of the five sites.

At each site, beneficiary face-to-face interviews will be conducted with six Demonstration participants, and four beneficiaries who declined to participate, for a total of 50 beneficiary interviews. Selection criteria for the beneficiary sample include: 1) participating or declining to participate; 2) date of home health episode of care; 3) use of adult day care services in the two weeks prior to joining the Demonstration or declining to join; and 4) gender. Beneficiary selection criteria and methods are discussed in detail in Appendix C. Beneficiaries meeting the criteria will be selected from the home health census, until a total sample size of 50 beneficiaries (10 per site) is drawn for face-to-face interviews. If any of the selected beneficiaries cannot be contacted or decline to participate, replacement beneficiaries will be added to the sample until 10 beneficiary interviews are completed per site.

2. Procedures for the Collection of Information

The enrollment data will be collected by home health clinical staff in the course of their completion of their initial patient assessment and other intake questions, as described in Appendix C. The site visits and interviews will be conducted by the field team consisting of three members of our research team. The face-to-face interviews with beneficiaries will take 45 minutes, on average. Complete specifications for the interview process are contained in Appendix C. Interview guides for the semi-structured interviews are found in Appendix E.

3. Methods to Maximize Response Rates and Deal with Issues of Non-response

Response rates for the enrollment question about prior use of adult day care will be very high, since it is part of the clinical intake. In order to increase response and simplify the question for beneficiaries, we will not ask them to make the distinction between social and medical day services.

Issues of non-response to the case study interviews are minimized by CMS’ provision of accurate contact information for respondents. In addition, the introductory letter from the CMS Privacy Board, and the initial telephone contact by Brandeis, explaining the purpose and confidentiality of the study, will be used to maximize response rates. During administration of the interviews, respondents will again be assured regarding confidentiality and privacy, and the evaluation’s importance to Medicare will be reinforced.

Based on prior experience interviewing elders at home, we expect a positive response rate of 75% among the beneficiaries we invite to participate. We will continue to invite participation until the total sample of 10 beneficiaries per Demonstration site is achieved. Study drop-outs will be replaced to maintain the sample of 10 beneficiaries per Demonstration site.

4. Tests of Procedures or Methods to be Undertaken

The interview protocols will be pre-tested at the first Demonstration site visit with two Medicare beneficiaries. This pre-testing will assess whether any question, or the overall interview, is too sensitive or burdensome. The analysis of interview results will use qualitative case study methods rather than statistics.

The enrollment data, including the question about prior use of adult day care, will be used for comparisons of Demonstration enrollees, eligible beneficiaries who refused to participate, and those deemed by the sites to be ineligible (based on criteria established by each site). For beneficiaries in each group, we will calculate and compare their mean use of and expenditures for Medicare-covered services by category (e.g. home health services, inpatient stays, skilled nursing stays, durable medical equipment) during the pre-demonstration and demonstration time periods. Examination of improvements in health or quality of life outcomes over time is permitted by linking interviews and utilization data to OASIS quality data.

Using CMS databases, a control group of matched comparison subjects will be created to permit analysis of outcomes for Demonstration participants compared to similar non-participating home health subjects within the same state. Two samples of comparison subjects will be created, using instrumental variable and propensity score methods, to compare Medicare utilization and expenditure measures between participant and control subjects to assess the impacts of the Demonstration. The methods for matching and selection of comparison subjects are described in detail in Appendix L, Analysis of Data.

5. Statistical Contact

Grant Ritter, PhD

Senior Scientist

Brandeis University

Schneider Institute for Health Policy

South Street, Waltham, MA 02454

e-mail address - ritter@brandeis.edu

telephone number (781) 736-3872.

Appendices

Appendix A: Legal Basis

Appendix B: Evaluation Research Questions Specified by RTOP No. CMS-05-031/ERD

Appendix C: Site Visits and Phone Interviews (Task 3b)

Appendix D: Satisfaction Survey (Task 3d)

Appendix E: Beneficiary Interview Guides

Appendix F: Data Use Agreement for Use of Centers CMS Beneficiary Data

Appendix G: CMS Privacy Board Notice of Approval

Appendix H: Plan Approved by CMS Privacy Board for Beneficiary Privacy Protections

Appendix I: Introductory Letter

Appendix J: Verbal Informed Consent Scripts

Appendix K: Brandeis University IRB Approval Notification

Appendix L: Analysis of Data

Appendix A: Legal Basis

H.R.1

Medicare Prescription Drug, Improvement, and Modernization Act of 2003 (Enrolled as Agreed to or Passed by Both House and Senate)

SEC. 703. DEMONSTRATION PROJECT FOR MEDICAL ADULT DAY-CARE SERVICES.

(a) ESTABLISHMENT- Subject to the succeeding provisions of this section, the Secretary shall establish a demonstration project (in this section referred to as the `demonstration project') under which the Secretary shall, as part of a plan of an episode of care for home health services established for a medicare beneficiary, permit a home health agency, directly or under arrangements with a medical adult day-care facility, to provide medical adult day-care services as a substitute for a portion of home health services that would otherwise be provided in the beneficiary's home.

(b) PAYMENT-

(1) IN GENERAL- Subject to paragraph (2), the amount of payment for an episode of care for home health services, a portion of which consists of substitute medical adult day-care services, under the demonstration project shall be made at a rate equal to 95 percent of the amount that would otherwise apply for such home health services under section 1895 of the Social Security Act (42 U.S.C. 1395fff). In no case may a home health agency, or a medical adult day-care facility under arrangements with a home health agency, separately charge a beneficiary for medical adult day-care services furnished under the plan of care.

(2) ADJUSTMENT IN CASE OF OVERUTILIZATION OF SUBSTITUTE ADULT DAY-CARE SERVICES TO ENSURE BUDGET NEUTRALITY- The Secretary shall monitor the expenditures under the demonstration project and under title XVIII of the Social Security Act for home health services. If the Secretary estimates that the total expenditures under the demonstration project and under such title XVIII for home health services for a period determined by the Secretary exceed expenditures that would have been made under such title XVIII for home health services for such period if the demonstration project had not been conducted, the Secretary shall adjust the rate of payment to medical adult day-care facilities under paragraph (1) in order to eliminate such excess.

(c) DEMONSTRATION PROJECT SITES- The demonstration project established under this section shall be conducted in not more than 5 sites in States selected by the Secretary that license or certify providers of services that furnish medical adult day-care services.

(d) DURATION- The Secretary shall conduct the demonstration project for a period of 3 years.

(e) VOLUNTARY PARTICIPATION- Participation of medicare beneficiaries in the demonstration project shall be voluntary. The total number of such beneficiaries that may participate in the project at any given time may not exceed 15,000.

(f) PREFERENCE IN SELECTING AGENCIES- In selecting home health agencies to participate under the demonstration project, the Secretary shall give preference to those agencies that are currently licensed or certified through common ownership and control to furnish medical adult day-care services.

(g) WAIVER AUTHORITY- The Secretary may waive such requirements of title XVIII of the Social Security Act as may be necessary for the purposes of carrying out the demonstration project, other than waiving the requirement that an individual be homebound in order to be eligible for benefits for home health services.

(h) EVALUATION AND REPORT- The Secretary shall conduct an evaluation of the clinical and cost-effectiveness of the demonstration project. Not later than 6 months after the completion of the project, the Secretary shall submit to Congress a report on the evaluation, and shall include in the report the following:

(1) An analysis of the patient outcomes and costs of furnishing care to the medicare beneficiaries participating in the project as compared to such outcomes and costs to beneficiaries receiving only home health services for the same health conditions.

(2) Such recommendations regarding the extension, expansion, or termination of the project as the Secretary determines appropriate.

(i) DEFINITIONS- In this section:

(1) HOME HEALTH AGENCY- The term `home health agency' has the meaning given such term in section 1861(o) of the Social Security Act (42 U.S.C. 1395x(o)).

(2) MEDICAL ADULT DAY-CARE FACILITY- The term `medical adult day-care facility' means a facility that--

(A) has been licensed or certified by a State to furnish medical adult day-care services in the State for a continuous 2-year period;

(B) is engaged in providing skilled nursing services and other therapeutic services directly or under arrangement with a home health agency;

(C) is licensed and certified by the State in which it operates or meets such standards established by the Secretary to assure quality of care and such other requirements as the Secretary finds necessary in the interest of the health and safety of individuals who are furnished services in the facility; and

(D) provides medical adult day-care services.

(3) MEDICAL ADULT DAY-CARE SERVICES- The term `medical adult day-care services' means--

(A) home health service items and services described in paragraphs (1) through (7) of section 1861(m) furnished in a medical adult day-care facility;

(B) a program of supervised activities furnished in a group setting in the facility that--

(i) meet such criteria as the Secretary determines appropriate; and

(ii) is designed to promote physical and mental health of the individuals; and

(4) MEDICARE BENEFICIARY- The term `medicare beneficiary' means an individual entitled to benefits under part A of this title, enrolled under part B of this title, or both.

Appendix B: Evaluation Research Questions Specified by RTOP No. CMS-05-031/ERD

Operational Issues

How did participating home health agencies determine which of their Medicare clients would be offered an opportunity to participate in the demonstration, and receive a portion of their home health services at a MADC facility? What criteria were used by HHAs to make these determinations? How did these HHA policies affect the selection of beneficiaries into the demonstration?
What was the impact of the demonstration on HHA patient intake practices, care planning, care delivery and discharge planning?
How did HHAs determine which services would be provided in MADC facilities and which would be provided in beneficiaries’ homes?
How did HHAs ensure coordination among their own staff providers and contracted providers, including MADC providers?

Characteristics of Participants and Non-Participants

How many beneficiaries participated in the demonstration, and elected to receive a portion of their home health care in a MADC facility? How many clients of participating HHAs were not offered participation in the demonstration? How many clients of participating HHAs were offered participation in the demonstration but declined?
What percentage of participating beneficiaries withdrew from the demonstration during their episode of care, and reverted to receiving all of their home health care at home? Why did beneficiaries withdraw from the demonstration?
How did beneficiaries who participated in the demonstration differ from those that were not offered the demonstration (including those for whom the demonstration was determined to be medically contraindicated, and those who were excluded for other reasons), in terms of their medical conditions, need for rehabilitative services, availability of family or other social supports or other important factors? How did beneficiaries who participated differ from those who were offered participation but declined?
What motivated beneficiaries and their caregivers to participate in the demonstration? For those who declined participation, what were their motives for declining?
How did participating HHAs compare to other HHAs in terms of their size, organizational structure, or the composition of their Medicare home health patient populations?
Did the demonstration result in a change in the patient mix for the participating HHAs?

Impact on Use of Services

What was the impact of participation in the demonstration on the amount and types of home health services provided beneficiaries under their plans of care?
How did participation in the demonstration affect the setting in which Medicare beneficiaries received home health services? What kinds of services were demonstration participants most likely to receive in a MADC facility rather than in their homes?
What was the impact of participation in the demonstration on beneficiaries’ use of Medicare services other than home health?
What percentage of demonstration participants were MADC clients prior to joining the demonstration? How many began using MADC in order to participate?
Did the demonstration result in an increase in the numbers of patients enrolled with participating MADC facilities? Did this increase result from an influx of Medicare beneficiaries who were users of the Medicare home health benefit?
For dual eligibles, what was the impact of participation in the demonstration on their use of Medicaid services? Was there any evidence that Medicare beneficiaries were becoming Medicaid eligible in order obtain funding for MADC services, so that they could participate in the demonstration?

Impact on Quality and Outcomes

How did the quality of care for demonstration participants differ from what was received by beneficiaries with the same health conditions who received all of their home health services at home?
How did the health and functional status outcomes of demonstration participants compare with the outcomes for beneficiaries with same health conditions who received all of their home health services at home?

Satisfaction

How did the level of satisfaction of demonstration participants and their caregivers compare with that of beneficiaries with same health conditions who received all of their home health services at home?
How was the demonstration viewed by beneficiaries, home health and MADC providers (professional and administrative staff), patient advocacy groups, physicians, discharge planners and other key constituencies?

Market Impacts

Has the existence of the demonstration affected the level or frequency of use of Medicare home health services by beneficiaries within participating home health agencies’ market areas?
How has the existence of the demonstration project affected the availability and use of MADC services in the participating providers’ market areas?

Cost Impacts

How did the costs for Medicare covered services (both home health care services and other services) for demonstration participants compare with costs for beneficiaries with same health conditions who received all of their home health services at home?
How did out-of-pocket costs for demonstration participants compare to those of beneficiaries with same health conditions who received all of their home health services at home, including out-of-pocket costs for Medicare-covered services, MADC services, transportation and other related costs?
What was the overall impact of the demonstration on federal Medicare expenditures, and federal and State Medicaid expenditures?

Provider Impacts

How did participation in the demonstration affect participating HHAs’ financial performance? What was the marginal effect of the demonstration on HHAs’ profit or loss for the episodes involved?
How did participation in the demonstration affect participating MADC facilities’ financial performance?

Overall Impact and Next Steps

Can the findings from this evaluation be used to predict the likely outcome (in terms of cost to Medicare, cost to beneficiaries, costs to States, impact on the home health care or MADC markets) of allowing home health services to be provided in MADC settings as a permanent feature of the Medicare program?
Can any changes to the demonstration be suggested that would improve its operation or outcomes, or that would improve its suitability for adoption as a permanent feature of the Medicare program?

Appendix C: Site Visits (Task 3b)

Task 3b Section of Final Design Report

for

Evaluation of the Medical Adult Day-Care Services Demonstration

DESY - append

Evaluation of the Medical Adult Day-Care Services Demonstration

Final Design Report

Contract Number 500-00-0031/5

Paul Boben, Ph.D., Project Officer

Walter Leutz, Ph.D., Project Director

Brandeis University

Waltham, MA

May 25, 2006

Site Visits and Phone Interviews (Task 3b) Section of Final Design Report

Site visits and phone interviews (Task 3b)

Overview: The goals of the site visits are to understand how the Demonstrations operate and to develop questions for the consumer satisfaction survey (Task 3d). Our case

study team will visit each Demonstration site once to interview managers and clinicians at both HHA and MADS programs, observe MADS operations, and interview collateral contacts including State Medicaid Agencies, competing Medical Day Service centers, and Advocacy Groups. Our site visit team will also interview beneficiaries in their homes with their caretakers about their choice to participate, experiences being served, and overall satisfaction with the services they receive. In this section we detail our sample selection, site interview protocols, approach to conducting the site visits, and plans for analyzing the site visit data.

Sample Selection: There are no site selection issues for the case studies, since the demonstration design selected five study sites, and the RTOP calls for case studies of each of the five sites. However, there are sub-site selection issues as three of the five sites have multiple MADS centers involved. In these cases, we will consult with the SC, the Project Officer (PO), and the study sites themselves to identify the MADS that is most advanced in implementing the project at the time of the site visit as indicated by enrollment levels.

At each site visit, we expect to interview, in both the HHA and the selected MADS, at least one administrator (one who can speak to finance and agency policies) and two clinicians (one who can speak to selection and in-take, and one who can speak to care planning and care provision). Our best understanding is that these functions will be represented by the Executive Director, the Medical Director, and the Director of Patient Services. To the degree that these functions are arrayed differently in different agencies, we will adjust our site interview schedule accordingly, interviewing different or additional respondents as needed. Consultations with the SC, the PO, and the sites themselves, in advance of our scheduled visit, will be used to clarify and agree upon who are the appropriate interview respondents at each site. We discuss this process in more detail in subsequent sections.

Each case study will also involve interviews with representatives of three collateral organizations: the State Medicaid Agency, competing MADS (if applicable), and advocacy groups. Identification and selection of these key informants will involve consultation with the SC, PO and the site themselves. If need be, we will additionally contact the Area Agency on Aging for information about competing MADS and relevant advocacy groups. In contrast with interviews with Demonstration site staff and beneficiaries, we will conduct interviews with collateral agencies by phone in advance of our site visit. Information gathered from these interviews will help inform the actual site visit, and if need be, follow-up phone contact will be made after the site visit to clarify new or conflicting information that surfaces during the site visit. We estimate the total sample of representatives at collateral agencies per site to range between 3 and 5.

In addition to interviewing staff at each of the Demonstration sites and collateral agencies, our site visit team will also interview beneficiaries: six beneficiaries who are participating in the Demonstration, and four beneficiaries who were offered an opportunity to participate in the Demonstration but declined. Selection criteria for the beneficiary sample include: 1) participating or declining to participate; 2) date of home health episode of care; 3) use of adult day care services in the two weeks prior to joining the Demonstration or declining to join; and 4) gender. We establish criterion two (date of service) to ensure that respondents have sufficient experience in the Demonstration and/or adequate recall of their experience to provide meaningful feedback to the interviewers. To that end, we will select among the pool of beneficiaries who are either actively in an episode of care (defined as at least 21 days into a 60 day episode) or within two weeks of completing an episode of care. We impose criterion three (prior use of adult day service) because we hypothesize that this condition may influence the degree to which a beneficiary is inclined to participate and/or their overall experience in the Demonstration. And criterion four (gender) is to ensure that males are sufficiently represented in our sample. In general, men tend to be underrepresented in adult day care settings. These criteria produce the following beneficiary sample:

	Beneficiary Sample Selection Criteria Demonstration Participant
Prior MADS Use	Yes		Declined
Yes	1 Male	2 Female	1 Male	1 Female
No	1 Male	2 Female	1 Male	1 Female

Each Demonstration site will provide data to Brandeis to support the identification and selection of our beneficiary sample. As part of the sites' Terms and Conditions, each Demonstration site will agree to collect the data specified in Table 1. As detailed in Task 3c, in the month prior to the scheduled site visit, the Demonstration site will forward to Brandeis Table 1, along with linked beneficiary contact information (beneficiary name, beneficiary primary caregiver if applicable, address, and phone number). Brandeis will enter these data into a spreadsheet, select the universe of beneficiaries that satisfy the date of service criterion and from among this group, generate a random sample of 10 beneficiaries stratified by the remaining selection criteria. In the section Conducting Site Visits we explain the subsequent process for contacting selected beneficiaries, introducing the study, securing informed consent for their participation, and scheduling and conducting the interviews.

Each HHA will enter into a written agreement with Brandeis University to collect and share these data about the enrollment, service utilization, and personal contact information for beneficiaries who could have been offered participation in the Demonstration. The items in the data agreement, as well as the collection procedures, will be submitted to and approved by CMS. The agreement is required by and will be approved by the Brandeis IRB for the protection of human subjects.

Interview Protocols: There are 11 categories of respondents for the on-site interviews. Nine of these fall under the category of professional staff and two under the category of beneficiaries (Table 2). To guide these interviews – both the face-to-face interviews with Demonstration site staff and beneficiaries, as well as our phone interviews with representatives of collateral agencies - we developed loosely structured expert interview guides for each category of respondent (see Appendix E). An expert interview guide, detailed to the particular interviewee, specifies the topics of the questions the researcher seeks to answer, and the information needed to provide the answer. The interviewers will ask the questions in an order, and with specific wording that conforms to the specific setting and respondent, and to the pattern of responses that develop in the interview. A series of follow-up probes are provided to ensure that each component of the information is obtained. The respondent will be given flexibility to provide the information in his and her own way and the opportunity to introduce additional information relevant to the answer being sought. This approach allows knowledgeable respondents to have input into the data collection and add to it while insuring consistency of data collection across like categories of respondents.

Table 2: Key Informant Categories Per Site Visit
Professional Staff	Beneficiaries
HHA Administrator	Demonstration Participants
HHA Clinician (outreach and intake)	Beneficiaries Declined to Participate
HHA Clinician (service delivery)
MADS Administrator
MADS Clinician (outreach and intake)
MADS Clinician (service delivery)
State Medicaid Agency
Competing MADS
Advocacy Group
Total Professional Staff Categories = 9	Total Beneficiary Categories = 2

Specific question topics for each category of key informant guide were derived from the eight analytic domains articulated in our proposal (operational issues, beneficiary characteristics, utilization, quality and outcomes, satisfaction, market effects, cost and other provider effects) and the questions we proposed to address qualitatively under each domain. Domains and question topics were sorted by the expertise of the respondent. For example, under the domain operational issues, we seek to understand, among other things, how participating home health agencies determine which Medicare clients to offer an opportunity to receive a portion of their home health services at a MADS facility. The protocol designed for the HHA Administrator includes several topics about the policy-making process by which participant selection criteria were established, the factors that were considered, and the intended effect. In contrast, protocols tailored to marketing and in-take staff, focus on implementation issues and outcomes: how well did the selection criteria work, problems encountered, and actual effect (how many and what type of beneficiary is being targeted and excluded from the Demonstration as a result of the established selection criteria). Not all domains are relevant to each key informant, but rather each protocol is tailored to the particular respondent and his/her area of expertise.

With respect to the site staff interviews, we will be at most using the same interview guide for five individuals (executive directors at each of the five HHA Demonstration sites, for example). Therefore we do not believe the site staff interview guides require OMB clearance. In contrast, the same beneficiary guide will be used for 20 beneficiaries who were offered an opportunity to participate in a Demonstration but refused (4 per site x 5 sites) and 30 beneficiaries who are participating in the Demonstration (6 per site x 5 sites), although these questions will adapted to the uniqueness of each Demonstration. Regardless, we believe the beneficiary interview guides require OMB review and clearance. Additionally, since our interviews do involve human subjects, all guides and procedures will need to be reviewed and approved by the Brandeis IRB. We will formally submit our application to the Brandeis IRB as soon as the final design report is approved by CMS. We have already initiated informal discussion with the Brandeis IRB staff and based on feedback to-date believe this study falls within the guidelines of an expedited review.

Conducting Site Visits: Our field team consists of three members of our research team (Leutz, Gurewich and Houghton). To organize the site visits, document review, preparatory interviews, and subsequent analysis and writing, we will assign a Brandeis site coordinator for each site (Leutz will oversee 2 sites and Gurewich 3 sites). For all qualitative work related to the site, the site coordinator will be the contact point for staff at the site, the SC, and Brandeis/Booz staff. We have used this arrangement successfully in numerous other qualitative field studies. On the visits, the site coordinator will be accompanied by one other member of the field team.

To prepare for a visit to a Demonstration site, well in advance of the visit, we will solicit and review documents from the Demonstration site and the SC (Task 3a). We will also conduct phone interviews with representatives of the site prior to our visit to familiarize ourselves with Demonstration design features and site operations. These phone calls may continue even after the site visit is completed to clear up questions and collect collateral information. We will work with the SC to identify Demonstration site coordinator at both the HHA and MADS sites, and will work with them to identify the key informants at each site (managers and clinicians) who will be best able to answer our questions when we visit the site. Together, we will confirm site visit dates and establish an interview schedule. We anticipate spending at least half a day at each Agency, with time available for observing operations at the MADS. During the site visit, we will conduct interviews with HHA and MADS staff in teams of two researchers: one member of the team responsible for guiding the interview and the other for taking handwritten notes. We will also ask respondent site staff if we can digitally record the interview.

To prepare for the component of our site visits that involves beneficiaries, we will also work with the site coordinator in advance of our site visit to secure the beneficiary enrollment data needed to support our selection of beneficiary respondents. As mentioned, as part of the Sites Terms and Conditions, each Demonstration site will agree to forward directly to Brandeis Table 1 along with beneficiary contact information. We will use the data specified in Table 1 to identify a random sample of beneficiaries (6 participants and 4 decliners).

In advance of the site visit, the following procedures have been approved by the CMS privacy board for contacting beneficiaries for the interviews. First, we will send the beneficiaries the Privacy Board letter to beneficiaries (Appendix I) notifying them of the phone call they will receive from Brandeis. We will follow up the letter with a phone call to the beneficiary (or his/her caregiver, if appropriate) and talk through a scripted informed consent in which we will describe the study, the types of questions we would like to ask them, inform them that the information they provide will remain confidential, and that it is optional for them to participate in the study and that not participating will not affect the services they receive in the Demonstration or from the home health agency (see Appendix J for Verbal Informed Consent; one for demonstration participants and one for beneficiaries who declined to participate in the demonstration). If a beneficiary verbally agrees to participate in the study, we will schedule a time to meet in his/her home during our scheduled site visit. Seeing beneficiaries in their homes and also observing them in the day care setting will provide us with rich detail about the opportunities and challenges of delivering care in two settings.

At the time of the actual interviews, we will again review the purpose of the study, how the information beneficiaries provide will be used, etc., to ensure that they are still consenting to the interview. In contrast to our interviews with professional staff, which we will conduct in teams of two, we will conduct the beneficiary interviews one-on-one, assigning one member of the team to 5 of the beneficiary interviews and the other member of the team to the remaining 5 beneficiaries. We will again request the permission to digitally record the interviews, and will also take hand-written notes.

Management and Analysis of Data: The full analysis of case study data will begin when the interviewers write up their notes with the aid of digital recordings of the interviews (with the permission of respondents). The notes will be organized into the topics they address, allowing for comparison of answers across respondents. When there are conflicts among staff from the same organization discussing the same thing, the team will decide whether this represents legitimate differences in perspectives or a misunderstanding the team needs to clear up by further discussions with the site. A range of responses from beneficiaries will mean different things for different types of issues, e.g., one would hope for some consistency in the way beneficiaries were presented options to participate, whereas there is likely to be more variation in satisfaction.

In reviewing the case study data for each site, including both interviews and documents (Task 3a), the site liaison for that site will write analytic memos, tables, flow charts, and other illustrative and synthetic documents to compile the story of how each site operates, as well as issues related to operation. Through the documents obtained on the methods of reimbursing MADS, through data from the sites/SC on enrollment and services delivered, and through the case study interviews, we will be able to tell the story behind the financial performance data. There will be a parallel set of memoranda and analyses from the beneficiaries’ point of view. These memoranda and other material will be shared first with the other member of the Brandeis site visit team, and then refined for sharing with the rest of the study team. At this point, we will hold extended meetings for presentation and discussion of findings to make sure the descriptions of each site stand up to questions. Additional analysis, or even additional data collection may be necessary in some area at this time. When the team is comfortable that all aspects and issues of the case study are covered, they will prepare the Draft Study Report (See Task 4b for the timing and outline).

The timeline allows for one month for CMS to review of the Draft Case Study Report. Based on the comments we will revise and deliver the Final Case Study Report within one month of receiving CMS comments. We have set delivery in Month 21 of the contract.

Appendix D: Satisfaction Survey (Task 3d)

Task 3d Section of Final Design Report

for

Evaluation of the Medical Adult Day-Care Services Demonstration

DESY - append

Evaluation of the Medical Adult Day-Care Services Demonstration

Final Design Report

Contract Number 500-00-0031/5

Paul Boben, Ph.D., Project Officer

Walter Leutz, Ph.D., Project Director

Brandeis University

Waltham, MA

May 25, 2006

Satisfaction Survey (Task 3d) Section of Final Design Report

Satisfaction Survey (Task 3d)

The satisfaction survey will be designed to assess the domains and degrees of satisfaction with home health care among participants and refusers. It will also collect information about the types and extent of out-of-pocket costs related to home care and adult day services. The survey will be developed based on staff and beneficiary/family member interviews in the Phase 1 site visits.

In regard to satisfaction, the interviews will focus on identifying and understanding areas and types of beneficiary satisfaction related to various aspects of the demonstration. The interviews will probe for satisfaction with care coordinators, service providers, settings, socialization, costs, value for the money, choice, quality, and other issues that arise in the interviews. Because we will have interviewed staff at each site concerning their perceptions of beneficiary satisfaction and dissatisfaction, we will conduct the beneficiary interviews with a longer list of items to probe, including some items that will be specific to the operations of each site. The interviews will allow beneficiaries to communicate their specific experiences and feelings about their participation in the demonstration and will help us develop a satisfaction survey that addresses a range of beneficiary experiences.

In regard to out-of-pocket expenses, the interviews will probe for costs related to travel, MADC fees, assistance from Medicaid and other sources, and uncovered home care services. Since these will be qualitative, open-ended interviews designed to elicit and probe for respondent experiences that cannot entirely be anticipated, we expect that respondents will report other items as well. Synthesis of the interview responses related to beneficiary spending will inform the choice of cost categories to be included in the satisfaction survey.

Because the content of the satisfaction questions will be synthesized from the interviews, it is not now possible to propose what they will cover. We have used qualitative background interviews to develop questions for quantitative follow-up surveys for similar populations (Leutz, Capitman et al. 2001; Leutz and Capitman forthcoming). The satisfaction survey will be designed to be administered by telephone. It will have approximately 15 items in yes/no and scale formats and take about 15 minutes. Depending on the interview findings about expenses, the questions in this area may be either yes/no regarding categories or open-ended. The survey will be pre-tested, revised, and administered in Phase 2.

Appendix E: Beneficiary Interview Guides

Demonstration Participants

Living Arrangement

Q1. Household members.

Q2. Needs for personal and household assistance.

Cooking, cleaning, laundry
Personal care
Walking or getting out of the bed or chair
Walker or a wheelchair

Q3. Who provides personal and household assistance; frequency.

Experience with Demonstration

Q1. Reasons and expectations for participating in the demonstration.

Q1a. Attendance at adult day services before the Demonstration.

Demo center or another center

Q2. Home health services received.

At home
At the adult day center
Beneficiary/caregiver role on decision-making (i.e. level of felt choice)

Q3. Activities at the adult day care center.

Positive experience
Negative experience

Q4. Medicaid enrollment.

Current
Prior to Demonstration
Demonstration staff facilitation of enrollment into Medicaid?

Q5. Out-of-pocket costs in the last two weeks: home health services in the home.

Nursing, personal care, household assistance, physical therapy
Level of satisfaction with value received, and reasons

Q6. Out-of-pocket costs: adult day center services.

Daily charge, transportation, personal services, other
Level of satisfaction with value received, and reasons

Q7. Level of satisfaction with home health services under the Demonstration.

Coordination of home health services with adult day and other services
Home health services in the adult day care center
Home health services in the home
Degree of choice in quantity and type of home health services
Overall quality of home health services received

Q8. Satisfaction with adult day services under the Demonstration.

Social interactions with other adult day participants
Overall quality of adult day care services
Degree of choice in quantity and type of adult day care services

Q9. Overall experience with the Demonstration.

What did you like best about the Demonstration?
What do you like least about the Demonstration?

Q10. Aspects of the Demonstration that should be changed.

Beneficiaries Offered But Declined To Participate

Living Arrangement

Q1. Household members.

Q2. Personal and household assistance.

Cooking, cleaning, laundry
Personal care
Walking or getting out of the bed or chair
Walker or a wheelchair

Q3. Who provides personal and household assistance; frequency.

Reasons for Declining to Participate

Q1. Reasons for decision not to participate in the Demonstration.

Q2. Attendance at an adult day care center: Demonstration center or other.

Current and/or past use
If yes: what activities or services

Q3. Changes to the Demonstration that would have made it more attractive.

Experience with Home Health Services

Q1. Types of home health services currently receiving.

Q2. Current Medicaid enrollment status.

Q3. Out of pocket costs in the last two weeks: home care services.

Nursing, personal care, household assistance, physical therapy
Level of satisfaction with value received, and reasons

Q4. (If applicable) Out of pocket costs: adult day center services.

Daily charge, transportation, personal services
Level of satisfaction with value received, and reasons

Q5. Level of satisfaction with home health services.

Coordination of home health services with adult day and other services
Home health services in the home
Degree of choice in quantity and type of home health services
Overall quality of home health services received

Q6. (If applicable) Level of satisfaction with adult day care services.

Social interactions with other adult day participants
Overall quality of adult day care services
Degree of choice in quantity and type of adult day care services

Appendix F: Data Use Agreement for Use of CMS Beneficiary Data

Appendix G: CMS Privacy Board Notice of Approval

Appendix H: Plan Approved by CMS Privacy Board for Beneficiary Privacy Protections

Appendix I: Introductory Letter

Appendix J: Verbal Informed Consent Scripts

(Verbal) Informed Consent for

Medical Adult Day Services (MADS) Demonstration

Participants

Brandeis University is conducting an evaluation of the Medical Adult Day Services Demonstration. We understand that you are currently participating in the Demonstration through (insert name of home health agency) and are receiving some of your home health services at (insert name of MADS). We are interested to learn how the Demonstration works and whether beneficiaries who are participating in the Demonstration like it. We would like to invite you to participate in the Brandeis evaluation study. If you agree to participate, we will schedule a time to come visit you in your home and will ask you some questions about your experience in the Demonstration: what you like, don’t like, suggested changes, etc.

If you agree to participate, all your answers will be completely confidential. We will not include your name or any other identifiers with your interview responses. The interview should take about 45 minutes to complete. If you do not want to participate in the interview, please feel free to say so. Your decision to participate or not participate in the Brandeis evaluation will not affect the quality of care you receive from (insert home health agency and MADS).

Would you like to participate in the Brandeis evaluation study of the MADS Demonstration? (If beneficiary indicates “yes,” proceed with scheduling a time for the interview; If beneficiary indicates “no,” thanks them for their time and end the call). .

If you are interested in the results of the study or have any questions, please feel free to contact (insert name of Brandeis contact) at Brandeis University in Waltham, Massachusetts at (insert Brandeis contact number).

(Verbal) Informed Consent for

Medical Adult Day Services (MADS) Demonstration

Decliners

Brandeis University is conducting an evaluation of the Medical Adult Day Services Demonstration. We understand that you were offered an opportunity to participate in the Demonstration at (insert name of home health agency) but declined to participate. We are interested to learn why you declined and would like to invite you to share that information with us by participating in the Brandeis evaluation study. If you agree to participate in the study, we will schedule a time to come visit you in your home to ask you a few questions. Our questions will focus on what you did not like about the Demonstration, the home health and adult day (if applicable) services you presently receive, and your satisfaction with those services

Appendix K: Brandeis University IRB Approval Notification

Brandeis University

The Heller School for Social Policy and Management

July 6, 2006

To: Walter Leutz, Faculty, Heller School for Social Policy and Management

Fr: Christopher Tompkins, Chair, Brandeis Committee for Protection of Human Subjects

Re: Protocol #06-135: Evaluation of the Medical Adult Day Services Demonstration

The Brandeis Committee for Protection of Human Subjects, operating under Federalwide Assurance #FWA00004408, has approved the above-referenced human subjects protocol renewal by expedited review in accordance with 45 CFR §46.101(b)7 (see attached). This approval is valid for one year effective July 6, 2006.

If your research, including data analysis, will continue beyond the approval expiration date of August 7, 2007, please submit a human subjects progress report and continuing review request in time to receive a new approval date that falls on or before that date. If your work will not continue beyond that date, you must still complete and submit this form as a progress report, indicating the date by which your project will terminate.

If you wish to request modifications to your approved protocol, please submit a Modification Request Form to the Committee for review.

Forms and procedures for progress report/continuing review requests and modification requests are available at: http://www.brandeis.edu/osp/hsintro.html.

Please contact Lorrie Clark (x6-2121, mclark@brandeis.edu) should you have any questions.

Mailstop 0:35

PO. Box 549110 Waltham, Massadmsetts 02454-9110

781-736-3800

781-736-3881 Fax

heller.brandeis.edu

Appendix L: Analysis of Data

Task 7f Section of Final Design Report

for

Evaluation of the Adult Day Services Demonstration

DESY - append

Evaluation of the Adult Day Services Demonstration

Final Design Report

Contract Number 500-00-0031/5

Paul Boben, Ph.D., Project Officer

Walter Leutz, Ph.D., Project Director

Brandeis University

Waltham, MA

May 25, 2006

Analysis of Data (Task 7f) Section of Final Design Report

Acquisition and analysis of administrative data from CMS and other sources (Task 7f)

Throughout Phase 2, we will continue to collect administrative data through the CMS website as begun and detailed under Task 3f above.

Overview and objectives of claims analysis. Demonstration effects will be evaluated through comparison of outcomes for participants and comparison groups of similar home health users who did not have the opportunity to participate in the demonstration. Although this demonstration does not provide a randomized control group, it should be possible to locate comparison beneficiaries matched to participants on factors that affect home health use, Medicare expenditures, and health outcomes, so that observed differences will be due to the demonstration. An alternative approach is to use statistical matching, accounting for measured and unmeasured differences to the extent possible. We will pursue both these approaches in this evaluation.

The demonstration intervention allows provision of Medicare home health services in adult day health centers. Thus it presumably supports outcomes for participants that are no worse than those achieved by standard Medicare home health, perhaps at lower or the same cost. It might also allow beneficiaries to receive home health services who otherwise would not have received them. Thus the following potential effects should be assessed using Medicare claims data, comparing situations with and without the demonstration:

For Medicare home health users: Medicare service utilization and expenditures; health and functional outcomes
For all Medicare beneficiaries: access to Medicare home health services
Impacts on Medicare home health agencies

The objectives of the quantitative analyses are therefore to:

Evaluate the impact of participation in the demonstration on beneficiaries’ use and cost of Medicare services and on their home health outcomes
Assess the impact of the demonstration on the rate of use (users per thousand beneficiaries) of Medicare home health services
Assess the impact of the demonstration on participating home health agencies and on the provision of home health in demonstration-served market areas.

Descriptive analyses of utilization by beneficiaries from demonstration agencies. The first Phase 2 analysis will be to compare the three groups of beneficiaries from participating demonstration HHAs: beneficiaries who participate in the demonstration by receiving some services at MADC, beneficiaries who are offered demonstration participation but refused, and beneficiaries who were not offered. Characteristics at entry (age, sex, marital status, home health diagnosis, admitted from hospitalization, functional status, informal support, residential location) will be compared for the three groups, using standard tests for comparison of means. For beneficiaries in each group, we will calculate and compare their mean use of and expenditures for Medicare-covered services by category (e.g. home health services, inpatient stays, SNF stays, DME equipment) during the pre-demonstration and demonstration time periods. Calculations and comparisons of rates of use of home health services among these groups in the demonstration and pre-demonstration periods should provide us important information about the impact of the Demonstration. These analyses could indicate how agencies are able to improve the completeness and efficiency of service delivery among demonstration participants, or may show that fewer services are provided for beneficiaries served in the central group setting. In addition, these comparisons will answer several questions about the demonstration: (1) Were demonstration participants previously enrolled at the home health agencies or are they new enrollees? If they were previously enrolled, (2) did they previously receive at home the services they receive at adult day care under the demonstration, or did they previously do without? and (3) What is the difference in frequency of service receipt for the services provided in adult day care? Answers to these questions will examine the extent to which the demonstration’s adult day care services replace delivery of the same services at home (hopefully with more efficiency), and the extent to which the demonstration’s home health services delivered in adult day care are supplemental and thus provide more care.

Analyses of Outcomes for Demonstration Participants and Matched Comparison Subjects. The Phase 2 analyses will match home health beneficiaries participating in the demonstration with similar home health subjects within the same state for the purpose of comparing outcomes. We propose two complementary approaches to these analyses, relying on two comparison groups. The first approach we describe uses instrumental variable adjustment for participants and non-participants in the sub-area of demonstration location. A second approach analysis will use propensity score matching of beneficiaries drawn from a different sub-region of each state. Using these samples, we will compare a number of Medicare utilization and expenditure measures and use the results to estimate the impacts of the Demonstration. It is advisable that we consider the empirical merits of both approaches in analyzing the data, since instrumental variable methods are only as good as the instruments, and in like manner propensity score matching methods depend on the creation of indices that effectively distinguish demonstration participants from non-participants.

• Instrumental Variable Approach: The instrumental variable approach capitalizes on the fixed nature of MADH services, located in a specific facility, which contrasts with the home-delivered nature of home health services delivery An instrumental variables approach can substitute for random assignment if similar people elect a given service based on a criterion that does not affect outcome (McClellan, McNeil et al. 1994; Horgan, Garnick et al. 2004). In this case, an appropriate instrument is distance from the beneficiary’s residence to the MADH site. The analysis sample is thus home health users in the demonstration market area, including those served by the demonstration home health agency. Participation in the demonstration is indicated by a dummy variable, and distance is incorporated as an instrument. Confining the study sample to the demonstration market areas will account for unmeasured differences in utilization and outcomes among market areas. (The five demonstration market areas differ from each other. They will be analyzed separately, and will also be analyzed using random effects models that prevent unmeasured differences among them from affecting estimation of the impact of the demonstration on outcomes.)

• Propensity Score Matching: Because instrumental variables approaches are not always successful in accounting for selection, we will also apply propensity score matching, another approach to simulating the conditions of random assignment. The propensity score approach is a method to match comparison subjects on variables that are known to affect Medicare utilization, time to nursing home entry, and home health outcome, so that the effect (or lack of effect) of the demonstration on these outcomes can be assessed. These variables include: personal and health characteristics including demographics (age, sex, race, marital status), admission source (community, hospital, SNF), functional status, other assessment and diagnostic attributes, and availability of home support.

However, home health outcomes have been shown to vary by type of agency as well as by personal and health characteristics at intake; and further, patterns of Medicare home health utilization show significant variation by region and state, not explained by beneficiary characteristics and agency type (Centers for Medicare and Medicaid Services 2005). This suggests that similar Medicare home health users may receive different amounts of service, and, even after accounting for utilization, may have different outcomes; and the same would be expected for demonstration participants and controls. This will be a challenge for the evaluation, and is discussed further below.

To determine the true impact of the demonstration, our analyses must identify and use beneficiaries served by other agencies, who are similar to (‘match’) beneficiaries who participate in the demonstration. To this end, we will employ a ‘propensity score’ approach, whereby we construct probit or logistic regression models on all home health beneficiaries from demonstration agencies to estimate their probability of participation (Rosenbaum and Rubin 1983). Parameter estimates calculated from these models will then be applied to home health beneficiaries from other areas of the state to determine the probability they would participate, if they were enrolled at the demonstration agency (the ‘propensity score’). The objective is to identify subjects residing far from the demonstration who have propensity scores that are approximately the same as beneficiaries participating in the demonstration. Matching can be done several ways as listed below, but all methods result in samples of demonstration beneficiaries and matched comparison subjects with similar propensity score distributions.

The propensity score estimation will include variables reflecting demographic characteristics, co-morbidities (e.g., based on CMS-Hierarchical Condition Categories (HCC) diagnostic indicators), functional status (e.g., from OASIS at admission), and prior year’s Medicare care utilization and costs. The matched sample will be used to estimate Medicare expenditures by type for demonstration participants had they not been in the demonstration, and, in like manner, the outcomes that demonstration beneficiaries might have experienced had the demonstration not taken place.

Geographic Location. Strong utilization differences across states suggest that comparison subjects should be drawn from the same state, from sub-areas that are similar to the location of the demonstration agency but are remote from it. Our first task will be to use the Area Resource File to select counties in the demonstration states that are similar in population density, proportion Medicare Advantage membership, and number of home health agencies per thousand Medicare beneficiaries to the counties served by the demonstration providers. Candidates for matching will be selected from home health users residing in these counties.

Treatment of Agency Characteristics. Home health agencies differ systematically in admission and utilization patterns, suggesting that comparison subjects might be selected in a nested fashion, by selecting patients of agencies that are similar to the demonstration agencies but are located in another part of the state. This will be explored, but would make the analysis substantially more complex, and it seems unlikely that agencies that are close replicas of the demonstration agencies will actually be identifiable within each state. Agency characteristics (annual Medicare visits, nonprofit or for profit status, visits per week for Medicare home health users, case mix as indicated by mean HHRG, average per visit costs, visit mix) cannot be included in the propensity score analysis nor as control variables in analysis of the propensity-matched samples. because the propensity score index will be fitted for participants and non-participants from the demonstration agency, so their agency characteristics do not vary.

The growing literature on use of OASIS to measure quality for Medicare home health recipients suggests that agencies are not uniform in their OASIS reporting. Restriction of potential comparison cases to beneficiaries with reported OASIS scores on admission and discharge could introduce bias. Selection of comparison agencies with high rates of OASIS reporting would skew the comparison group toward patients of those agencies, by definition.

We will consult with the Project Officer about the role of agency characteristics in selection of comparison cases. It must be recognized that matching home health users by agency type would require analysis of claims to develop information at the agency level, adding a substantial step to the analysis.

Propensity Score Matching Methods. Several technical matching approaches will be explored. In the "nearest neighbor" approach, the demonstration beneficiary is matched to the comparison subject with closest propensity score. Ties are determined based on random ordering of both samples before matching begins. When continuous, as well as categorical, variables are used in the model, ties are very unlikely. In the "caliper" approach, an allowable difference in propensity score is determined beforehand (e.g., within 0.05). For each demonstration subject, a matched comparison subject is randomly selected from all subjects with scores within the allowable range. Finally, in the "stratification" approach, demonstration subjects are put into groups based on propensity score cut-off values (quintiles, deciles, etc.). The same cut-off values are used to group comparison subjects. Each demonstration subject is then matched to a comparison subject, whose propensity score is within the same range. We will also carry out classic propensity score analysis, using only the demonstration agency’s home health patients but matching them based on propensity score

To adjust for possible patient-mix differences and demonstration selection effects, variables used in our propensity score matching models will include the following:

Beneficiary demographics – sex, age, race, marital status, informal support
County characteristics - urban/rural status based on Urban-Rural Continuum codes (known as Beale codes, see (U. S. Department of Agriculture 2006))
Functional status on admission (OASIS)
Co-morbidities and CMS-HCC conditions
Past utilization of home health services

Based on preliminary analyses and further discussions with the Project Officer, the variables employed for propensity score matching can be used in one of three ways:

To define which subjects are in the sample for constructing the regression model
As independent variables in the regression model - or
As variables requiring exact match before using scores to complete the match.

• Methods of Analyses for Participant/Comparison Beneficiaries. Using a pre-post, treatment-comparison experimental design, the demonstration participants and matched samples will be used to: (1) examine and compare utilization patterns and expenditures for Medicare-covered home health services and other Medicare-covered services. (2) examine and compare health outcome differences indicated in OASIS and in subsequent Medicare claims (e.g., rates of ambulatory care sensitive hospitalizations and nursing home entries, indicated by place of service). These comparisons will be done overall and possibly for subclasses of interest (e.g., by frailty category), if they are determined to have sufficient size to generate stable summary statistics.

When an outcome variable can be meaningfully measured for both the pre-demonstration and the post-demonstration period, two methods of analyses, unadjusted and regression-adjusted difference-in-differences analyses, will be used to estimate the effect of the Demonstration on utilization or expenditure outcomes of interest using the propensity-matched sample; and the instrumental variable analysis will also focus on differences due to participation in the Demonstration. Outcome areas where findings differ significantly will be further investigated to identify patient mix effects or other factors associated with such differences. The two methods of analyses provide somewhat different perspectives on the impact of the demonstration. The simple difference-in-differences estimates will assess the unadjusted effect of the demonstration on outcomes, not taking into consideration changes in the types of beneficiaries receiving care once the demonstration begins, and will be applied to the matched samples. The second method of analyses, employing multivariate regressions, will estimate the effect of the Demonstration on outcomes controlling for important covariates (e.g., beneficiary characteristics). Such an approach adjusts for changes in the types of beneficiaries who receive care during the demonstration. This will be applied to the demonstration agency populations using instrumental variable analysis and to the propensity-matched samples. These methods are described further below.

A simple ‘difference-in-differences’ t-statistic will provide an unadjusted estimate of the demonstration effect:

where Y is the outcome of interest, subscripts D and C distinguish demonstration and comparison subjects, and subscripts 0 and 1 distinguish between time 0 (the pre-demonstration period) and time 1 (the demonstration period). Thus the unadjusted demonstration effect represented above summarizes the difference in the changes from period 0 to period 1 in outcome (Y) for demonstration (D) and comparison group(C) beneficiaries. This measures the magnitude of change in outcome Y attributable to the demonstration.

Multivariate regressions will provide estimates of demonstration effects after adjustment for beneficiary characteristics such as demographics, diagnoses and co-morbidities, and prior (previous year) utilization for home health services and other medical care. In addition, the model may include geographic factors (defining market areas) that are associated with outcomes (e.g., urban/rural status). Because regional factors help to explain outcome variation, they may be useful even though demonstration and comparison beneficiaries come from the same areas . An example of such a multivariate regression model is:

where Y is the outcome of interest, D is the demonstration/comparison indicator, ‘time*D’ is the interaction between time and D, and X₁, X₂_,…,X_n are beneficiary or region level covariates included in the model as adjustors. The Greek letters , , , , and the ’s are coefficients to be estimated. The key coefficient is which estimates the effect of the demonstration on outcome Y. Other potential model formulations will be explored, including hierarchical linear models with beneficiaries clustered within agency and/or agencies clustered within market area. Whereas this simplified model conducts a single regression for all market areas, we may also explore separate regressions for each pair of demonstration home health users and comparison home health users.

The particular outcome, time to nursing facility entry, involves a ‘censored’ variable, meaning it will not be observed for a sizable proportion of the sample. Given the censored nature of this variable, we propose to study it using two survival analytic techniques: a log rank test, which is analogous to an unadjusted test of differences between demonstration and comparison subjects, and a Cox proportional hazard model, which is analogous to estimating the difference between demonstration and comparison subjects after adjusting for important covariates. As before these two methods of analyses, will provide somewhat different but complementary perspectives on this important outcome of interest.

Outcome Measures. To understand the full effects and consequences of the demonstration, several Medicare utilization and expenditure measures, indicators of quality, and health outcomes will be compared between participating beneficiaries and their comparison subjects. Table 4 provides a list of outcome measures and the methods used in their analyses. The source for the data for time of nursing home entry will be Medicare bills for physician visits in nursing homes and transportation of lab samples.

Table 4: Outcome Measures and Method of Analysis

Utilizations and expenditures
Total Medicare Home Health Services Hospital inpatient Outpatient facility Physician Services Skilled Nursing Ambulatory sensitive condition hospitalizations	Unadjusted difference-in-differences comparisons; Multivariate OLS regression models; Multivariate HLM regression models;
Indicators of Quality
Measures from OASIS	Unadjusted difference-in-differences comparisons, Multivariate logistic regression models;
Censored Time Variables
Time to Nursing Home Entry	Logrank test; Cox proportional hazard model

Measures of quality will be derived from OASIS based on recent studies; (Madigan and Fortinsky 2004; Schlenker, Powell et al. 2004; Schlenker, Powell et al. 2005).

Independent Variables. Beneficiary and county level characteristics listed in Table 5 will be used as covariates in the multivariate regression models.

Market area analysis. Difference in difference analyses of market area measures of Medicare home health use, pre/post and demonstration versus comparison region, will be performed to determine whether the demonstration has affected utilization per thousand beneficiaries or amount of use (episodes per beneficiary, visits per episode). A full analysis of this issue requires a definition of home health market area and inclusion of other factors that affect utilization rates that may also change over time (population age distribution, hospital discharge rate, utilization of other post-acute care modalities, rate of nursing home residence). For purposes of this design, we will compare home health utilization rates for the counties or Consolidated Metropolitan Statistical Areas (CMSAs) of demonstration sites to rates for other sub-areas in the demonstration states pre- and post-demonstration. This will be placed against the backdrop of changes in home health services utilization gleaned from aggregate CMS state and national statistics. The observable time span will be quite short, from 2005, the year prior to demonstration implementation, to the latest possible year for observation of Medicare claims under the evaluation contract, likely to be 2007. This does not provide much time for market trends to develop in response to the Medicare expansion.

Table 5: Independent Variables

Category

Characteristic

Beneficiary level

Age, gender, race, marital status, informal support

Income proxy (e.g., average per capita income for the 65 and over age category within 5-digit zip code)

Co-morbidities/HCC conditions

Functional Status

Utilization and expenditure during prior year:

Hospitalizations

Skilled Nursing Facility Stays

Home health utilization

Emergency department care / urgent care

Distance to home health agency¹

County level

Number of physicians / PCP physicians per 100K

population

Number of hospitalizations per 1K population

Average Medicare expenditure per beneficiary

Rate of use of home health services

¹ distances calculated on basis of latitude and longitude translations of mailing addresses of home health agency and beneficiaries, as reported in CMS database.

Provider impacts. Effects on demonstration providers’ financial performance, specifically their cost to treat Medicare home health beneficiaries prior to the demonstration versus their costs during the demonstration, will be studied. The financial information for this study will come from Medicare cost reports requested from the providers themselves for two pre-demonstration years and as many demonstration years as possible under the time frame of the evaluation (2006 and 2007).

Claims Analysis Timeline. While no start date is established yet, the MADS demonstration is not likely to begin before the summer of 2006. The evaluation project’s first deliverable of Phase 2 is a Final Interim Report due in March 2008 (Month 30 after the start of the contract). Since a draft of this report is due in January 2008, and given a 6-month time lag for claims to be processed and appear in CMS SAF files, the only claims that will be available for analysis for this report will be the enrollees who begin and complete episodes before the late spring of 2007. Given this, it is best to focus this claims analysis on its descriptive component and provide an early, but by no means settled, view of the enrollment and adult day care utilization at demonstration agencies. In the Interim Report then we will measure initial participation rates, and investigate selection effects by identifying the distinguishing characteristics of beneficiaries in our three groups: those who participate in the demonstration, those who are offered participation but decline, and those who are not offered participation. Delay in the startup of the demonstration beyond the late spring of 2006 will of course decrease the time period for enrollment and limit the claims available for interim report analyses even further. The evaluation’s Final Report is due in April 2009 (month 43). This second report should be able to study claims through approximately the first 24 months of the demonstration (through the summer of 2008). Claims analyses in the Final Report will focus on its second component, testing hypotheses concerning the impacts of the demonstration and estimating the size of effects with respect to utilization and expenditures, quality, and some health outcomes. For the comparative analyses we will investigate beneficiaries who first participate in the demonstration in the first 12 months of the demonstration and examine their Medicare utilizations, expenditures, and outcomes during a 6-month follow-up period. As a general rule, Home Health claims take 6 months to be considered 95% complete. In consideration of this 6-month lag and required delivery dates, our project has the following claims analysis timeline. These dates are summarized in Table 6.

Table 6: Claims Analysis Timeline

Time Period	Task	Data
1 Aug 2007	Acquire IDs of home health beneficiaries at demonstration agencies and resulting participation status (participate, offered but declined, not offered)	Enrollment records of home health beneficiaries at demonstration agencies, June 2006 – March 2008
Aug 2007	Acquire Medicare information, OASIS, and claims of home health beneficiaries at demonstration agencies available by this time (including pre-demo and during demo periods).	100% NCH Part B, 100% SAF inpatient, outpatient, SNF, home health, and OASIS files, Oct 2005 – March 2008
Oct – Nov 2007	Analyze Medicare enrollment information and claims for descriptive component in Interim Report
1 Jan 2008	Submit draft of Interim Report to CMS
March 2008	Submit Final Interim Report to CMS
Jan. 2008	Acquire home health services claims from SAF file; use to help select comparison agencies	SAF Home Health files, 2006-2008
Feb 2008	Acquire IDs of adult day care participants at demonstration agencies enrolled during first year of demonstration (Oct ’06 - Sept 07) and home health beneficiaries at comparison agencies served during the same time period	Enrollment records of adult day care participants at demonstration agencies and home health beneficiaries at comparison agencies, Oct 2006 - March 2008
Mar.2008	Acquire Medicare claims for beneficiaries at demonstration and comparison agencies for year before demonstration participation (or pseudo-enrollment as comparison subject)	100% NCH Part B, 100% SAF inpatient, outpatient, SNF, and home health files, Oct. 2006 – March 2008
July – Aug 2008	Create propensity score matching models; construct matched samples of beneficiaries in demonstration and comparison regions.
1 Sept 2008	Acquire Medicare claims for beneficiaries at demonstration and comparison agencies for 6 months period following enrollment in demonstration (or pseudo-enrollment for comparison subjects)	100% NCH Part B, 100% SAF inpatient, outpatient, SNF, and home health files, Oct 2006 – Mar 2008
Oct. – Dec 2008	Perform claims analyses
Feb 2009	Submit draft of project’s Final Report
April 2009	Submit Final Report

References

Centers for Medicare and Medicaid Services (2005). Medicare Home Health Utilization by State Calendar Year 2000.

Horgan, C. M., D. W. Garnick, et al. (2004). Substance Abuse and Mental Health Services in Health Plans: Results From a Nationwide Study

Summary Report to Respondents, Center for Behavioral Health, Schneider Institute for Health Policy, Heller School for Social Policy and Management, Brandeis University

National Institute on Alcohol Abuse and Alcoholism (Grant #2 R01 AA10869-04A1)

National Institute on Drug Abuse (Grant #1 R01 DA10915-05).

Leutz, W. and J. Capitman (forthcoming). "Social HMO Performance: Needs, Meeting Needs, and Satisfaction." Journal of Aging and Social Policy.

Leutz, W., J. Capitman, et al. (2001). "A limited entitlement for community care: how members use services." Journal of Aging and Social Policy 12(3): 43-64.

Madigan, E. A. and R. H. Fortinsky (2004). "Interrater reliability of the outcomes and assessment information set: results from the field." Gerontologist 44(5): 689-92.

Marshall, B., M. Long, et al. (1999). "Case management of the elderly in a HMO: the implications for program administration under managed care." Journal of Healthcare Management 44(6): 477-491.

McClellan, M., B. J. McNeil, et al. (1994). "Does more intensive treatment of acute myocardial infarction in the elderly reduce mortality? Analysis using instrumental variables [see comments]." Jama 272(11): 859-66.

Rosenbaum, P. and D. B. Rubin (1983). " The Central Role of the Propensity Score in Observational Studies for Causal Effects." Biometrika 70: 41-55.

Schlenker, R. E., M. C. Powell, et al. (2004). "Home health outcome patterns." Home Health Care Serv Q 23(3): 69-85.

Schlenker, R. E., M. C. Powell, et al. (2005). "Initial home health outcomes under prospective payment." Health Serv Res 40(1): 177-93.

U. S. Department of Agriculture. (2006). "Measuring Rurality: Rural-Urban Continuum Codes." Retrieved 4/6/2006, 2006, from http://www.ers.usda.gov/briefing/rurality/RuralUrbCon/.

File Type	application/msword
File Title	Supporting Statement
Author	stason
Last Modified By	CMS
File Modified	2006-08-01
File Created	2006-08-01

Form CMS-10204 - Instruments (Appendices B,C,D,E)

Evaluation of the Medical Adult Day-Care Services Demonstration - Phase I

CMS-10204.Draft Supporting Statement.8-1-06

Evaluation of the Medical Adult Day-Care Services Demonstration - Phase I

OMB: 0938-1018

Background

A. Justification

1. Legal and Administrative Justification

2. Use of Information

3. Collection of Information

4. Duplication of Similar Information

5. Impacts on Small Businesses

6. Less Frequent Data Collection

7. Special Circumstances

8. Federal Register Notice/Outside Consultation

9. Payments/Gifts to Respondents

10. Confidentiality

11. Sensitive Questions

12. Burden Estimate (Hours & Wages)

13. Capital Costs

14. Annualized Costs to the Federal Government

15. Program Changes

16. Publication and Tabulation Dates

17. Expiration Date for OMB Approval of Information Collection

18. Exceptions to “Certification for Paperwork Reduction Act Submissions” Statement

B. Collections of Information Employing Statistical Methods

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

4. Tests of Procedures or Methods to be Undertaken

5. Statistical Contact

Appendices

Appendix A: Legal Basis

Appendix B: Evaluation Research Questions Specified by RTOP No. CMS-05-031/ERD

Appendix C: Site Visits (Task 3b)

Demonstration Participant

Prior MADS Use

Table 2: Key Informant Categories Per Site Visit

Appendix D: Satisfaction Survey (Task 3d)

Appendix E: Beneficiary Interview Guides

Appendix F: Data Use Agreement for Use of CMS Beneficiary Data

Appendix H: Plan Approved by CMS Privacy Board for Beneficiary Privacy Protections

Appendix I: Introductory Letter

Appendix J: Verbal Informed Consent Scripts

Appendix K: Brandeis University IRB Approval Notification

Appendix L: Analysis of Data

Table 4: Outcome Measures and Method of Analysis

References