AHRQ Memo

1. We better understand now, given the passback responses, that this project aims to observe the implementation process of LEAN tools in “real world” settings. Part of this entails minimizing the number of research constraints imposed on the sites, for example, by not specifying certain outcomes in advance that the sites should be exploring. Rather, the idea is to let the sites determine their own evaluation plan and figure out how they will assess the success or failure of their LEAN implementation. If that is the case, then we would like to see these evaluation plans in advance of implementation for the prospective study sites. At minimum, these plans should describe the problem they are trying to solve, the LEAN tool they have chosen as a result, and how they plan to assess “success” or “failure.” It is important to get this information at the outset, even before the intervention has begun. If AHRQ has those now, we would like to see them. If not, we would like AHRQ to submit change requests as they become available and prior to implementation. These evaluation plans should include information that would normally be the subject for OMB review (e.g. sampling plans, non-response bias, etc.). In other words, if each site had to complete a supporting statement Part A and B for their individual sites, what would they say?

Response:

As indicated in OMB’s question, the purpose of this project is to observe Lean/TPS implementation in real world settings and to conduct naturalistic observation of the sites. As part of our observation, we will have the opportunity to see whether and how the sites define and measure the success (or not) of their Lean/TPS efforts. We will gather information about what they do, will ask for any existing evaluation plans during our first site visit, and will share information about the sites’ evaluation plans and activities as part of our findings. In other words, the purpose of our project is not to shape evaluation efforts at the sites -- only to study them and report on them.

The responses indicate that questions will be asked during the first site visit. However, it is not clear when the first site visit will take place. Some of the instruments suggest that the first site visit will not happen until 3 months into the prospective studies. Also, some questions that ascertain baseline information seem to be planned for the 2^nd site visit (see for example question 2.0 in the in-person interview guide). Given that these are meant to be prospective studies, we feel strongly that all baseline information should be ascertained prior to or right at the start of the intervention, not 3 months into the intervention and certainly not at the 2^nd site visit.

Response:

We agree. We plan to conduct all site visits for the prospective studies before the implementation of Lean/TPS. All data about sites’ implementation plans will be gathered during the first prospective site visit.

2. It is still not clear from the instruments which questions are planned for retro and prospective arms of the study, and it is also not clear which questions are for baseline and which are for follow-up. A simple change of verb tense does not, for example, address what happens to questions that really only need to be asked once. Please separate the instruments in prospective and retrospective instruments, and further subdivide the prospective instruments into baseline and follow-up instruments.

Response:

We have created and supplied three separate topic guides: (1) a guide for the retrospective site visits; (2) a guide for the first prospective site visit; and (3) a guide for the second prospective site visit.

3. We understand that 2 other sites are still to be picked. We would like AHRQ to submit change requests when they have been selected. Those change requests should include the information in #1 of this passback.

Response:

Contingent on the successful conclusion of negotiations that are currently progressing well, the two sites will be as follows:

Park Nicollet Clinic

Park Nicollet Health Services is an integrated care system that includes Methodist Hospital and Park Nicollet Clinics, which can serve as potential case study sites. Park Nicollet is based in St. Louis Park, MN. Methodist Hospital is a 462-bed facility with over 1,000 salaried physicians. Park Nicollet Clinic includes 24 clinics in the Minneapolis area, with both salaried and community practicing physicians.

Justification: Park Nicollet adds to our diversity of settings, because the organization is implementing Lean in its ambulatory care settings. In addition, Park Nicollet has been implementing Lean since 2003, and provides an opportunity to examine conditions affecting the sustainability of Lean.

Garfield County Memorial Hospital

Garfield County Memorial hospital is a 25- bed, critical access hospital (the first of this type designated in the state) located in Pomeroy, Washington. The Level 5 trauma care facility offers an emergency room (open 24/7), acute care, physical therapy, trauma certified staff, full service lab, full service X-ray, and cardiac monitoring. Garfield County Memorial hospital is part of the Garfield County Hospital District. 

Justification: Garfield County adds diversity to size, settings, and geography of our cases. Garfield County is a very small hospital, is a critical access hospital, and is located on the West Coast. The Garfield case provides an opportunity to examine Lean implementation in a setting with limited on-site expertise in quality improvement. Examining Lean in a setting like Garfield may shed light on challenges facing many small, rural hospitals seeking to implement Lean.

4. Finally, when responses are not assessed or validated against “objective data” (e.g. claims about perceived patient satisfaction that are not validated against a press-ganey-like survey, or claims about returns on investment that aren’t validated against administrative/cost data, etc.), we feel strongly that it should be made clear (e.g. in all publications of these results) that these assessments are subjective in nature.

Response:

We agree. In addition to citing other limitations, in all publications of these results we will explicitly note when reported data is subjective in nature.