| Site Index |
|
|
         |
|
••Fall 1998 / Vol 2, No 4 |
|
|
|
|
|
|
|
Clinical Contributions Operating Room Benchmarking: The Kaiser Permanente Experience
| to
pdf >> In 1994, a Kaiser Permanente (KP) Interdivisional Task Force--comprising 30 surgeons, anesthesiologists, perioperative managers, and technical staff--completed a comprehensive, internal Operating Room (OR) Benchmarking Study. The study used 11 metrics in three categories--OR Productivity, OR Costs, and Satisfaction--and set operational targets, or benchmarks, for many of the metrics. The resulting Operating Room Best Practices (ORBP) report describes many business and clinical Best Practices that may be expected to produce substantial performance improvements. The 1994 study estimated potential organizational savings of $72.6 million. In a 1996 follow-up study, the KP-California Division demonstrated actual savings of $10.1 million for three of the metrics combined. By showing the clinical and economic benefits of collaboration among surgeons, anesthesia personnel, and other OR staff, our study has also led ORs throughout KP to use a multidisciplinary problem-solving approach instead of giving perioperative managers sole responsibility for improving OR efficiency. Introduction Specifically, other health care organizations were offering lower rates, promised better access to ambulatory services, and appeared to design more flexibility into their health plans. Our leading edge was especially at risk in the three KP Divisions that maintained hospitals in addition to medical offices--California, the Northwest, and Hawaii. To address these business challenges, the Operating Room (OR) Benchmarking Study was conducted. Benchmarks are operational targets that are set for each of several cost and productivity metrics and that are determined from a combination of information in the literature, results of data analysis, professional experience, and operational expertise. The OR was chosen as the focus of the study because of the high level of expenditures in this complex environment and because of its interdependence with other departments and hospital systems. The goal of the study was fourfold: to learn what makes a good operating room; to identify Best (Business) Practices; to identify potential cost savings; and to provide a model for KP interdivisional cooperation. Methods The scope of the study did not include minor and special procedure rooms outside the OR "boundaries" or ORs within the labor and delivery units; staffing practices in areas or departments other than described above (ie, scheduling office, housekeeping); reusable items (eg, instruments, capital equipment); staff who process surgical instruments, engage in general handling of materials, or who do not work in the OR. Even though the scope of the study did not cover these departments or functional areas, relevant issues were discussed and anecdotally documented in interviews and site visits of candidate high-performing facilities. Participant Selection
Study Metrics To measure cost, the team used three metrics: cost of OR labor, cost of anesthesia labor, and cost of materials. To measure satisfaction, the team used four metrics: patient satisfaction, OR staff satisfaction, physician satisfaction, and CRNA satisfaction. Formation of Benchmarking Teams Equalizing Comparability of Facilities Differences in labor costs (eg, geographic, union-related) were also considered when comparing operational costs. For example, the staffing analysis used full-time equivalents (FTEs) instead of payroll dollars, and the materials cost analysis used a standardized unit cost per procedure. In addition, main ORs and ASUs were compared and benchmarked separately because as different types of facilities (ie, inpatient versus outpatient), they differed in type and case-mix of patients, scheduling practices, facility layout, and hours of operations. To ensure that information was sufficiently uniform for comparison among all facilities, existing data bases were used for workload statistics (although some data had to be reentered in a standardized format). The Benchmarking Process Phase 1: Data Collection To collect information about these operational processes, detailed survey questionnaires were designed. The survey data as well as information obtained from standard payroll and nonpayroll reports enabled us to identify candidate high-performing facilities. To ensure that these survey data were reliable despite the length and complexity of the survey tool, the technical staff held individual meetings with each major stakeholder in the OR (eg, the perioperative manager and chief or manager of anesthesiology). To evaluate patient satisfaction, a questionnaire was given to each surgical patient during a specific timeframe before discharge. (The patient satisfaction survey was also designed to help us identify major predictors of surgical patients' overall satisfaction.) Patients were requested to complete the form and return it by mail in the stamped envelope provided. To evaluate satisfaction of the OR personnel, we mailed surveys to the anesthesiologists, surgeons, and nurse anesthetists at their home addresses; OR staff received their surveys at the workplace. Phase 2: Identifying Best Practices. Two-day site visits to 11 candidate high-performing facilities were conducted to validate the information received, to determine the practices that made the site a high performer, and to identify the Best Practices. Best Practices included processes that promote high performance as well as standards that are realistically achievable by other facilities. During our survey of these high-performing facilities, we reviewed clinical outcomes to ensure that the Best Practices identified were not likely to have adverse impact on patients. (Best Practices were explored further if any resulting outcome fell outside the acceptable range.) This approach was taken so that clinical outcomes would be compared among all facilities and measured against industry standards instead of being statistically correlated with Best Practices. During the site visits, clinical staff interviewed the managers and
supervisors responsible for clinical out At facilities with a low return rate for patient satisfaction surveys, patient satisfaction was restudied to increase the statistical significance of the sample. Phase 3: Implementation To communicate with all concerned parties simultaneously while addressing the concerns of first-line managers (who might need the information to explain results to their managers), we used meetings, presentations, and publications as part of our communication strategy. Specifically, information constantly flowed throughout the project to and from all perioperative managers, chiefs, and managers of the anesthesiology departments via our meetings with them during the initial survey. This information flow was achieved by soliciting input from these personnel, by conducting group meetings to discuss preliminary results, and by keeping personnel informed of results by mail. In addition, meetings with the Interdivisional OR Task Force were held about every six months to obtain input and feedback regarding progress. After the results of the study were prepared, the Steering Committee was contacted to inform them of the findings, implications, and potential issues. The study report was distributed to all parties concerned: perioperative managers, directors of nursing, administrators, chiefs of surgical services, and chiefs of anesthesiology. Results were also presented to the Medical Group administrators and hospital administrative personnel in each Division as well as at group presentations for the chiefs of surgical specialties as requested. We also organized a teleconference with all 42 sites and invited key stakeholders, administrators, and senior corporate managers to participate. The program included an interactive question-and-answer session. Phase 4: Recalibration Estimating Savings Results To calculate the benchmark for OR Labor Cost, the cost for all ORs was compared with an "optimal" staffing model developed for different-sized ORs. Similarly, to calculate the benchmark for Anesthesia Labor Cost, the cost for all ORs was compared with an "optimal" staffing model developed for different-sized ORs. The benchmark for OR Material Cost was set to the 10th-lowest cost of materials observed among the 42 facilities studied. Also included in this target was the case-mix acuity at each facility. Best Practices Identified by the Study Eight Best Practices contributed most toward positive change: OR productivity, labor, and materials; anesthesia labor; and satisfaction of patients, OR staff, physicians, and CRNAs. (For a complete list of these Best Practices, please contact the authors.) Best Practices in OR productivity (OR Utilization and OR Performance) included use of all-day blocks of time, on-time start for first case of the day, physician serving as OR director, effective OR Committee, streamlined preoperative processes, overlapping turnaround processes, accurate procedure cards, scheduling guidelines, and routine block reallocation. Best Practices for OR labor included use of two staff members per OR; RN-to-ORT ratio of 65:35 (now set at 60:40 throughout KP); use of 2.4 to 2.5 staff members per OR (in electively scheduled blocks); use of short shifts; staffing to demand; and use of part-time staff, per diem staff, or both. Best Practices for OR materials included standardization of materials with compliance monitoring; move from surgeon preference cards to procedure cards; presence of an OR materials coordinator and an OR Cost Awareness Committee; use of "reusable" instead of "disposable" materials; and designation of supplies as "available" instead of "open." Best Practices for anesthesia labor included assignment of anesthesiologists to OR as primary care provider for high-risk patients; Anesthesia Care Plan formulated by anesthesiologist with assistance of CRNA; CRNA participation in preoperative evaluation of all but high-risk patients and patients who require complex care; and appropriate CRNA education and credentials in regional anesthesia if opportunities are provided to practice in this area. Anesthesia care received and confidence of patients with OR staff were predictors of high overall patient satisfaction. Quality of staff, quality of instrumentation, and level of responsibility in job predicted high overall OR staff satisfaction. Performance of OR nursing staff, physical characteristics of OR suite, and availability of equipment and supplies in the OR predicted high overall physician satisfaction. Quality of OR staff service, relationships and communication with other hospital staff, and quality of service provided by surgeons predicted high overall CRNA satisfaction. Potential and Actual Organizational Savings Identified by the Study In the 1996 follow-up study of three of the metrics--OR Labor, OR Utilization, and OR Performance--at California facilities, realistic potential savings were calculated as $18.7 million (ie, a figure equivalent to two-thirds of total theoretical potential savings, as explained in Discussion). Of this $18.7 million in potential savings, the 1996 follow-up study showed that an actual savings of $10.1 million was realized (Figure 3). The major areas of savings were OR Utilization (which increased from 81% to 86%), OR Performance (which improved in main and ASU ORs), and RN-to-ORT ratio (which changed from 71% RNs to 67% NRs). Total savings identified by these three metrics were equivalent to 54% of the savings originally projected and were believed to constitute a major accomplishment, considering that no formal implementation project efforts were conducted after publication of the original study results. Discussion Partly as a consequence of this extensive contact, the project took nearly three years to complete. This duration was longer than originally intended, but it is vindicated by the final study report's attention to detail, credibility, reliability, expansiveness, reality of operational understanding, the acceptance it has garnered from most key stakeholders, and its vision for future development of benchmarks. Challenges Learnings Another positive learning involved the high level of knowledge and clinical expertise of the Core Team. Indeed, the importance of high-level clinical and technical expertise of the Core Team personnel was invaluable to the success of this project. During the site visits, the Core Team was able to identify many Best Practices because of its knowledge in the relevant areas and because of the opportunity afforded the Core Team to compare with other facilities visited. Staff at the facilities were often not aware that their practice was unique or different from that of other ORs. The Core Team's level of expertise was important also for the acceptance it brought the report and for the "buy-in" for implementation obtained from most key stakeholders. A third--and major--positive learning resulted from issuing a practical report, the Operating Room Best Practices (ORBP) report. This report is a practical, "user-friendly" document, not merely theoretical or conceptual: To implement changes in practice, a manager needs only to identify the area for potential improvement, look up the Best Practices associated with the area, and develop an action plan. The Best Practices are concrete suggestions found in high-performing (Best Performer) facilities and are transferable to ORs and ASUs at most KP locations. Other Learnings Need for Allocating Resources The project was expensive with regard to the people, time, and travel required. A budget was not prepared at the beginning of the project, and we do not know the actual cost to the organization to produce this study. For future projects of this magnitude, we would recommend that a formal budget be prepared so that cost-benefit decisions can be made. Need for Narrowed Scope Considerations in Reporting Potential Savings The decision to publish the total overall potential savings was made by the Task Force and was supported by the Steering Committee; however, publication of this dollar figure caused those who only read or heard the "bottom line" of the study to place unrealistic expectations on the managers who would be required to implement the cost reductions. Therefore, for purposes of the 1996 follow-up study, we reduced by a third the potential savings figure reported for the three metrics. Need for Promoting Acceptance of Results Need to Integrate Data Organizational Effects of the Study This study has changed the culture of our organization in how it looks at the OR. Before the advent of this project, most administrators believed that saving money and improving efficiency in the OR was the responsibility of individual perioperative managers. By emphasizing clinical and economic benefits of collaboration among surgeons, anesthesia personnel, and the OR staff, and by using the Task Force and Core Team as role models, the report has helped to make the multidisciplinary approach to problemsolving in the OR the accepted norm. In studying some "sacred cows" in OR practice, we have shown that some of these can be justifiably eliminated without eventuating any adverse clinical outcome. Each day, talented and innovative perioperative managers and physicians are discovering new ways to improve the cost-effectiveness and productivity in their ORs. We should continue our efforts in external benchmarking to add another dimension--especially outside the KP organization--to increasing the efficiency and cost-effectiveness of our ORs. Bibliography Albert Mariani, MD, Hawaii Region
The Operating Room Benchmarking Study was the largest effort to date by the KP Medical Care Program to understand that incredibly complex and very expensive health care environment known as the OR. The Task Force consisted of analysts, OR directors, hospital administrators, anesthesiologists, CRNAs, and surgeons. The Task Force members were experienced, knowledgeable, and dedicated to what turned out to be several multiday meetings over three years. In the Hawaii Region, the OR metrics proved to be the most useful and have become the benchmarks whereby we measure our OR efficiency. I have found several "thumbnail" metrics useful in managing the surgery department and its approach to the OR. For example, a well-managed OR has 85% utilization: If utilization is less than that, expensive OR time is being wasted; if more than that, the OR schedule is getting too crowded--which results in unacceptably long waits for surgery as well as the need for overtime and extended working hours. In addition, a minute of OR time costs about $13. This concept is useful when considering the cost-effectiveness of timesaving equipment or supplies. Moreover, an hour of OR time costs roughly the same as a day of hospitalization. Careful definitions of OR Turnaround Times (Room Turnaround Time, Surgeon Turnaround Time) can be useful for quantifying what really is going on regarding turnaround. Unfortunately, we were unsuccessful in our efforts to collect this information in a public way to challenge discrepancies and thus to validate the data. The data have been collected but taking action on the data would be difficult without independent validation, which would require a focused study. Finally, marked cost efficiencies can be achieved by assigning cases to the appropriate setting. The OR costs about twice as much as an ambulatory surgery unit (ASU), with short-stay cases being of intermediate cost. These OR metrics give us an instrument that measures performance by commonsense, widely accepted definitions that allow longitudinal comparisons over time. We have adopted the metrics of the Task Force to measure the efficiency of our OR. There are some caveats. We should remember that this is a Best (Business) Practices study, which means that although quality of service was acknowledged to be an important determinant of organizational success, this study emphasized cost. Some evidence showed a trend toward a mild negative correlation between cost and satisfaction. Despite Herculean efforts to standardize data, this task was impossible without on-site inspection. However, this standardization of on-site data would be prohibitively expensive if done for ongoing evaluation of OR services in every service market. Collecting comparable data in the same market over time would be more feasible because the same market uses the same information services infrastructure over time. Thus, not all intuitively good ideas could be quantified. Some were accepted in our Region simply because they made sense on the basis of extensive OR management experience. Uncovering new ideas was the most beneficial effect of the attempt to standardize the data. While I am obviously biased and while there are few endeavors that depend upon teamwork as much as in the OR, there is a hierarchy of required satisfaction determined by personal risk and responsibility. That hierarchy looks something like this: patients>>surgeons>anesthesiologists>CRNA>staff. This concept was not addressed in the report. At each meeting, never more than 3 of approximately 30 participants were surgeons, and only 1 of these surgeons attended all sessions. Some of the report's recommendations must be read in this light. The report notes that the Permanente Medical Groups are managed by influence rather than authority. In my experience, influence based on correct data is far more powerful than authority for achieving desired results, and this report represents a substantial and meaningful effort to provide such data. Max Wirjo, MD, Southern California Region As we all remember, the ORBP Task Force initially created controversies and criticism--if not skepticism--from different directions when it was created. Even we had to settle our differences in the final days of finishing that document. Furthermore, looking at the criticisms carefully, I notice that they entirely differ from each other depending on the specialty and department that produced the criticism. On the basis of this experience alone, I concluded that the ORBP report is probably the best document or set of recommendations that we have ever created, and that we should be proud of that. Even now, three years later, we can still look at the ORBP report and check whether our practices match its recommendations. The ORBP report is certainly specific to our practice and culture, but I think the report would apply and be useful to any health care organization that has a setup similar to ours. My only disappointment has been that the ORBP document has been underutilized--that not enough commitment was given to implementing it. This could well be due either to the criticism heard at the time or to the lack of champions for the report's recommendation, but I believe that time will show the ORBP report being used as a reference for OR Best Practices. Thanks for asking me to comment and reflect on this important document.
Gene Golfus, MD, Northern California Region Before I begin "picking" on this report, I would like to emphasize that I think Kay Stodd deserves a medal for it and for her continued work on ORs for this organization. I do not wish to be misunderstood; I am not criticizing Kay. To the contrary: this report is a monumental achievement, and it offers central points that allowed me to begin to think about ORs. That being said, I paraphrase a famous author: It was the best of reports, it was the worst of reports. Actually, the report was good; the way it has been interpreted and implemented is, well, less good. I think the ORBP study was a fantastic attempt to put together an organized look at a very complex area: the OR. Much is useful in the 1994 ORBP report as a stepping-stone to continued thinking about ORs, but many of the conclusions and uses of the report are misguided. Moreover, much of the good has been lost and forgotten, and much of the error, revered. Many of the caveats were good, but no one remembers them. The value in this report is largely as a beginning, and a stimulus for further progress in the OR. The ORBP report--and the facility profile that came from its work--had an impact on me and on the OR I work in. In 1991, the Vallejo OR was at 45% utilization, whereas today it is at 93% utilization. Much of that increase came from this report, which encouraged addition of an OR director and stimulated the OR manager, Sharon Fine, of KFH and myself to work as a collaborative team to improve OR utilization. The sections on professional satisfaction of surgeons, OR nursing, and anesthesia are also very good beginnings. The report was correct in noting that an OR that is already achieving utilization beyond 85% may have difficulty adding urgent add--on cases. We are experiencing that situation now; the report was correct--we have overshot the mark. However, there are things in the report that, in retrospect, I do not think hold up as well. For example, the ever-rising bar of the "25th percentile" Best Practice time was accepted by the senior leadership but is less than clear as an attainable goal--and may not be a desirable one. The OR timeline is split into time intervals, and mean operative time for each interval is calculated for each OR. The time that is the fastest at the level of the 25th percentile (ie, the upper fourth in time) is considered the "Best Practice." The assumption then is that if each OR across the Region were to come to the best 25th percentile in each time slot through a case, then the Region could save $20 million. The amount of time your OR took "to set up the case" or "from cut to close" beyond the Best Practice 25th percentile would be seen as potential savings by multiplying "excess" minutes times dollars per minute. The problem presented by this measure is seen when you look at a cataract operation in two different facilities, one that took 77 minutes total time and another that took 80 minutes--that is, it took the same amount of time by the level of accuracy we can achieve using this measure. If these two procedures were to take "the same" total time, then calculating the savings of some minutes times dollars for each of their parts is not going to save any money. This error is compounded by each time period in each OR and thus creates very large error in evaluating potential savings. Even if an operation can be done faster, being done 6.2 minutes faster at the end of a day does not save money. To save money, ORs must be closed and people not paid. So if you can learn to work faster and do more cataract operations per day and thus complete your caseload of cataract operations in fewer OR days, you can then close ORs and send people home instead of paying them--then you can save money. If you get more efficient and do more operations and then do more operations on those extra OR days, well, that costs--not saves--money, because it costs money to do more surgery. By adopting this procedure, you have also not begun to answer--or even address--questions about either cost-effective surgery or quality-vs-cost balances. The total mean times for completing a specific procedure would need to be compared, and if the result exceeds some statistical limit, you would need to go back and examine the steps. This process had an impact on my CSA in that $3 million was removed from the OR's budget as allocated for 1996 to determine the 1997 budget. This budgetary reduction was largely based on the report's numbers and led to an unrealistic, unobtainable budget. The ORBP report has also stimulated us to go back and realize that timelines--and the definitions we base those timelines on--are not the same across the Region. This discrepancy could affect the way time is recorded in each box, and it probably means we are not comparing like items. The Executive Steering Committee of the RPMG compiled in April 1998 and is now implementing a new set of consolidated definitions and a timeline to be used by all ORs in our Region. These definitions will match national definitions agreed upon by AORN (American Operating Room Nurses) and AACD (American Association of Anesthesia Clinical Directors). I will conclude with the point that the report is good but should not be taken as "gospel." The report is a wonderful stimulus for further learning. We should not throw it away, but we should question and improve on its most valuable beginnings.
Tom Janisse, MD, Northwest Region Initial Comments "Unfortunately, if we implement many of the recommendations in the staffing paper, the public perception will be focused on the premise that 'cheap and second-rate' is our motto instead of 'quality and safety.'" "We can't allow administrators with no idea what our OR/medical center is like to dictate staffing based on theory." "I am distinctly unimpressed with the input from people actually doing clinical work, from our own department's experience." "It [the report] picks out things from various practices and puts them together in a fictional Best Practice." "It gives only lip service to quality and service." "It is my concern that an administrator will look at this document and firmly establish initial guidelines without looking at the big picture, because the big picture isn't totally presented." Operations Group An "Open OR" The concept of an "open OR"--either a halfday block or a whole-day block--proves valuable for creating a "just-in-time" method of adjusting for another area of complexity, ie, the variable nature of surgical procedures, patients, equipment, and OR capacity. The OR Benchmarking Study Task Force tried to construct a model--or a dynamic equation--for evaluating and predicting this variability so that the schedule could be built and resources made available to routinely improve OR effectiveness. To construct this model, the Task Force researched four factors:
Creating this formula was at first difficult to conceptualize and then became a daunting task to apply in practice, given the time and resources required as well as other priorities. In the end, if you had an "open OR" you could dynamically shift cases or reconfigure case lists that day to adjust for a critically ill patient, a prolonged surgical procedure, or unexpected delays caused by equipment malfunction. In practice, this procedure seemed to work well by supplementing the experience of the OR Team in anticipating these complexities and scheduling appropriately. If OR utilization was in the 60% - 70% range, then an open OR in effect existed. When OR utilization increased to an optimum 85% of capacity, then the open OR had to be incorporated into the schedule ahead of time. It took a great deal of persistence to achieve this, considering the apparent importance of cases vying for use of OR space. Staffing Model The ORBP document contains language that I drafted specifically for implementation of the staffing model. We felt this language was imperative for the model to operate well and for it to maintain high levels of quality and service. "It is obvious that we cannot reduce costs only by removing people. We can only drive down Anesthesia Labor costs by driving up anesthesiologist and CRNA productivity. This increased productivity is possible only in a high-performance system that requires high levels of competence and performance from anesthesiologist, CRNA, and technician staff." Progress toward superior, cost-effective care may require a substantial ongoing education and training process for all anesthesia staff." In large part, this reasoning accounts for why the Task Force recommended that Best Practices be implemented over a broad time line of 18 to 24 months and that each site look at its own OR practice in the larger context of its anesthesia department practice. Procedure Time and Turnover Time Conclusion Acknowledgment: The Medical Editing Department, Kaiser Foundation Research Institute, provided editorial assistance.
|
|
|
|
|||||||||||||||||||||||
|
|
|
|
|
|
|
 |
|
Dr.
Mariani did his training in Urology at the Mayo Clinic, after which
he joined the Hawaii Permanente Medical Group in 1980. He was appointed
Chief of Surgical Services in 1983 and has served in that capacity
since. He has served on the Interregional New Technologies Committee
since 1991 and authored the Hawaii Specialty Care Model which has
been featured at national meetings.
Medical
Director of OR and Perioperative Services, Napa-Solono (Vallejo),
California; and Co-chair, Regional Perioperative Management Group
(RPMG).
Former
Chief of Anesthesia, Northwest Permanente; Member, Interregional
ORBP Task Force member.