Results of the First National Kaiser Permanente Continuing Medical Education Needs Assessment Survey
By David W Price, MD, FAAFP; Chris C Overton, MS; Joy Pfeifer Duncan; Donna A Wamsley; Carol Havens, MD; Jill Steinbruegge, MD; KM Tan, MD; Marc Klau, MD; Ann Hellerstein, MD; Felice Klein, RN, MN; Debra Mipos, MPA; Leslie Francis, MBA, MHA

Introduction

The Kaiser Permanente National Continuing Medical Education Committee (KPNCME) was formed in 1998 to help provide continuing medical education (CME) opportunities and CME credit for Kaiser Permanente (KP) physicians through a variety of methods, including national conferences and enduring materials (eg, journals, Internet modules, CD-ROM). The KPNCME protects the "brand image" of the KP name by accrediting national KP conferences and by assuring that they adhere both to internal standards and to standards set by the Accreditation Council for Continuing Medical Education (ACCME). KPNCME members serve as liaisons to national CME conferences, seek to improve quality of program development, and serve as resources for design and delivery of CME programs. The KPNCME also provides a venue for members of the Permanente Medical Groups nationwide to share innovative CME ideas and programs. The KPNCME mission statement is included in Appendix A.

Needs assessment is an important part of planning CME programs. This assessment is used to develop objectives for educational programs by linking learners' goals with quality, utilization, or other performance data; with local and regional initiatives; and with emerging medical information. Needs assessment can also be conducted to evaluate general preference for future topics of meetings; for preferred schedule and location of programs; and for preferred educational format. To leverage resources, the KPNCME developed a programwide, general needs assessment designed to provide information to regional and national CME planners. We present national aggregate data collected from the first KP national CME needs assessment (Regional Directors of CME have received data on their respective medical groups). These national results generally reflect regional results.

Methods

Survey Tool for Assessing KP Clinicians' CME-Related Needs
For a subset of physicians of known length of experience and medical specialty, a survey was designed to assess motivation for attending CME programs; barriers to CME program participation; preferred format for CME; preferred schedule and location of national CME programs; and preferred future topics of CME programs.

Survey Development and Distribution
The initial hard-copy survey was developed in 1998 and was pilot- tested among Permanente physicians in the KP Colorado and Northwest Regions. Internal correlations were made to reduce redundancy and shorten the questionnaire. A pilot version of the revised survey was also posted on the KP Northwest Region's Physicians and Surgeons Web site from July through September 2000. After Web masters, Web site coordinators, and education staff refined their interoffice workflow, the survey was made available online to PMG physicians nationwide from November through December 2000. In addition, paper copies of the survey were mailed to PMG physicians nationwide in November 2000. Physicians were given the option of responding to the paper version of the survey or to the survey on the Internet. As incentive to complete and return the survey, a drawing for a $25 gift certificate was held in each medical group for physicians returning a completed paper survey; for physicians who completed the online survey, a drawing for a $100 gift certificate was held in each medical group.

Responses to the online survey were automatically captured in a Microsoft Access database before conversion to a Microsoft Excel file. Results of the paper survey were manually entered directly into the Microsoft Excel database. National and regional data were then analyzed and reported.

Statistical Analysis

The Statchek Statistical Analysis Program, version 1986 (Detail Technologies, Inc, Princeton, NJ) was used to determine the accuracy of the survey results.

Results

Survey Return Rates and Demographics of Respondents
Of 10,959 surveys distributed to PMG physicians, 1976 surveys (1351 paper, 625 online) were returned for an overall return rate of 19.1% (Table 1). On the basis of sample size and return rate, preference of Permanente physicians was determined to within ±2% (95% confidence level).

Table 2 depicts medical specialty and years in practice for survey respondents. No national database of KP physicians exists for comparison, but distribution of specialty is considered generally representative of KP physicians nationwide. The category for anesthesiology was inadvertently omitted from the survey instrument; as a result, many respondents who described their medical specialty as "other" are anesthesiologists. The next national needs assessment will correct this oversight.

Table 3 summarizes responses of physicians when asked to rate influence of several factors on their choice of CME programs according to a six-point scale ranging from least influential (score of one) to most influential (score of six). Responses showed that Permanente physicians choose CME on the basis of self-perceived need and tend not to be influenced either by perspectives held by peers or by performance feedback. Permanente physicians also select topics related to attaining new skills that represent a major part of their practice or that provide an opportunity to improve clinical outcome. Choice of CME program is influenced more by location of program than by time of day at which the program is offered.

Topics of greatest interest to respondents are listed in Table 4. Because more than 40% of responses received were submitted by internists, pediatricians, and family physicians, preference reflected in these responses indicates the broad nature of primary care practice.

When asked to rate, using a five-point scale, their preference for CME format on the basis of format utility and effectiveness, KP physicians expressed strong preference for group learning activities (Table 5). Most respondents indicated that they do not yet view newer, computer-based CME format (eg, CD-ROM, Internet) as useful. However, 29% of all respondents and 40% of the 625 online respondents (data not shown) indicated that online format was "most useful." One in five respondents overall and 27% of online respondents (data not shown) rated floppy disk- or CD-ROM-based format as "most useful." We will be interested to reassess the usefulness of the computer-based CME format over time as our physicians increasingly use computers (eg, for access to the KP National Computerized Information System and the Permanente Knowledge Connection) in the workplace.

Asked to assess, using a four-point scale, effectiveness of various types of CME format in helping change or improve their practice, physicians stated that they preferred interactive sessions and perceived these sessions as the best way to gain knowledge and improve practice (Table 5). Compared with physicians responding to the paper version of the survey, physicians responding online were more likely to rate the online format (59% of respondents) or the floppy disk or CD-ROM format (44% of respondents) as "almost always" helpful for changing their practice (data not shown). These results were consistent with respondents' stated preference rated on the
basis of format utility (Table 5).

Asked to identify, using an eight-point scale, barriers to their participation in CME programs, most respondents indicated that program location, schedule, and cost are the greatest barriers to participation (Table 6). Responses seemed to show positive experience with CME as well as belief that CME is valuable, important, and helpful for career advancement.

Asked to rate, using a three-point scale, broad categories of CME as they relate to helping clinicians provide patient care, respondents indicated that they valued topics relating to evidence-based medicine and clinical guidelines and did not assign high priority to CME programs designed to enhance presentation skills, office-based teaching of students and residents, or cultural competence (Table 7).

Asked their preference for schedule and location of KP national CME programs, nearly 60% of respondents stated that they had not attended any of these programs and favored rotating the time of year and location of KP National CME programs to facilitate attendance by the largest number of clinicians (Table 8). Respondents stated that they were willing to travel to attend national CME programs and agreed that these programs should be designed to accommodate family and relaxation time.

Discussion

Results of this survey offer valuable information for planning future CME programs. Specifically, although consistent with the published findings that self-perceived need is a powerful motivator for physicians to attend CME,^1,2 results shown in Table 3 suggest that we are missing the opportunity to use our readily available physician performance data to design CME programs and to help our physicians select CME programs. Individual physicians and their supervisors may not perceive that data regarding quality of care, medical utilization, and feedback from peers or patients are connected to CME. Partly for this reason, CME might not be included in routine evaluation of physician job performance. Providing "hard data" can complement self-perception in helping clinicians develop individual CME programs designed to improve practice outcomes, clinical knowledge, and physician comfort.

Table 5 shows that KP physicians strongly prefer lecture and other group activities and that physicians feel the lecture format can help change medical practice. This preference for the lecture format might be a consequence of physicians' long familiarity with the format (dating back to college and medical school) and the opportunity the format provides them to interact with colleagues. However, by themselves, CME lectures rarely lead clinicians to change their behavior or lead to improved patient outcomes.^3-7 Nonetheless, the lecture format has been suggested as valuable for raising awareness--in particular, awareness of new information--and for helping clinicians to decide on practice changes.^7-9 Lectures can continue to be used selectively--especially in these ways--but must be combined with more interactive learning format to allow participants to practice new skills and discuss how to implement new practices and behavior. Tools for facilitating and reinforcing behavioral change should therefore be designed for use before and after completion of CME programs so that clinicians may improve their chances of achieving desired practice outcomes.^{3,4,6,7,10,11}

We were not surprised to find that physicians perceive evidence-based medicine and clinical guidelines as most valuable for providing clinical care (Table 7). Presentation skills and office-based teaching and precepting are more likely to be important to the minority of respondents who are organizational leaders, CME faculty, or regular teachers of medical students and residents. That customer service, communication, and cultural competence were the CME topics considered least valuable highlights the gap between the importance of these topics as perceived by clinicians and the importance of the topics as perceived by administrators who select organizational initiatives. To help close this gap, we suggest that these topics be incorporated into clinical curricula at the "examination room" level. For example, instead of teaching physicians how to be "culturally competent," a more appropriate strategy might be to incorporate into a clinical context use of skills specifically related to diverse populations. By relating skills directly to patient care, physicians will increasingly find relevance in topics such as customer service and cultural competence.

Next Steps for the KPNCME Committee

As a result of this needs assessment, the KPNCME Committee has developed several follow-up steps for enhancing CME programs:

Recommendations

On the basis of these results, the KPNCME Committee has developed the following recommendations:

Recommendations for CME Planners and Organizational Leaders
Explicit, strong links should be created between CME planning (individually, locally, and nationally) and measures of quality and utilization as well as other objectively measured data.

At all levels--from local department chiefs to Regional Medical Directors--Permanente leaders should encourage physicians to select CME programs on the basis of individual performance data and subjectively perceived need. One strategy for accomplishing this goal would be for each physician's regular performance evaluation to incorporate discussion of completed and planned CME activities.

Local, regional, and national CME program planners should incorporate high-level quality and utilization data into selection of topics for CME programs.

When designing group educational events or interventions, CME planners should, whenever possible, provide opportunities for interaction among learners and between CME faculty and learners. Conferences should incorporate multiple teaching formats, including lectures (to introduce new information) and small, skill-based, interactive groups (to provide structured opportunity for practicing skills, networking, and sharing ideas).

Skills such as cultural competence, customer service, and patient-physician communication should be taught in the context of "clinical curriculum" by teaching how these specific skills relate to care of patients with specific problems. This approach will show the practicality and utility of these skills for physicians and will increase the likelihood of implementing new skills in medical practice. CME planners should consider formally evaluating how practice and patient outcomes are affected by this method of teaching these skills.

To reach a broader potential audience, KP national CME programs should be given at rotating locations and on various dates.

Recommendations for Physicians as Learners
To enhance their learning experience, physicians can:

Acknowledgments:

We would like to thank Wendy Ray for her help in using the Statchek^® program; and Michelle Bolke, who helped put the needs assessment into online format.

References:

Purpose
The Kaiser Permanente National CME Program seeks to improve the skills and effectiveness of individual clinicians in the clinical encounter and physician leaders in the development and management of the care delivery system so that patients receive high quality, caring, affordable health care. By providing learning opportunities designed to improve the clinical, behavioral, and leadership skills and knowledge of physicians and physician leaders, the National CME Program supports the organization's national strategy to improve the health of Kaiser Permanente members and the health of the communities which are served by Kaiser Permanente.

With a National CME Program in place, Kaiser Permanente will be well positioned to realize the organizational vision of becoming the world's leader in improving health through high quality, caring, affordable, integrated health care.

Content Areas
In addition to focusing on clinical and behavioral competencies in the provision of care to patients, we believe it is critical for physicians to receive relevant training in leadership and management tools and strategies. Therefore, the scope of the National Permanente CME program will focus on three areas: Clinical skills (cognitive and technical); service behavior skills (both clinician-patient communication as well as interpersonal relationship skills); and leadership and management skills (business knowledge and skills, systems thinking, problem-solving, etc).

Target Audience
Permanente physicians practice in many states. Our program is designed to offer CME activities to all Permanente physicians in the United States as well as to affiliate medical group physicians and to individual practitioners with whom we contract to provide care to Kaiser Permanente members. The educational activities are suitable for including other members of the health care team as appropriate.

Types of Activities Provided
We will provide educational activities to improve the clinical, technical, behavioral, and leadership skills of Kaiser Permanente physicians. These activities will support the communication of new scientific information and clinical guidelines; acquisition of technical, behavioral, medical informatics skills; dissemination of clinical and delivery system innovation and best practices; and development of leadership knowledge and skills.

We will develop and implement activities using a variety of learning modalities, such as conferences, workshops, symposia, videoconferencing, self-administered and enduring materials such as videotapes, CD-ROM, and Internet-based CME activities.

Expected Results
We expect our educational activities to improve the care delivered by Kaiser Permanente physicians and by associate providers in the following ways:

Ultimately, educational activities should improve health outcomes, patient satisfaction with the care received, clinicians' satisfaction with their work, and effectiveness of the delivery systems to support the care provided to patients. Although the impact of any individual educational activity on these outcomes cannot be easily determined, the Kaiser Permanente National CME Program seeks to ensure that educational activities are planned with these results as goals.

Reproduced by permission of: Jill Steinbruegge, MD, AED for Physician Development, The Permanente Federation, Oakland, CA.