A Focus on KP HealthConnect
••Fall 2004/Vol. 8, No. 4

Clinical Contributions

A similar version of this article was published in: The American Journal of Managed Care 2004 Sep;10(9):643-8. | to pdf >>
The Role of Research in Integrated Health Care Systems: The HMO Research Network
By Thomas M Vogt, MD, MPH, FAHA; Jennifer Elston Lafata, PhD; Dennis D Tolsma, MPH; Sarah M Greene, MPH

 

Abstract Integrated care systems have unique advantages for conducting research. The HMO Research Network (HMORN) includes research centers associated with 13 large integrated care systems whose research focuses on improving health and health care delivery using the extraordinary platform provided by these health systems. We conducted literature reviews and surveys and interviews with directors of HMORN research centers, research investigators, and selected support staff in order to identify the characteristics of the research in HMORN centers and to present examples of how this research has affected health and health policy. The 13 HMORN member health systems deliver health care to 13 million people. HMORN research centers have access to large, defined populations, comprehensive medical information, extensive computerized data systems and to medical care delivery systems that offer extraordinary research opportunities. HMORN centers publish about 1200 scientific articles each year and received about $180 million in external research funding in 2002, most of it from NIH, CDC, and other federal sources. More than 2000 research studies are currently underway at these centers, which employ approximately 1500 persons in the research activities. HMORN research centers have had a profound impact on health policy and care. New technologies are steadily expanding the research capacities of these research groups. Increased collaboration between academic and HMO researchers would enhance the work of both.

Introduction

The failure to efficiently translate research findings into care delivery has become a national crisis.¹ The Institute of Medicine (IOM) has emphasized the critical need to develop information systems that are designed to address the needs of clinical research.² This article discusses the work of the HMO Research Network (HMORN). The HMORN includes 13 research centers affiliated with integrated care delivery systems. These centers have the information systems that the IOM says are needed, and these systems are growing increasingly more complex and sophisticated.

Research in Integrated Care Settings--The HMO Research Network

Integrated care health maintenance organizations (HMOs) provide the optimal mix of population base, electronic medical and financial databases, and longitudinal observation for much health research. They are especially well situated for research addressing issues such as the costs and effectiveness of prevention and treatment practices, the organization of care, secular trends in diseases, and relative priorities on how to apportion scarce resources. The member organizations of the HMO Research Network are carrying out research that is crucial to improving the quality, availability, and effectiveness of health care. The 13 HMORN institutions carry out 90% of all research conducted by HMOs with formal research centers.³ This article describes the importance and extent of this research.

In 1961, Kaiser Permanente (KP) Northern California formed the Division of Research (DOR), the first of the integrated care research centers. The KP Northwest Center for Health Research in Portland, OR, followed three years later. Both centers are professionally autonomous health research centers in the public domain that use the integrated care system as a laboratory for research that improves care. These two research centers grew steadily over the years, competing successfully with universities for federal research grants and developing lines of research that influenced care at the national level. By the early 1970s, their research had already led to changes in federal regulations relating to Medicaid coverage.⁴ Today, six KP research centers employ more than 100 scientists and 1000 staff and publish about 600 articles per year in peer-reviewed medical literature.

Over the ensuing decades, other integrated care systems across the nation began to recognize the value of affiliated research centers that provide expert investigators the autonomy to develop and fund their own lines of research. In 1994, research centers in these geographically dispersed systems established a professional organization, the HMO Research Network (HMORN) in order to encourage high-quality, public domain research within HMOs. The 13 HMORN centers represent integrated care systems with approximately 13 million members. Collectively, they publish about 1200 scientific publications per year. These members include several that are exclusively prepaid group practices, several that are mixed models, and one that is an IPA-model HMO. These research centers vary in structure and organization, as do their parent health plans. However, they all have in common access to a defined population of members and access to data, much of it electronically available, that permits longitudinal evaluation of care practices. Their placement within large health systems makes feasible rigorous evaluation of alternative approaches to care.

HMORN research centers study a broad range of health and health care issues. Research conducted by member organizations is in the public domain, and the principle products of their studies are peer-reviewed publications. Table 1 lists the members of the HMORN and selected characteristics of their research centers. These centers are funded primarily through competitive grants and contracts from federal, foundation, and proprietary funding organizations and not by dues from health plan members. The annual HMORN scientific meeting combines presentations of scientific papers with seminars on how to develop collaborative research studies that permit the member organizations to collectively address research questions that cannot be carried out within single centers. HMORN members either already have fully automated electronic medical records (EMR) systems or are in the process of installing them. As they come online, these EMR systems will provide unprecedented opportunities to evaluate alternative approaches to treatment, long-term outcomes of care, cost effectiveness and cost benefits, rare disease epidemiology and treatment, and many other critical health care issues.

Research from the HMORN centers has profoundly affected the organization, delivery, and quality of care, federal and state health policies and regulations, plan benefits, and many other aspects of health and health care delivery. The 19 studies summarized in Table 2^5-29 are examples of HMORN projects that have influenced health care, health law, and health policy. At any given point in time, the members of the Research Network are conducting more than a thousand research studies.

HMORN research centers have also been key participants in some of the nation's most important multisite National Institutes of Health studies including the Multiple Risk Factor Intervention Trial (MRFIT),³⁰ the Systolic Hypertension in the Elderly Program (SHEP),³¹ the Study of Osteoporotic Fractures (SOF),³² the Trials of Hypertension Prevention (TOHP),³³ the Beta-Blocker Heart Attack Trial (BHAT),³⁴ the Women's Health Initiative (WHI),³⁵ the Dietary Approaches to Stop Hypertension (DASH) trial,¹⁴ and many others.

HMORN Collaborative Programs

In 1998, the HMORN received its first multi-institutional program award from the National Cancer Institute. The Cancer Research Network (CRN) includes 11 of the HMORN members. The CRN established an infrastructure to foster and facilitate development of new cancer research initiatives within integrated care systems. In addition, they conducted three large, multi-institutional research studies, each addressing questions that cannot be addressed within a single health system. These projects evaluated the impact of tobacco policies and training on smoking rates and patient satisfaction in health plans,^36,37 outcomes and the reasons for occurrence of late-stage breast and invasive cervical cancer among female plan members with full access to preventive screening services,³⁸ and effectiveness of earlier mammography and prophylactic mastectomy in reducing breast cancer mortality. The CRN has been renewed through 2007 with three new projects. In addition, 15 cancer research projects have been funded through the CRN network as separate grant applications, and several others are pending. Funded CRN-affiliated projects address a broad array of cancer research issues, including expanding enrollment in cancer clinical trials, cancer epidemiology, end-of-life care, effects of therapy on survival, and HRT use patterns.

The HMORN currently participates in five national, multisite research networks (Table 3): the CRN, the Center for Education and Research in Therapeutics (CERT), the Integrated Delivery System Research Network (IDSRN), the Cancer Care Outcomes Research and Surveillance (CanCORS) group, and the Vaccine Safety Datalink program. The first two are supported by the Agency for Healthcare Research and Quality (AHRQ), the third by the National Cancer Institute (NCI), and the

last by the Centers for Disease Control and Prevention. The HMORN also has formal partnerships with the American Association of Health Plans and the Alliance of Community Health Plans for the conduct of public health research.

Research Advantages in Integrated Care Systems

Defined population base--An entire population of plan members permits long-term observation of both numerator (the sick) and denominator (the population). This observation permits the estimation of rates that is essential to understanding changes over time, cause-and-effect relationships, and factors associated with disease incidence and treatment outcome.

Stable population base--Long-term cohort studies are critical elements of hypothesis formation and cause/effect determination (eg, Framingham Study, Study of Osteoporotic Fractures). These studies, though, are very expensive. However, in integrated care systems, many cohorts exist naturally, and data on those cohorts are already present in electronic form. This permits long-term cohort studies to be conducted retrospectively and at reasonable cost. The experiences of health plan members can serve to identify readily observable secular trends and outcomes of system interventions and can also serve as dependent variables when identifying risk factors and their interactions. Five of the 11 CRN sites examined the stability of enrollment of colorectal cancer cases to assure that study results were not skewed due to disenrollment of persons with cancer. Between 91% and 95% of survivors of cases were still enrolled two years after diagnosis, and 81-90% were still enrolled after five years.³⁹

High volunteer rates--Health plan members are more likely to respond to appeals for research volunteers when that appeal is from their own health plan. Recruitment and retention rates from integrated care system research cohorts are substantially higher than those from community recruitment.

Representativeness--Large, often nonprofit, integrated care systems are usually, though not always, demographically representative of their geographic populations. Medicaid and Medicare recipients can, and do, enroll as members assuring representation across age and income groups.

Diverse ethnicity--The members of the HMO Research Network are highly diverse ethnically, culturally, and geographically. One member (KP Hawaii) draws 75% of its membership from minority groups. KP Northern and Southern California each include very large numbers of Hispanic, Asian, and African-American groups. Henry Ford Health System (Detroit) and KP Georgia both have large African-American populations. Lovelace in New Mexico has a high concentration of Hispanic members. Working together, these groups can use existing databases to examine issues on ethnic diversity and its relation to care and care outcomes.

Comprehensive medical records--In private practice medicine, one person may see several physicians, each of whom maintains a separate medical record. All diagnoses, medications, lifestyle habits, and other pertinent information are not included in any single record. Many integrated care systems maintain comprehensive medical records that may include information across inpatient and outpatient settings. As they move toward electronic records (see below), this practice will become universal.

Electronic data lead to easy "preliminary" studies; rare disease studies--Successful research requires preliminary data, often collected at considerable expense, time, and energy. Comprehensive data information systems in integrated care can serve much of this need. Many research ideas can be successfully developed without pilot funds beyond those required for retrieving and analyzing data from existing databases. The evolution and various components of these data systems has been described elsewhere.⁴⁰

Electronic data also permit identification of uncommon diseases and treatments. Five of the 11 CRN sites identified 132,580 cancer cases for one study, including 2680 pancreatic cancer cases, 2788 ovarian cancers, 2986 bladder cancers, and 5147 non-Hodgkins lymphomas.³⁹

Automated medical records--Perhaps the greatest innovation in medical care in the 21st century will grow out of the shift to EMRs. EMR systems have the capacity to facilitate use of consensus guidelines, to minimize drug interactions and reactions, to design prevention and care plans that are individually tailored, and to provide explicit, detailed information on where care is being delivered according to optimal or suboptimal standards. In addition, these systems will provide extraordinary epidemiologic opportunities to observe disease trends, disease outcomes, and disease/risk-factor associations. They will provide health economists the opportunity to study the relative costs and effectiveness of different approaches to care and will assist in designing and evaluating alternative structures for delivering care.

Ability to test efficacy of care alternatives--Large, integrated delivery systems often experiment with innovations in delivery. These innovations can be rigorously evaluated when trained researchers are involved. Many health systems pride themselves on their innovations, but sound evaluation requires rigorous methodology. KP supports the Garfield Memorial Fund for the purpose of providing support to its research centers for development, implementation, and evaluation of system innovations.

Location inside of health care systems--The presence of the HMORN centers within health care systems encourages interactions and critical partnerships among researchers, clinicians, and managers early in the research process. This interaction facilitates implementation and testing in real-world settings. The perspectives of managers, clinicians, and staff often lead researchers to modify naïve assumptions about what will work or will not work within their systems and to support development of functional innovations that can be successfully translated into practice. Dissemination of scientific findings into health care is a serious national problem.¹ The presence of research in health settings also helps managers, clinicians, technicians, and staff to respect research activities as legitimate and to view support of the research enterprise as legitimate and integral to providing care. Facilitating research serves the clinicians, the patients, and, ultimately, the entire system. These partnerships create an environment that facilitates the translation of research findings into practice.

Research budgeting expertise--All large health systems engage to some degree in proprietary research (eg, research funded by drug companies). Recent audits at two of the HMORN institutions showed that for every dollar they took in for proprietary research, they spent between two and three dollars. HMORN research centers can provide expertise to their parent systems for assuring that research budgets cover actual costs and do not drain funds from health care premiums.

Why Should Integrated Care Systems Participate in Research?

Outcomes management requires an infrastructure in which population-based outcomes can be readily assessed.⁴¹ There are no comparable environments for addressing many of these issues in the United States. We believe that these advantages are so powerful that large integrated ca re systems actually have a social obligation to participate in research as a part of the healthcare process. The reasons for academic researchers to form effective partnerships and collaborations with HMO-based research centers are also compelling.¹⁸

Members benefit--Research provides members an opportunity to make a contribution. An unpublished survey done by the senior author of more than 300 volunteers in a randomized trial of hypertension medications showed that most volunteered primarily to help others and not because they expected personal benefit. Research makes some therapies available to patients earlier than would otherwise be the case. This strategy is particularly valuable when standard therapy offers little benefit.

Physicians and staff benefit--Research participation helps clinicians to stay abreast of new developments; it provides new activities that make their work more interesting and relevant; and it enhances job satisfaction and retention. Research also makes physicians advocates for change when research findings support that change. Research brings additional skills and perspectives into the health care setting.

Interdisciplinary research--The HMO research environment fosters multidisciplinary research.⁴ This integration of disciplines is essential to understanding the complex interrelationships of health services and their outcomes.

Discussion

"The US health care system becomes a more embarrassing disaster each year ..."⁴³ and it is "failing in front of our eyes"⁴⁴--particularly with respect to our ability to synthesize the mountains of information required to optimize care. The reasons for the morbid state of US health care are rooted in our medical history and our economic structure. They arise from complex, confusing, and constantly changing reimbursement processes; from perverse incentives that encourage excessive services; from our love affair with expensive technologies; from our inability to stop doing what doesn't work; from a legal system that encourages fault finding and paranoia instead of remedial action; and from the lack of a systematic means for learning from our mistakes and for translating those learnings back into practice. Research that takes advantage of integrated care system opportunities cannot address all of these problems, but it is a critical step in the needed information synthesis. Research within integrated care systems can develop new concepts and methods that define basic goals; design practical tools that document the nature and magnitude of problems and outcomes; evaluate strategies and interventions for improving care; and evaluate new models, programs, and systems.⁴⁵ A balanced research portfolio requires investigator-initiated development of theory, methods and measures, organizational and systems research, effectiveness and cost-effectiveness studies, management and implementation research, results that can be understood and integrated into practice, and development of researchers skilled in these areas.⁴⁵ Research priorities should be based on clinical realities and economic epidemiology and also should be guided by recognizing deficiencies in conventional wisdom.⁴⁶

In integrated care systems, these key factors intersect. Economics pushes those systems to avoid unnecessary services, quality assurance processes such as the Healthplan Employer Data Information Set (HEDIS) push them toward quality improvement, and the clinical setting requires them to take into account the real-world realities of delivering care. Their settings are ideal for testing and evaluating various preventive and treatment services and for evaluating different organizational structures.

The proliferation of EMR systems will dramatically enhance the quantity and quality of performance assessments both within and across health care systems.⁴⁷ These systems will also greatly improve the capacity to perform inexpensive retrospective evaluations. They can assess quality of care at the patient, provider, clinic, and system levels and can prompt clinicians on current guidelines, potential drug interactions, and overdue services. They permit prospective cohort studies to be performed retrospectively, allow identification of rare diseases for recruitment into studies, and facilitate cost-effectiveness analyses of various approaches to treatment. All of these advantages afforded by EMRs depend on the presence of diverse, defined populations such as those found in integrated care systems.

The HMORN recognizes that access to these extraordinary resources is a public trust. The HMORN vision is to "transform US health care through research on the diverse populations served by integrated health systems." Its aim is "to become the premier resource for population-based research by drawing on the unique member and geographic diversity of the network and its organization, human capital, and data resources." HMORN accomplishes these aims through public domain research that serves the public interest. Integrated care research has moved into the mainstream of health care research in the US. In the future, it will become increasingly important in the formation of policy and practice.

In summary, integrated care systems are such an ideal setting to conduct many types of applied medical research that the larger integrated care delivery systems have a social obligation to actively support and participate in such research. High-quality medical care requires high-quality research and evaluation. Clinicians, managers, and the public must come to view research as an integral and essential part of what health systems do. The design of data systems in large health care organizations needs to include considerations relating to research and evaluation. University researchers should develop closer partnerships with their health system research colleagues to improve the quality and quantity of research in these settings.⁴²

Science is the Basis of Medicine

Good science leads to better decisions and systems that are effective in supporting those decisions. Where would you prefer to send a loved one for the best possible care? Most people think immediately of research institutions--the Mayo Clinic, Sloan Kettering, MD Anderson, Cleveland Clinic, etc. Although less widely recognized, the members of the HMO Research Network are quietly joining this elite group.

Acknowledgments
The authors would like to thank the many persons who contributed information for inclusion in this article, including: Jerry Gurwitz, MD (Meyers Primary Care Institute); Diana Petitti, MD, MPH, Robert S Zeiger, MD (Kaiser Permanente Southern California); Barbara Stoddart, Arne Beck, PhD, Paul Barrett, MD (Kaiser Permanente Colorado); Margaret Gunter, PhD (Lovelace Clinic Foundation); Eric Larson, MD (Lovelace Clinic Foundation); Andrew Nelson, MPH, Kate Rardin, MPH, Barbara Olson Bullis (HealthPartners of Minneapolis); Mary Durham, PhD, Don Freel, (Kaiser Permanente Northwest); Ryne Lee (Kaiser Permanente Hawaii); Richard Platt, MD, MPH, and Ann Plasso (Harvard Pilgrim).

References

1. Rosenberg RN. Translating biomedical research to the bedside: a national crisis and a call to action. JAMA 2003 Mar 12;289(10):1305-6.
2. Sung NS, Crowley WF Jr, Genel M, et al. Central challenges facing the national clinical research enterprise. JAMA 2003 Mar 12;289(10):1278-87.
3. Nelson AF, Quiter ES, Solberg LI. The state of research within managed care plans: 1997 survey. Health Aff (Millwood) 1998 Jan-Feb;17(1):128-38.
4. Greenlick M, Pope C, Freeborn D. Health Care Research in an HMO: Two Decades of Discovery. Baltimore: The Johns Hopkins University Press;1988:183-91.
5. Colombo T, Saward EW, Greenlick MR. The integration of an OEO health program into a prepaid comprehensive group practice plan. Am J Public Health Nations Health 1969 Apr;59(4):641-50.
6. Greenlick M. Medical service to poverty groups. In: Somers A, editor. The Kaiser Permanente Medical Care Program: A Symposium, New York: Commonwealth Fund; 1971. p 138-51.
7. Greenlick M, Lamb S, Carpenter TJ, et al. Kaiser Permanente's Medicare Plus Project: a successful Medicare prospective payment demonstration. Health Care Financing Rev 1983(4):85-97.
8. Black SB, Shinefield HR, Lampert D, et al. Safety and immunogenicity of oligosaccharide conjugate Haemophilus influenzae type b (HBOC) vaccine in infancy. The Northern California Kaiser Permanente Vaccine Study Center Pediatrics Group. Pediatr Infect Dis J 1991 Feb;10(2):92-6.
9. Black SB, Shinefield HR, Fireman B, Hiatt R, Polen M, Vittinghoff E. Efficacy in infancy of oligosaccharide conjugate Haemophilus influenzae type b (HbOC) vaccine in a United States population of 61,080 children. The Northern California Kaiser Permanente Vaccine Study Center Pediatrics Group. Pediatr Infect Dis J 1991 Feb;10(2):97-104.
10. Mullooly JP, Bennett MD, Hornbrook MC, et al. Influenza vaccination programs for elderly persons: cost-effectiveness in a health maintenance organization. Ann Intern Med 1994 Dec 15;121(12):947-52.
11. Selby JV, Friedman GD, Quesenberry CP Jr, Weiss NS. A case-control study of screening sigmoidoscopy and mortality from colorectal cancer. N Engl J Med 1992 Mar 5;326(10):653-7.
12. Hollis JF, Lichetenstein E, Vogt TM, Stevens VJ, Biglan A. Nurse-assisted counseling for smokers in primary care. Ann Intern Med 1993 Apr 1;118(7):521-5.
13. Hollis JF, Vogt TM, Stevens V, Biglan A, Severson H, Lichtenstein E. The Tobacco Reduction and Cancer Control (TRACC) Program: team approaches to counseling in medical and dental settings. In: Tobacco and the clinician: interventions for medical and dental practice. Washington, DC: US Dept of Health and Human Services, Public Health Service; 1994. (NIH publication 94-3693. NCI Smoking and Control Monograph 5.)
14. Appel LJ, Moore TJ, Obarzanek E, et al. A clinical trial of the effects of dietary patterns on blood pressure. DASH Collaborative Research Group. N Engl J Med 1997 Apr 17;336(16):1117-24.
15. Moore T, Svetkey L, Lin, P, Karanja N, Jenkins M. The DASH Diet for Hypertension. New York: The Free Press; 2001.
16. Friedman GD, Collen MF,m Fireman BH. Multiphasic health checkup evaluation: a 16-year follow-up. J Chronic Dis 1986;39(6):453-63.
17. Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA 2003 Mar 5; 289(9):1107-16.
18. Ownby DR, Johnson CC, Peterson EL. Exposure to dogs and cats in the first year of life and risk of allergic sensitization at 6 to 7 years of age. JAMA 2002 Aug 28;288(8):963-72.
19. Flamm B, Lim OW, Jones C, Fallon D, Newman LA, Mantis JK. Vaginal birth after cesarean section: results of a multicenter study. Am J Obstet Gynecol 1988 May;158(5):1079-84.
20. Long-term effects of budesonide or nedocromil in children with asthma. The Childhood Asthma Management Program Research Group. N Engl J Med 2000 Oct 12;343(15):1054-63.
21. Cummings P, Koepsell TD, Grossman DC, Savarino J, Thompson RS. The association between the purchase of a handgun and homicide or suicide. Am J Public Health;1997 Jun;87(6):974-8.
22. Wagner Eh. The role of patient care teams in chronic disease management. BMJ 2000 Feb 26;320(7234):569-72.
23. Thompson DC, Nunn ME, Thompson RS, Rivara FP. Effectiveness of bicycle safety helmets in preventing serious facial injury. JAMA 1996 Dec 25;276(24):1974-5.
24. Scholes D, Stergachis A, Heidrich FE, Andrilla H, Holmes KK, Stamm WE. Prevention of pelvic inflammatory disease by screening for cervical chlamydial infection. N Engl J Med 1996 May 23;334(21):1362-6.
25. Maciosek MV, Coffield AB, McGinnis JM, et al. Methods for priority setting among clinical preventive services. Am J Prev Med 2001 Jul;21(1):10-9.
26. Coffield AB, Maciosek MV, McGinnis JM, et al. Priorities among recommended clinical preventive services. Am J Prev Med 2001 Jul;21(1):1-9.
27. Nerenz DR, Bonham VL, Green-Weir R, Joseph C, Gunter M. Eliminating racial/ethnic disparities in health care: can heath plans generate reports? Health Aff (Millwood) 2002 May-Jun;21(3):259-63.
28. McDuffie RS Jr, Beck A, Bischoff K, Cross J, Orleans M. Effect of frequency of prenatal care visits on perinatal outcome among low-risk women. A randomized controlled trial. JAMA 1996 Mar 20;275(11):847-51.
29. Beck A, Scott J, Williams P, et al. A randomized trial of group outpatient visits for chronically ill older HMO members: the Cooperative Health Care Clinic. J Am Geriatr Soc 1997 May;45(5):543-9.
30. Multiple Risk Factor Intervention Trial. Risk factor changes and mortality results. Multiple Risk Factor Intervention Trial Research Group. 1982. JAMA 1997 Feb 19;277(7):582-94.
31. Perry HM Jr, Smith WM, McDonald RH, et al. Morbidity and mortality in the Systolic Hypertension in the Elderly Program (SHEP) pilot study. Stroke 1989 Jan;20(1):4-13.
32. Bauer DC, Gluer CC, Cauley JA, et al. Broadband ultrasound attenuation predicts fractures strongly and independently of densitometry in older women. A prospective study. Study of Osteoporotic Fractures Research Group. Arch Intern Med 1997 Mar 24;157(6):629-34.
33. Stevens VJ, Corrigan SA, Obarzanek E, et al. Weight loss intervention in phase 1 of the Trials of Hypertension Prevention. The TOHP Collaborative Research Group. Arch Intern Med 1993 Apr 12;153(7):849-58.
34. The beta-blocker heart attack trial. beta-Blocker Heart Attach Study Group. JAMA 1981 Nov 6;246(18):2073-4.
35. Shumaker SA, Reboussin BA, Espeland MA, et al. The Women's Health Initiative Memory Study (WHIMS): a trial of the effect of estrogen therapy in preventing and slowing the progression of dementia. Control Clin Trials 1998 Dec;19(6):604-21.
36. Rigotti NA, Quinn VP, Stevens VJ, et al. Tobacco-control policies in 11 leading managed care organization: progress and challenges. Eff Clin Pract 2002 May-Jun;5(3):130-6.
37. Solberg LI, Hollis JA, Stevens VJ, Rigotti NA, Quinn VP, Aickin M. Does methodology affect the ability to monitor tobacco control activies? Implications for HEDIS and other performance measures. Prev Med 2003 Jul;27(1):33-40.
38. Reisch LM, Fosse JS, Beverly K, et al. Training, quality assurance, and assessment of medical record abstraction in a multisite study. Am J Epidemiol 2003 Mar 15;157(6):546-51.
39. Field TS, Cernieux J, Buist D, et al. Retention of enrollees following a cancer diagnosis within health maintenance organizations in the Cancer Research Network. J Natl Cancer Inst 2004 Jan 21;96(2):148-52.
40. Fishman PA, Wagner EH. Managed care data and public health: the experience of Group Health Cooperative of Puget Sound. Annu Rev Public Health 1998;19:477-91.
41. Epstein RS, Sherwood LM. From outcomes research to disease management: a guide for the perplexed. Ann Intern Med 1996 May 1;124(9):832-7.
42. Mechanic RE, Dobson A. The impact of managed care on clinical research: a preliminary investigation. Health Aff (Millwood) 1996 Fall;15(3):72-89.
43. Kennedy D. When the price is wrong. Science 2003 Aug 15;301(5635):895.
44. Bloom FE. Presidential address. Science as a way of life: perplexities of a physician-scientist. Science 2003 Jun 13;300(5626):1680-5.
45. Feldman PH, Kane RL. Strengthening research to improve the practice and management of long-term care. Milbank Q 2003;81(2):179-220, 171.
46. Kristein M. Determining research priorities. Clin Invest Med 1983;6(2):57-8.
47. Clancy CM, Eisenberg JM. Outcomes research: measuring the end results of health care. Science 1998 Oct9;282(5387):245-6.

 

 

To Clinical Contributions contents list >>

To full contents list >>