OBSERVATIONS - may 14, 2009
Just Tell Me: How Thick is the Ice?
I was asked recently to help put some policy context around newly released research findings. But how do you tell the difference between just another medical study and one that deserves policy considerations?
It brought to mind my days twenty-five years ago at the American Enterprise Institute, when I worked with Herb Stein, noted economist and former head of President Nixon’s Council of Economic Advisers. The father of Ben Stein (teacher turned comic actor, columnist, and game show host), Herb was a very funny guy on his own. When we asked how to respond to press calls about the latest economic data, he gave us a blanket statement to use: “Pay no attention to the weekly/monthly/quarterly numbers; we need to see the monthly/quarterly/annual data.”
Herb was being facetious, but his “wait for more data” mantra describes the response many researchers take when asked about the significance of their findings. Almost invariably, studies conclude with the need for further research; rarely does anyone conclude that the answer has been found and now we can move on to a new question. The truth is that often researchers don’t think beyond getting their study published and getting the next project funded. So it is left to rest of us to figure out its significance.
I think of three kinds of significance: statistical, clinical and practical. Statistical significance is the easiest one to get—all you usually need is a large enough sample—but the least useful as it may or may not tell you what to do. It is necessary (and all too often sufficient) to get study published. Clinical significance raises the bar in linking a correlation to an outcome of interest, but sometimes the result isn’t actionable. (Knowing that breast and prostate cancer rates in Marin County are abnormally high is not a prescription for migration to San Francisco.) So that leaves practical significance: the results of the study are meaningful enough that we can and should take action.
There is where we need to sit down with researchers and ask some probing questions: How new is this? Is it profoundly different from what we thought before? Is this preliminary evidence or confirming evidence? How convinced are you that your research is definitive, credible, believable and reliable? Are the implied action steps plausible?
Obviously, we don’t want one piece of research to wag the dog. That can lead to being whipsawed by each new finding—coffee is bad for you one day, good for you the next. (My mother, the nutrition teacher, avoided all of this by noting, “There are no bad foods, just bad diets.”) At the same time, we can’t be paralyzed by the notion that more research is needed. So what we want researchers to tell us is how thick the ice is. We don’t want to fall into the lake, but if we’re always told it’s too thin, we can never go skating. Sometimes it’s thick enough—just tell us.
- Murray Ross, PhD, Vice President, Kaiser Foundation Health Plan, and Director, KP IHP
|
OBSERVATIONS
|
About
