
Types
of Scientific Evidence
January 1999
What
kinds of scientific evidence are valid for substantiating a health
benefit claim?
When
advertising says a health benefit is "proven by clinical research"
-- what does this mean? If someone dismisses reports of the benefits
of a traditional society's herbal remedies as "just anecdotal evidence"
-- what does that mean? What is anecdotal evidence, and how reliable
is it? What kinds of scientific evidence are available, and how
do we evaluate it?
Clinical
research
Clinical
research is an experiment done with people, rather than animals
or in a test tube. It says nothing about whether the experimenters
followed careful standards and research guidelines. (Ways of evaluating
clinical research are described in the next article.)
Anecdotal
evidence
Anecdotal
evidence is informal evidence that has not been subjected to rigorous
scientific analysis. It runs the gamut from a friend telling you,
"This supplement worked for me," to the accumulated wisdom of a
culture that has been using local plants for centuries both for
healing diseases and enhancing well-being.
Critics
of dietary supplements often dismiss this accumulated wisdom as
"just anecdotal evidence," but the fact that this knowledge has
not been scrutinized with modern statistical techniques does not
mean it has no value. Many of our modern drugs have come from this
kind of cultural knowledge of medicinal herbs: aspirin, quinine,
digitalis, and curare are four examples.
Statistical
surveys
As
new mathematical tools were developed during the past 150 years,
the science of statistical analysis brought heightened accuracy
to scientific research. We now know how to study relatively small
but representative samples of people and extrapolate the results
to the population at large.
The
evidence that smoking causes cancer and that oatmeal lowers cholesterol
were gathered from statistical surveys that compared the cancer
rates in smokers and nonsmokers, and the blood cholesterol levels
in people whose diets did or did not include oatmeal.
Potential
sources of error
The
main risk of error in any kind of research is that the difference
between the two groups can be due to chance rather than caused by
smoking or eating oatmeal. (Sometimes when you toss 10 coins, you
get 2 heads and 8 tails.) To offset this risk, researchers often
repeat their work with larger or different populations: people from
a different country or culture, or with older people or younger
people -- or with different researchers.
Random
chance is not the only factor that can produce "false positives":
apparent health benefits that are not due to the treatment or supplement
being studied.
Natural
healing and placebos
The
body heals itself quite efficiently, and the mind's belief that
it is receiving effective medicine is in itself a very powerful
healer. For decades, doctors gave sugar pills to their patients
when no other treatment was available simply because the placebo
effect is so strong.
On
October 13, 1998, The New York Times published an
article on the power of placebos and how they work. The article
reported that patients who had undergone "sham, placebo" knee surgery
experienced the same amount of relief from pain and swelling as
those who had the real operations, even two years after the surgery.
It also quoted research findings that placebos and modern antidepressant
drugs work about equally well, and that 42 percent of balding men
given a placebo either stopped losing hair or grew more of it. In
different cultures, the placebo effect for various treatments can
vary from as low as 6 percent to almost 100 percent.
Double-blind
clinical trials
Critics
of alternative medicine and dietary supplements often say that the
claimed results are "only the placebo effect at work." Given the
power of placebos, this is not necessarily a bad thing. The value
derived from taking drugs -- or even from undergoing surgery --
is certainly strengthened by the belief that the treatment will
work. In order to know whether a substance is directly causing a
health benefit, the question becomes: "Does this substance provide
health benefits that are stronger than a placebo?"
To
test this question, researchers use double-blind clinical trials,
which means that neither the researchers nor the subjects know who
is getting the real substance being tested and who is getting the
placebo. Research has shown that when doctors know which is the
sugar pill and which is the real thing, this knowledge affects the
results even if the patients don't know.
Evaluating
scientific research
Well-designed
double-blind clinical trials are the strongest kind of scientific
evidence for supporting a supplement's health claim, but a poorly
designed trial can distort the results and be almost worthless.
To learn how to investigate health claims in more detail, read the
next article on evaluating scientific
research.
See
also SQ's links to information
about specific supplements.
Standard
terminology: Terms used to describe dietary supplements, particularly
botanicals, are not always well-defined or consistently applied.
A name on one product label may not mean the same thing on another.
Evaluating scientific research: In evaluating scientific
research (both surveys and double-blind trials), it's important
to know two things . . .
|