
Evaluating
the Research
January 1999
In
evaluating scientific research (both surveys and double-blind trials),
it's important to know two things:
-
How many people were involved (how large was the sample size)?
The larger the groups of people, the less likely it is that
the results are due to chance.
-
What is the probability (P) that the difference between the
groups is due to chance or other irrelevant factors? The smaller
the P value, the stronger the evidence. A value of P<.05
is good, P<.01 is better, and P<.001 means an incredibly
tiny possibility that the result was due to chance.
Also,
is it possible that the differences or similarities between the
groups are due to some other factor entirely? This question can
be quite subtle and difficult to assess, especially if many potential
causes are involved.
Making
sense out of conflicting evidence
Double-blind
studies with small P value results are the strongest kind of scientific
evidence available, yet different groups of researchers doing double-blind
studies sometimes get different results. One study will say the
substance is effective; another will say it's not. Whom do we believe?
If
twenty studies were done, and seventeen conclude that a supplement
produces a health benefit where only three conclude it does not,
the chances are good that the supplement does in fact provide a
health benefit.
Funding
source
Another
factor in judging research is the source of funding. When the results
of a study work to the advantage of the sponsor, and the study's
results are different from those obtained by independent researchers,
it suggests that the methods used in the study may have been biased.
Peer
review
Disagreements
and the potential for flawed research are the reasons why responsible
journals use the peer-review process to evaluate a research study
before they decide whether to publish it. If an article was published
in a peer-reviewed journal, you can be sure that other experts have
examined and critiqued the researchers' methods and findings.
What
kind of evidence is strongest?
In
examining the evidence, how does a well-informed consumer decide
whether to take a particular supplement? Each individual must decide
whether the likelihood and magnitude of a benefit are worth the
cost.
-
If
a friend says, "It worked for me," then the supplement might
be worth trying.
-
If
a culture's accumulated botanical knowledge indicates that many
people have benefited in a particular way, this is stronger
evidence that the supplement provides specific health benefits.
-
If
a survey of thousands of people shows more than 60 percent got
good results, the chances are good that you might too.
-
If
several double-blind studies show twice as many of the experimental
group benefited than the control (placebo) group, with a P value
of .01 or less, you can be quite sure that the supplement performs
as claimed for most people.
See
also SQ's links to information
about specific supplements.
What
kinds of scientific evidence are useful for substantiating a health
benefit claim? When
advertising says a health benefit is "proven by clinical research"
-- what does this mean?
|