Types of Scientific Evidence
January 1999

What kinds of scientific evidence are valid for substantiating a health benefit claim?

When advertising says a health benefit is "proven by clinical research" -- what does this mean? If someone dismisses reports of the benefits of a traditional society's herbal remedies as "just anecdotal evidence" -- what does that mean? What is anecdotal evidence, and how reliable is it? What kinds of scientific evidence are available, and how do we evaluate it?

Clinical research

Clinical research is an experiment done with people, rather than animals or in a test tube. It says nothing about whether the experimenters followed careful standards and research guidelines. (Ways of evaluating clinical research are described in the next article.)

Anecdotal evidence

Anecdotal evidence is informal evidence that has not been subjected to rigorous scientific analysis. It runs the gamut from a friend telling you, "This supplement worked for me," to the accumulated wisdom of a culture that has been using local plants for centuries both for healing diseases and enhancing well-being.

Critics of dietary supplements often dismiss this accumulated wisdom as "just anecdotal evidence," but the fact that this knowledge has not been scrutinized with modern statistical techniques does not mean it has no value. Many of our modern drugs have come from this kind of cultural knowledge of medicinal herbs: aspirin, quinine, digitalis, and curare are four examples.

Statistical surveys

As new mathematical tools were developed during the past 150 years, the science of statistical analysis brought heightened accuracy to scientific research. We now know how to study relatively small but representative samples of people and extrapolate the results to the population at large.

The evidence that smoking causes cancer and that oatmeal lowers cholesterol were gathered from statistical surveys that compared the cancer rates in smokers and nonsmokers, and the blood cholesterol levels in people whose diets did or did not include oatmeal.

Potential sources of error

The main risk of error in any kind of research is that the difference between the two groups can be due to chance rather than caused by smoking or eating oatmeal. (Sometimes when you toss 10 coins, you get 2 heads and 8 tails.) To offset this risk, researchers often repeat their work with larger or different populations: people from a different country or culture, or with older people or younger people -- or with different researchers.

Random chance is not the only factor that can produce "false positives": apparent health benefits that are not due to the treatment or supplement being studied.

Natural healing and placebos

The body heals itself quite efficiently, and the mind's belief that it is receiving effective medicine is in itself a very powerful healer. For decades, doctors gave sugar pills to their patients when no other treatment was available simply because the placebo effect is so strong.

On October 13, 1998, The New York Times published an article on the power of placebos and how they work. The article reported that patients who had undergone "sham, placebo" knee surgery experienced the same amount of relief from pain and swelling as those who had the real operations, even two years after the surgery. It also quoted research findings that placebos and modern antidepressant drugs work about equally well, and that 42 percent of balding men given a placebo either stopped losing hair or grew more of it. In different cultures, the placebo effect for various treatments can vary from as low as 6 percent to almost 100 percent.

Double-blind clinical trials

Critics of alternative medicine and dietary supplements often say that the claimed results are "only the placebo effect at work." Given the power of placebos, this is not necessarily a bad thing. The value derived from taking drugs -- or even from undergoing surgery -- is certainly strengthened by the belief that the treatment will work. In order to know whether a substance is directly causing a health benefit, the question becomes: "Does this substance provide health benefits that are stronger than a placebo?"

To test this question, researchers use double-blind clinical trials, which means that neither the researchers nor the subjects know who is getting the real substance being tested and who is getting the placebo. Research has shown that when doctors know which is the sugar pill and which is the real thing, this knowledge affects the results even if the patients don't know.

Evaluating scientific research

Well-designed double-blind clinical trials are the strongest kind of scientific evidence for supporting a supplement's health claim, but a poorly designed trial can distort the results and be almost worthless. To learn how to investigate health claims in more detail, read the next article on evaluating scientific research.

See also SQ's links to information about specific supplements.

Standard terminology: Terms used to describe dietary supplements, particularly botanicals, are not always well-defined or consistently applied. A name on one product label may not mean the same thing on another.

Evaluating scientific research: In evaluating scientific research (both surveys and double-blind trials), it's important to know two things . . .