Open Writing

"On September 18, 2007, a few dozen neuroscientists, psychiatrists, and
drug-company executives gathered in a hotel conference room in Brussels
to hear some startling news. It had to do with a class of drugs known as
atypical or second-generation antipsychotics, which came on the market in
the early nineties. The drugs, sold under brand names such as Abilify,
Seroquel, and Zyprexa, had been tested on schizophrenics in several large
clinical trials, all of which had demonstrated a dramatic decrease in the
subjects' psychiatric symptoms. As a result, second-generation
antipsychotics had become one of the fastest-growing and most profitable
pharmaceutical classes. By 2001, Eli Lilly's Zyprexa was generating more
revenue than Prozac. It remains the company's top-selling drug.

"But the data presented at the Brussels meeting made it clear that
something strange was happening: the therapeutic power of the drugs
appeared to be steadily waning. A recent study showed an effect that was
less than half of that documented in the first trials, in the early
nineteen-nineties. Many researchers began to argue that the expensive
pharmaceuticals weren't any better than first-generation antipsychotics,
which have been in use since the fifties. 'In fact, sometimes they now look
even worse,' John Davis, a professor of psychiatry at the University of
Illinois at Chicago, told me.

"Before the effectiveness of a drug can be confirmed, it must be tested
and tested again. Different scientists in different labs need to repeat the
protocols and publish their results. The test of replicability, as it's known,
is the foundation of modern research. Replicability is how the community
enforces itself. It's a safeguard for the creep of subjectivity. Most of the
time, scientists know what results they want, and that can influence the
results they get. The premise of replicability is that the scientific
community can correct for these flaws.

"But now all sorts of well-established, multiply confirmed findings have
started to look increasingly uncertain. It's as if our facts were losing their
truth: claims that have been enshrined in textbooks are suddenly
unprovable. This phenomenon doesn't yet have an official name, but it's
occurring across a wide range of fields, from psychology to ecology. In
the field of medicine, the phenomenon seems extremely widespread,
affecting not only antipsychotics but also therapies ranging from cardiac
stents to Vitamin E and antidepressants: Davis has a forthcoming analysis
demonstrating that the efficacy of antidepressants has gone down as much
as threefold in recent decades. For many scientists, the effect is especially
troubling because of what it exposes about the scientific process. If
replication is what separates the rigor of science from the squishiness of
pseudoscience, where do we put all these rigorously validated findings
that can no longer be proved?
Which results should we believe? Francis
Bacon, the early-modern philosopher and pioneer of the scientific method,
once declared that experiments were essential, because they allowed us to
'put nature to the question.' But it appears that nature often gives us
different answers....

"[Joseph Banks Rhine, a psychologist at Duke, came to call this trend
toward a reduction in the strength of proof for a theory he had developed
in the early nineteen-thirties] the 'decline effect.'

"According to John Ioannidis, an epidemiologist at Stanford University, the
main problem is that too many researchers engage in what he calls
'significance chasing,' or finding ways to interpret the data so that it
passes the statistical test of significance - the ninety-five-per-cent
boundary invented by Ronald Fisher. 'The scientists are so eager to pass
this magical test that they start playing around with the numbers, trying to
find anything that seems worthy,' Ioannidis says. In recent years,
Ioannidis has become increasingly blunt about the pervasiveness of the
problem. One of his most cited papers has a deliberately provocative title:
'Why Most Published Research Findings Are False.'

"The problem of selective reporting is rooted in a fundamental cognitive
flaw, which is that we like proving ourselves right and hate being wrong.

'It feels good to validate a hypothesis,' Ioannidis said. 'It feels even better
when you've got a financial interest in the idea or your career depends
upon it. And that's why, even after a claim has been systematically
disproven' - he cites, for instance, the early work on hormone replacement
therapy, or claims involving various vitamins - 'you still see some
stubborn researchers citing the first few studies that show a strong effect.

They really want to believe that it's true.' ...

"The disturbing implication of a study [conducted in the late
nineteen-nineties by John Crabbe, a neuroscientist at the Oregon Health
and Science University] is that a lot of extraordinary scientific data are
nothing but noise. The problem, of course, is that ... dramatic findings are
... the most likely to get published in prestigious journals, since the data
are both statistically significant and entirely unexpected. Grants get
written, follow-up studies are conducted. The end result is a scientific
accident that can take years to unravel.

"This suggests that the decline effect is actually a decline of illusion. While
Karl Popper imagined falsification occurring with a single, definitive
experiment - Galileo refuted Aristotelian mechanics in an afternoon - the
process turns out to be much messier than that. Many scientific theories
continue to be considered true even after failing numerous experimental
tests. Verbal overshadowing might exhibit the decline effect, but it
remains extensively relied upon within the field. The same holds for any
number of phenomena, from the disappearing benefits of
second-generation antipsychotics to the weak coupling ratio exhibited by
decaying neutrons, which appears to have fallen by more than ten
standard deviations between 1969 and 2001....

"Such anomalies demonstrate the slipperiness of empiricism. Although
many scientific ideas generate conflicting results and suffer from falling
effect sizes, they continue to get cited in the textbooks and drive standard
medical practice. Why? Because these ideas seem true. Because they make
sense. Because we can't bear to let them go. And this is why the decline
effect is so troubling. Not because it reveals the human fallibility of science,
in which data are tweaked and beliefs shape perceptions. (Such
shortcomings aren't surprising, at least for scientists.) And not because it
reveals that many of our most exciting theories are fleeting fads
and will soon be rejected. (That idea has been around since Thomas Kuhn.) The
decline effect is troubling because it reminds us how difficult it is to prove
anything. We like to pretend that our experiments define the truth for us.

But that's often not the case. Just because an idea is true doesn't mean it
can be proved. And just because an idea can be proved doesn't mean it's
true. When the experiments are done, we still have to choose what to
believe."

author: Jonah Lehrer
title: "The Truth Wears Off"
publisher: The New Yorker
date: December 13, 2010
pages 52-57

**

Please visit http://www.delanceyplace.com/index.php