Trust in science would be improved by study pre- registration
In an ideal world, scientific
discoveries would be
independent of what scientists
wanted to discover. A good
researcher would begin with
an idea, devise a method to
test the idea, run the study as
planned, and then decide
based on the evidence
whether the idea had been
supported. Following this
approach would lead us step-
by-step toward a better
understanding of nature.
Unfortunately, the life
sciences are becoming
increasingly estranged from
this way of thinking. Early in
their training, students learn
that the quest for truth needs
to be balanced against the
more immediate pressure to
"publish or perish" . For a
junior scientist to compete at
securing a permanent
academic position, her top
priority must be to publish in
journals with the greatest
prestige and impact. If she
survives to become a senior
scientist, she's likely to then
pass this lesson on to her own
PhD students.
This publishing culture is
toxic to science. Recent
studies have shown how
intense career pressures
encourage life scientists to
engage in a range of
questionable practices to
generate publications -
behaviours such as cherry-
picking data or analyses that
allow clear narratives to be
presented, reinventing the
aims of a study after it has
finished to "predict"
unexpected findings, and
failing to ensure adequate
statistical power. These are
not the actions of a small
minority; they are common,
and result from the
environment and incentive
structures that most scientists
work within.
At the same time, journals
incentivise bad practice by
favouring the publication of
results that are considered to
be positive, novel, neat and
eye-catching . In many life
sciences, negative results,
complicated results, or
attempts to replicate previous
studies never make it into the
scientific record. Instead they
occupy a vast unpublished
file drawer.
The scientific community is
well aware of these problems
- in fact, we've known about
them for decades . The
problem is that any one
scientist opting to work
beyond the system
immediately disadvantages
herself relative to her peers.
The only solution is
structural reform, and to this
end some of us have recently
taken steps to drive change.
Since May this year, the
journal Cortex, a peer-
reviewed outlet for science on
the nervous system and
behaviour, has offered
authors the opportunity to
publish a type of article
called a registered report .
Unlike traditional scientific
publishing, in which
manuscripts are peer
reviewed only after studies
have been completed,
registered reports are
reviewed before scientists
collect data. If the scientific
question and methods are
deemed sound, the authors
are then offered "in-principle
acceptance" of their article,
which virtually guarantees
publication regardless of how
the results turn out.
The journals Attention,
Perception & Psychophysics
and Perspectives on
Psychological Science have
launched similar projects .
Both initiatives borrow from
the now-established
requirement that clinical
trials pre-register their study
protocols. But these new
initiatives go even further by
offering publication of the
eventual results in the same
journal, regardless of what is
found.
By basing editorial decisions
on the question and method,
rather than the results, pre-
registration overcomes the
publication bias that blocks
negative findings from the
literature . And by conducting
peer review both before and
after a study is completed,
questionable practices to
increase "publishability" are
greatly reduced. The aim here
isn't to punish the academic
community for playing the
game that we created; rather,
we seek to change the rules of
the game itself.
Critics have argued that pre-
registration is overzealous
and will hinder exploration,
meaning serendipitous
findings would remain
hidden to us. We agree that
exploration is vital, but while
this concern is
understandable, it is also easy
to guard against. For
instance, the registered
reports initiative allows
authors to report on any
aspect of their data - even
when such analyses are not
registered at the outset.
However, these outcomes are
clearly labelled as
exploratory to make them
distinct from the pre-planned
analyses. Registered reports
also require authors to
publicly release their raw
data so that other scientists
can explore the results in
unanticipated ways, now and
in the future.
Our publishing culture is
conservative and slow to
evolve. Following the Cortex
initiative, some of us have
witnessed quiet resistance to
pre-registration from other
journals. These outlets fear
that agreeing to publish
papers before seeing the data
could lock them into
publishing negative results or
other findings conventionally
regarded as "boring". This is
despite the fact that clear-cut
negative outcomes can be
tremendously informative,
telling us which potential
interventions don't work or
which suspected phenomena
don't actually exist .
The deeper concern of
journals is that pre-
registration threatens existing
"prestige" hierarchies and
could reduce a journal's
impact factor - a metric that
is arguably meaningless as an
indicator of scientific quality
and, in fact, predicts the rate
of article retractions due to
fraud .
Nobody can expect scientists
to sacrifice their livelihoods
or those of their proteges for
the good of the cause. So, as a
group of scientists with
positions on more than 100
journal editorial boards, we
are calling for all empirical
journals in the life sciences -
including those journals that
we serve - to offer pre-
registered articles at the
earliest opportunity. The
guidelines for the initiatives
at Cortex and Perspectives are
straightforward, and while
specific kinds of studies will
require specific solutions, the
general principle is widely
applicable. For pre-
registration to benefit science
it must be embraced by a
broad spectrum of journals.
Study pre-registration doesn't
fit all forms of science, and it
isn't a cure-all for scientific
publishing. But it is a crucial
part of urgent wider reform.
Our publishing culture is no
longer fit for purpose and the
time has come to offer
scientists a genuine
alternative to "publish or
perish". If the life sciences
are to preserve a legacy of
truth, journals must welcome
pre-registration with open
arms.
