Information Technology Reference
In-Depth Information
By the Numbers: Track Record, Flawed Reviews,
Journal Space, and the Fate of Talented Authors
Warren Thorngate and Wahida Chowdhury
Psychology Department and Institute of Cognitive Science
Carleton University, Ottawa, Canada
warren_thorngate@carleton.ca,
WahidaChowdhury@cmail.carleton.ca
Abstract.
We conducted a computer simulation of hundreds of competitions for
limited journal space, varying (a) the correlation between the talent of authors
and the quality of their manuscripts, (b) the correlation between manuscript
quality and quality judged by peer reviewers, (c) the weights reviewers and
editors gave judged quality versus number of previous publications (tract
record), and (d) the proportion of manuscripts accepted for publication. The
results show that even small decreases in the correlations, and small increases in
the weight given to track record, quickly skew the outcomes of the peer review
process, favouring authors who develop a track record of publications in the
first cycles of journal publication while excluding many equally-talented or
more-talented authors from publishing (the
Matthew Effect
; Merton, 1968).
Implications for declines in the quality of published manuscripts and for
wasting talent are discussed.
Keywords:
peer review, reputation, track record, competition.
1
Introduction
Scientists frequently submit manuscripts for publication; grant proposals for funding,
applications for jobs, tenure, promotion, sabbaticals, fellowships and other limited
resources. Their submissions are usually assessed by other scientists as part of the
venerated tradition called
peer review
. The assessments are important; many scientific
careers are made or broken by the outcomes of competitions for these resources
(Thorngate, Dawes, & Foddy, 2009; Thorngate, Liu & Chowdhury, 2011).
Peers can employ many kinds of information in making their assessments. Some
information is intrinsic to the submission: the freshness of its ideas, the relevance of
its literature, logical derivation of its predictions, soundness of its research methods,
cogency, etc. Other information is extrinsic to the submission, including information
about the reputation of its author. Reputation is often induced from a list of the
author's previous scientific achievements (a curriculum vita), also known as the
author's
track record
. The primary purpose of the present study was to determine how
the outcomes of competitions for journal space are influenced by the relative
importance peers attach to submission quality versus track record.
