Information Technology Reference
In-Depth Information
less time-consuming and more convenient and thus would be preferred. If the
correlations were slightly lower, say, r = +0.98, an occasional decision error could
occur, rejecting a superior manuscript while publishing an inferior one. Still, the
speed and convenience of utilizing track record might outweigh the costs of a rare
error.
However, the correlations between talent and manuscript quality, and between true
quality and judged quality, have never been shown to approach r = +1.0. At more
realistic levels, such as those used in our simulation, the deleterious effects of
weighing track record cannot be ignored. With moderate correlations found in
empirical studies (typically between r = +0.3 and r = +0.6; see Petty, Fleming &
Farbigar, 1999; Thorngate, Dawes & Foddy, 2009; Whitehurst, 1984), for example,
our simulation shows that increasing the weight given track record reduces the
proportion of the best manuscripts published and of the most-talented authors with the
best track records by 10-20%. The resulting errors of omission might not affect the
eventual progress of science, but they can slow it down and can limit or end the
careers of many talented scientists. Indeed, the errors can discourage more-talented
but less-fortunate scientists from continuing their careers, while encouraging less-
talented but more-fortunate scientists to continue publishing.
We note, however, the usual caveats. Our simulation selected a small set of factors
we believed could influence the quality of articles published and the fate of
prospective authors. Dozens of other factors are likely to have additional influences.
Such factors include opportunities for rejected authors to resubmit their manuscripts,
to find alternative journals, to submit manuscripts jointly authored by colleagues with
established track records, or to improve their talent with practice. They also include
factors indirectly related to peer review, for example, factors related to tenure,
promotion, research funding priorities, and grant success. Interactive effects of these
factors are likely to reduce the generality of our results. Conversely, some of our
results are likely to moderate these interactive effects.
The results of our simulation suggest that, when less weight is given to track
record, a higher proportion of talented authors are published, resulting in a higher
proportion of good, published articles and greater number of talented authors with
good track records. This leads to the prescription that track record should be given no
weight at all. As many have noted, however (see Allesina, 2012), reliance on
judgments of manuscript quality does not itself eliminate judgment errors, and the
judgments themselves are much more demanding than judgments of track record,
reducing the number of qualified reviewers eager to volunteer. The ends and means of
peer review are thus in some conflict.
How might the conflict be resolved? Several proposals have been made and
innovations tried. New journals, blind reviews, anonymous authorship, reviewer
training, additional reviewers, shorter manuscripts, clearer editorial policies and
explicit review criteria exemplify popular innovations. Alas, a vastly improved review
system has yet to emerge from their use. Newer proposals, including many utilizing
Internet technology and simulated with agent-based models, offer more possibilities
(see, for example, Allesina, 2012; Grimaldo & Paolucci, 2013; Herron, 2011).
