Information Technology Reference
In-Depth Information
There is likely a high correlation between submission quality and track record, so
there is likely to be much redundancy in the information the two provide. The
redundancy strengthens the argument of some peers that there is little to be gained
from assessing track record in addition to submission quality. However, it also
strengthens the argument of other peers that there is little to be gained by assessing
submission quality in addition to track record. Variations of these arguments emerge
in debates about whether research funding should support people or projects --
whether the proof of the pudding is in the eating or in the reputation of the chef. Some
funding agencies resolve such debates by requiring peers to assess both submission
quality and track record, giving weight to both (for example, Canada's Natural
Sciences and Engineering Research Council, 2012).
Still, submission quality and track record are not the same. The judged quality of a
submission depends largely on the submission's author and the peers who judge it.
Judgments of track record depend on the author's submission history and the
successes it brings. Peer reviews of a current submission are not supposed to be
influenced by the author's previous submissions. In contrast, track records are defined
by previous submissions, so the success or failure of an author's previous submissions
can influence the fate of a current one.
The cumulative effect of track record on the fate of current submissions exemplifies
a positive feedback loop amplifying the effects of chance as much as the effects of
scientific talent (Thorngte & Hotta, 1995). The chance effects can be illustrated by
considering two young scientists, Alice and Bob, who submit their first research grant
proposals to a funding agency that weighs track record more heavily than proposal
quality in choosing whom to fund. Peers judge Alice to be slightly more talented than
Bob and her proposal to be somewhat better than Bob's. Bob, however, has nine
publications, and Alice only seven, a deficit of two publications resulting from the
accidental death of her supervisor. So if the agency can fund only one of their proposals,
Bob is more likely to receive funding because Bob's has the better track record. As a
result, Bob will have more resources for increasing his scientific output than will Alice,
and thus will likely have an even more impressive track record than Alice's in any
subsequent competition, further increasing his likelihood and decreasing her likelihood
of new funding. As the future unfolds, Alice's career would likely flounder while Bob's
career would soar, not because Bob was a better scientist than Alice, but because the
funding agency could not fund both, because it weighed track record more than
submission quality, and because Alice's supervisor had the temerity to die.
With funding, perhaps Alice would have made more substantial contributions to
science than would Bob. Perhaps not. The relation between the talent of scientists and
the quality of their submissions is almost certainly imperfect, so it is quite possible
that scientists with lesser talent will sometimes produce better science than their
betters. Talent, like any psychological characteristic, expresses itself statistically in a
long run of independent opportunities. Track records break the rule of independence,
so those who, by talent or by chance, establish track records before others do are
likely to retain a career-long competitive advantage.
The tendency for early judgments of track record to snowball during a scientist's
career was recognized by Merton (1968) in his analyses of interviews with Nobel Prize
winners and diaries of other scientists. He reports, “[Nobel Prize laureates] repeatedly
observe that eminent scientists get [disproportionate] credit for their contributions to
science while [relatively] unknown scientists tend to get disproportionately little credit
