Information Technology Reference
In-Depth Information
each author's track record according to the formula in Equation 1. The result was a
single number for each manuscript+author pair representing the pair's overall
merit
.
In order to examine the effects of changing the weights given to manuscript quality
versus track record on the three outcomes previously listed, we systematically
changed the values of
wmq
and
wtr
(= 1 -
wmq
) in Equation 1. In different simulation
runs, weight
wmq
was set to 1.0 (no weight given to track record), 0.9, 0.8, 0.7, 0.6,
…, and 0.1 (almost all weight given to track record).
Once overall merit was calculated, the simulation proceeded to rank order all 100
merit scores. It then selected the manuscripts with the
npub
(20 versus 40) highest-
ranked merit scores to be “published.” The publication counters of the
npub
published
authors were incremented by 1 to signify an increase of 1 in their track record.
The publication counters of the 100-
npub
unpublished authors were, of course, not
incremented.
To illustrate the simulation process, consider again Alice and Bob. Suppose Bob,
assigned a talent score of 80, submits a manuscript with a true quality score of 73 to
the first issue of Journal J. Similarly, suppose Alice, assigned a talent score of 83,
submits a manuscript with true quality score of 76 for the first issue of Journal J.
Reviewer X gives Bob's manuscript a judged quality score of 79 and Reviewer Y
gives it a score of 71 for an average of 75. Reviewer X gives Alice's manuscript a
judged quality score of 70 and Y gives it a score of 76 for an average of 73. Because
Alice and Bob start with no publications, any weight given to track record does not
effect their merit scores. If the editor of J had room to publish both manuscripts, then
Alice and Bob would both add one publication to their track record. But if the editor
could publish only one of their manuscripts, then Bob's higher-judged manuscript
would prevail. Bob would then increase his/her track record from zero to one
publication while Alice, despite superior talent and a better true-quality manuscript,
would not.
The resulting difference of one publication in track record could compound itself
in subsequent outcomes. Suppose Alice and Bob each submit a new manuscript for
the second issue of Journal J. This time Reviewers X and Y both judge Alice's new
manuscript to be better than Bob's, so the judged-quality ranking of her manuscript is
higher than his. But, thanks to his publication in the first issue of J, Bob's track record
is now higher than Alice's. Again, if the editor had room for both manuscripts, then
Alice and Bob would both improve their track record. But if the editor had room for
only one of their new manuscripts, a quality-versus-track-record ranking conflict
would ensue. Giving more weight to manuscript quality would lead to Alice's first
publication, equaling her track record with Bob's. Giving more weight to track record
would put Bob's track record two publications ahead of Alice's, again preventing the
second of Alice's superior true-quality manuscript from being published, and making
him even more likely than Alice to be published in the remaining competitions.
In summary, the simulation attempted to determine the outcomes of the seemingly
complex interactions among resource scarcity (journal space), chance (imperfect
correlations between talent, true manuscript quality, and judged quality), and editorial
decision rule (weights given to judged quality versus track record) as journals
repeated their publication competitions. We did so to learn under what conditions the
use of reputation information might improve the average quality of publications and
the track record of those who author them.
