Information Technology Reference
In-Depth Information
tests of subjects should use sufficiently large samples and appropriate controls. Com-
parison to existing work is an important part of the demonstration of validity. A new
algorithm that is inferior to existing alternatives is unlikely to be significant.
Evaluation of Papers
The process of evaluating a paper involves asking critical questions such as those
listed under “Critical reading” earlier in this chapter. In addition, there are further
questions that should be asked of a paper that is under review, which should not just
be correct but should be suitable for the likely audience.
Is the contribution timely or only of historical interest?
Is the topic relevant to the venue's typical readership?
What is missing? What would complete the presentation? Is any of the material
unnecessary?
How broad is the likely readership?
Can the paper be understood? Is it clearlywritten? Is the presentation at an adequate
standard?
Does the content justify the length?
Of these, contribution is the single most significant component, and requires a value
judgement. The presence of a critical analysis is also important: authors should
correctly identify the strengths, weaknesses, and implications of their work, and not
ignore problems or shortcomings. It is easier to trust results when they are described
in a balanced way.
Most papers have an explicit or implicit hypothesis—some assertion that is
claimed to be true—and a proposal or innovation. Try to identify what the hypothesis
is: if you can't identify it, there is probably something wrong, and if you can, it will
help you to recognize whether all of the paper is pertinent to the hypothesis, and
whether important material is missing.
The quality of a paper can be reflected in its bibliography. For example, how
many references are there? This is a crude rule-of-thumb, but often effective. For
some research problems there are only a few relevant papers, but such cases are the
exception. The presence of only a few references may be evidence of bad scholarship.
Also, some authors cite a reasonable number of papers without actually citing related
literature, thus disguising a core bibliography that is far too short. If many of the
references are by the author, it may be that some of them are redundant. If only a
couple of the references are recent, the author doesn't appear to be familiar with
other research. Similarly, be suspicious of papers with no references to the major
journals or conferences in the area. Expect author-provided evidence of novelty and
innovation, via the right citations.
Occasionally an author submits a paper that is seriously incomplete. No effort has
been made to find relevant literature, or the proofs are only sketched, or it is clear
that the paper has never been proofread, or, in an extreme case, the paper does little
