Biology Reference
In-Depth Information
isolation of others, which is particularly useful to gain insights into
the behavior of complex alignment pipelines. For instance, L¨yty-
noja and Goldman used simulated sequences to expose the system-
atic underrepresentation of the number of insertions by many
aligners, which is especially true as sequence divergence and the
number of sequences increases [
21
].
At the same time, the high level of flexibility afforded by
simulation ties in with its biggest drawback: all observations
drawn from simulated data depend on the assumptions and simpli-
fications of the model used to generate these data. The vague
notion of “realistic simulation” is often used to justify reliance on
simulations capturing relevant aspects of real data, but simulations
cannot straightforwardly, if at all, account for all evolutionary
forces. The risk thus becomes the benchmarking of MSA programs
in scenarios of little or no relevance to real biological data.
For instance, Golubchik et al. investigated the performance of six
aligners by simulating sequences in which gaps of constant size
were placed in a staggered arrangement across all sequences [
22
];
although this scenario might be useful to emphasize a more general
problem in aligning regions adjacent to gaps, its very artificial
nature makes it a poor choice to gauge the extent of that problem
on real data.
A further potential risk is the use of simulation settings more
favorable to some packages than others [
23
]. For instance, the
selected model of sequence evolution might resemble the underly-
ing model of a particular aligner and thus provide it with an
“unfair” advantage (i.e., presumably unrepresentative of typical
situations) in the evaluation. Even when the evaluation is con-
ducted in good faith, the high complexity of many MSA
aligners—particularly in terms of
implicit
assumptions
and
heuristics—can make it challenging to design a fair simulation.
3 Consistency Among Different Alignment Methods
The key idea behind consistency-based benchmarks is that different
good aligners should tend to agree on a common alignment
(namely, the correct one) whereas poor aligners might make differ-
ent kinds of mistakes, thus resulting in inconsistent alignments.
Confusingly, this notion of consistency among aligners is different
from that of consistency-based aligning, which is an alignment
strategy that favors MSAs consistent with pairwise alignments
[
24
,
25
]. In the context of benchmarking, the relevant notion is
the former—referred to by Lassmann and Sonnhammer as “inter-
consistency,” cf. “intra-consistency” for the latter [
26
].
Practically, benchmarking by consistency among aligners can be
implemented using measures such as the overlap score [
26
],
a symmetric variant of sum-of-pairs. From a set of input alignments,
