Information Technology Reference
In-Depth Information
Both mathematicians and software engineering researchers have collected some
benchmarks, which have helped to advance the state of the art in the generation of
small covering arrays. For example,
•
Charlie Colbourn maintains a collection of covering arrays of strength
t
=
2
,
3
,
4
6. The sizes are the smallest; but the tables are not shown. See
http://www.
,
5
,
•
Sloane maintains a Library of Orthogonal Arrays at
•
US National Institute of Standards and Technology (NIST) also publishes some
covering arrays (
http://math.nist.gov/coveringarrays/
)
. The tables are given explic-
itly; but they are not necessarily optimal in terms of size.
•
IBM Haifa Research Lab [
13
] collected several testing problems from different
domains, such as telecommunication, health care, storage, and banking, and for
testing different aspects of the system, such as data manipulation and integrity,
protocol validation, and user interface testing. The benchmarks are available at
section, we will present a few more applications and benchmarks.
Ahmed and Zamli [
1
] selected
flex
as a case study. It is a lexical analyzer,
which was obtained from the Software-artifact Infrastructure Repository (SIR—
http://sir.unl.edu/portal/index.php
)
. Ahmed and Zamli applied variable strength cov-
ering arrays to the testing of
flex
. They considered 11 parameters, 5 of which were
exercised with higher interaction strength.
Cohen, Dwyer, and Shi [
3
,
4
] studied several nontrivial highly configurable soft-
ware systems: SPIN (model checker and simulator), the GCC compiler, Apache
HTTP server and Bugzilla.
•
SPIN [
9
,
10
] can be used as a verifier or as a simulator. The simulator has 18
factors: 13 binary options, and 5 options each with 4 different values. The verifier's
configuration model consists of 55 factors:
2
42
3
2
4
11
(
·
·
)
. This model includes a
total of 49 constraints.
After analyzing the documentation of the GCC Compiler (version 4.1),
1
Cohen,
Dwyer, and Shi constructed three different configuration models: (1) a compre-
hensive model accounting for all of the GCC options, which has 1462 factors
and 406 constraints; (2) a model that eliminates some machine-specific options,
which has 702 factors and 213 constraints; and (3) a model that focuses on the
machine-independent optimizer of GCC, which has 199 factors and 40 constraints.
•
•
Apache HTTP server is a popular web server. Cohen, Dwyer, and Shi studied the
documentation for the server (version 2.2) and established a combinatorial testing
model, which has 172 factors:
2
158
3
8
4
4
5
1
6
1
(
·
·
·
·
)
. They also identified seven
constraints.
Search WWH ::
Custom Search