Hardware Reference
In-Depth Information
Table 5.
TC definition distribution by research context
Context
Definition category
Process
States
Contract
Other
Defined
Cost/ROI
6
1
1
3
11
Management / Project
11
4
1
3
19
OO/ MBT/UML/SOA
12
6
2
6
26
Measurements/Metrics
1
1
1
1
4
Automation/Regression
10
1
1
2
14
Total
18
11
2
8
38
The lack of formal TC definition and the fact that most studies do not include any
definition raise several questions: Is such a definition required? Are there deficiencies in
the existing definitions? What are the implications of the lack of a formal definition?
We maintain that a formal definition is indeed required, encouraged by the fact that
in real-world testing of life-threatening projects a formal definition is an important
part of the testing guidelines. For example, based on the IEEE standard, chapter 6 of a
manual for testing safety applications in a nuclear reactor environment greatly elabo-
rates on TC types, definitions, content, and documentation [64]
.
Four types of TCs are
specified: 1) verification TC, 2) validation TC, 3) demonstration TC, 4) general suit-
ability TC. Each TC is defined by a general description including reference number,
geometry, flow features, experimental data, existing simulations, related experiments,
and rating of the challenge the test case poses. These details should be accompanied
by further documentation describing the test environment for each TC.
It is suggested that a formal TC definition could render several benefits if satisfy-
ing at least four requirements: 1) Unambiguousness: such TCs would be uniformly
understood by the various stakeholders participating in a testing endeavor, 2) Gener-
alizability: TCs would hold upon transforming from one platform to another, from
one testing domain to another, and so on, 3) Quantifiability: only quantifiable TCs
would be sensibly measured, and 4) Automatability: some might argue that this trait is
an outcome of the above three characteristics, yet we chose to explicitly indicate it as
a desirable feature because of its importance.
Unambiguousness
ensures a unified view shared by all professionals involved in
software testing regardless of their prior experience, background, testing environ-
ments, methods and techniques. This trait is important because it will ease the current
'Tower of Babylon' dominating the testing world, and rive sharing expertise among
various testing schools and perceptions.
Generalizability
ensures maintaining testing
assets and investments along various testing efforts, namely, TC generation tools and
techniques would be valid in different testing environments.
Quantifiability
is clearly
beneficial because of the importance of the TC as a fundamental metric. Currently,
measurements involving counting TCs are clearly inconsistent. Finally, there is no
need to explain the benefits rendered by the ability to
automate
TC generation, execu-
tion and management. Several attributes are mandatory for TC automation, among
them is a formal definition of the TC structure.
Examining the existing definitions by the four categories illustrates the deficiencies
in each type. The
input-process-output-objective
definitions are generally unambiguous,
