Hardware Reference
In-Depth Information
but not necessarily generalizable. For example, non-functional requirements, such as
testing a user experience, are difficult to define using this type of definition. Likewise,
the 'process' part of the TC can vary in size and complexity hence difficult to quantify
and measure. For instance, a process can be as simple as 'check for existence of a certain
value' or quite complex as 'create a customer order'. Consequently, this type of defini-
tion is problematic to automate. The
state & transitions
definitions may satisfy the un-
ambiguousness and quantifiability traits but are hardly generalizable since they stem
from the state-machine world, therefore not transferrable to other testing domains. For
example states and transitions that are a result of dynamic environmental conditions and
data would be rather impossible to define as a finite number of states and transitions.
TCs defined as States & Transitions, however, are quite convenient to quantify and
automate due to their origination in the state-machine domain. The
contract
group of
definitions is becoming popular, mainly in SOA platforms, yet these definitions clearly
violate the unambiguousness criterion. For example, Aichernig [16] defined a test as a
contract between the user and the software provider, Mikhailova et al. [65] defined
testing as a contract between the system under test and its environment, and Bruno et al.
[66] thought it was a contract ensuring service compliance between releases. Clearly,
only a formal definition of the contract, such as the one attempted by Aichernig [16] is
unambiguous. For similar reasons it cannot be generalized, quantifiable or automatable
unless formalized. Finally, it is quite obvious that the
other
definitions do not meet most
of the above requirements.
We maintain that the absence of a formal definition for TCs causes test planning,
execution, and monitoring malfunctioning. For example, reporting testing effort es-
timation or testing progress by number of executed TCs is clearly misleading, often
resulting in projects not meeting time and budget constraints, or in inadequate soft-
ware quality. Testing automation efforts are likewise contingent upon formal defini-
tion of TCs, hence its absence is possibly one of the barriers to a broader diffusion of
automation tools. These shortcomings are quite likely among the causes for the huge
annual economic damage as a result of inadequate software testing infrastructure and
processes reported by the US Department of Commerce [1]. Hence, further work
towards a formal TC definition that meets the above requirements is advocated.
6 Conclusions
TC is a cornerstone for planning, designing, and monitoring testing projects, as well as a
means for work, effort and cost estimation. This work demonstrated not only the cen-
trality of the TC but also the variance among TC definitions. Further, the official profes-
sional taxonomies, for example those presented in the joint ISO-IEEE Guide to the
Software Engineering Body of Knowledge - SWEBOK does not explicitly define TC.
This situation is possibly a barrier to improving the testing infrastructure leading to
higher software quality, therefore decreasing the enormous resulting damage. It is
suggested that establishing a formal, unambiguous, generic, quantifiable and struc-
tural definition for a TC would be a significant contribution to the world of software
testing, and software quality in general. Such a definition would pave the way to stan-
dard TC generation techniques, as well as to measurement and evaluation tools.
