Hardware Reference
In-Depth Information
Table 2.
Number of tests is used as a metric for testing monitoring and control [46]
Project online trade
Date: 5/23/2007
Feature
tested
Total
Tested
#
Complete
%
Complete
#
Success
%
Success
Open Acct 46 46 100 41 89
Sell Order 36 25 69 25 69
Buy Order 19 17 89 12 63
……
Totals 395 320 81 311 79
Similarly, IBM published reporting metrics for testing the software developed by
various vendors under IBM's supervision for the Sydney Olympic Games, all based
on counting number of TCs [49]: 1) Number of test cases defined, 2) Number of test
cases executed, 3) Number of test cases with failures but no associated defect records
4) The percentage of test cases attempted, used as an indicator of progress relative to
the completeness of the planned test effort.
TCs has also been used for testing effort estimation in few works where overall
project effort has been estimated based on distinctive estimation of the various devel-
opment phases [50-52]. In an attempt to overcome the problem of counting TCs of
various size and complexity Nageswaran [53] suggested using function points where
the number of TCs can be determined by the function points estimate for the corre-
sponding effort. Following this approach Aranha & Borba [54] presented a scheme
for collecting execution points for calculating and estimating testing efforts. It should
be noted, however, that none of these works formally defined the TC term although.
Evidently, TCs have been used as metrics for testing effort estimation, as well as
for testing monitoring and control. Common to most of the techniques suggested in
these works is the reliance on counting TCs, with only minimal reference to the fact
that TCs lack a standard definition and tend to greatly differ.
4.4 Test Case Definitions
As stated earlier, a thorough literature survey has been conducted in order to study
where and how TCs are defined. Interestingly, in spite of a plethora of research about
software quality assurance, few works formally define a TC, although most use this
term quite intensively. Perhaps most notable is the fact that an explicit definition of a
TC could not be located in the 2004 version of SWEBOK. Rather, the TC appears as
an integral part of the general software testing definition:"
ï
Software testing consists of
the dynamic verification of the behavior of a program on a finite set of test cases,
suitably selected from the usually infinite executions domain, against the expected
behavior" [55, p. 5-1]. Nonetheless, several definitions have been retrieved, classified
into four dominant approaches: 1) input-process-output-objectives, 2) states and tran-
sitions, 3) contractual approach, and 4) other definitions.
The
input-process-output-objectives
perspective conceptualizes a TC as a set of in-
puts into a pre-defined process, aimed at yielding a desired output, based on the test
