Hardware Reference
In-Depth Information
In the first chapter of his topic "Software Testing: A Craftsman Approach" Jorgensen
[17] reviewed the TC concept, noting that the TC was the key to the success of the test-
ing process. He distinguished between TCs identified by the functional requirements
(functional testing) and TCs identified by the software structure (structural testing).
In an attempt at identifying "what is a good test case?" Kaner [18] maintained that
a good TC was one that gave the required
information
which was the objective of the
particular test. He counted several testing objectives each requiring a different type of
TCs, and acknowledged that TCs greatly vary and hence using them as metrics is
problematic: "Also, under this definition, the metrics that report the number of test
cases are meaningless. What do you do with a set of 20 single-variable tests that were
interesting a few weeks ago but now should be retired or merged into a combination?
Suppose you create a combination test that includes the 20 tests. Should the metric
report this one test, twenty tests, or twenty one?" (p. 2).
Later works by Grindal and Colleagues [19, 20] included a review of mechanisms
to render software testing more efficient and effective, heavily relying on TC selec-
tion and execution, since they maintained that testing is "loosely considered to be the
dynamic execution of test cases" [19, p. 2]. An interesting approach has been adopted
by the aerospace industry where the Conformance and Fault Injection (CoFI) method-
ology has been used [21, 22]. Under this methodology, TCs were differentiated be-
tween those that aim at confirming the appropriate behavior of the tested product and
those that are aimed at creating faulty situations. The authors suggested a structured
approach to the definition of the two types of testing, and as a result, a systematic
creation of the relevant TCs.
Because of the centrality of the TC in the testing process, and due to the significant
effort invested in designing and generating TCs especially in large or complex pro-
jects, several studies have elaborated on TC management processes and tools. For
example, Desai [23] from Bell Laboratories described a tool which managed the con-
figuration and inventory of TCs separately from the testing tasks, compatible with the
IEEE 829 standard. A later work described a TC management and tracking tool,
where the term 'test item' is used in a context similar to TC [24], making the TC con-
cept even more ambiguous in the absence of a formal definition. The need to auto-
mate the generation and management of TCs was demonstrated in Jorgensen's [25]
work, where he noted that it took 141,306 TCs to test version 5.0.1 of Acrobat
Reader. It is noteworthy that Jorgensen did not define a TC unit in this work as the
basis for the counting method although the term was extensively used in this com-
mentary.
The likely variability among TCs has been acknowledged by Nagappan [26] who
developed the Software Testing and Reliability Early Warning (STREW) metric suite
for software testing effort estimation, using TCs as one of the model metrics. He
warned, however, that using TCs as a metric might not be well defined since "….one
developer might write fewer test cases each with multiple asserts checking various
conditions. Another developer might test the same conditions by writing many more
test cases, each with only one assert" (p. 39). This variability among TCs should be
taken into account when defining effort estimation model parameters. Table 1 shows
that TCs can greatly vary, for example by complexity, size (whether containing many
asserts or one assert), or by origin (requirements or other), hence cannot be unified as
indicating a singular metric.
