Hardware Reference
In-Depth Information
3 Methodology
The following methodology has been employed in order to identify the literature
relevant for this review. First, Google and Google Scholar were used as search en-
gines to find sources with the keywords: "software testing", "software reliability",
"testing methodology", "black box testing", and "test case". This first search effort
yielded about 150 papers and about 25 topics dated 1982 onwards, which were
scanned for relevance by reading their abstracts. Looking at citations appearing in the
elicited papers implied that there is merit in further expanding the search by using the
following keywords: "TDD", "SOA", "agile", "software cost estimation", "software
project management", "testing projects", "test case generation", and "testing automa-
tion". This search yielded about 100 additional papers and about 10 topics spanning
the years 1980 till 2008.
A similar search has been conducted on leading journals and conferences, for
example relevant IEEE and EMSE journals and ICST conferences that directly or
indirectly included topics represented by the above keywords. These three rounds of
literature search resulted in a database of about 300 papers, books, and conference
proceedings. Endnote 9.0 has been used as the reference management tool, where
research notes have been added for classification purposes.
This reference database has been then reviewed, and each reference has been clas-
sified to sub-topics as in Table 4 (a paper could be related to more than one sub-
topic), as well as whether or not it included a formal definition of a TC. Those papers
which contained such a definition were further categorized based on the nature of the
definition, as appears in Table 3.
While classifying the papers, additional references and topics were searched by
scanning their reference list, which resulted in about 40 additional papers, bringing
the total number of papers and books reviewed to about 340, of which 267 directly
referred to TCs.
4 Literature Review
4.1 Historical Overview of the TC Concept
The TC concept appeared as a central concept underlying testing processes since the
beginning of formal software testing, for example as part of the Systematic Test &
Evaluation Processes (STEP) model [10], which defined feedback loops between
software development and testing. Three sources for TC generation were identified:
directly from the requirements, stemming from performance requirements, and based
on system's design [10]. A formal definition of a TC, however, was not included. In a
study published in the same year, Ostrand & Balcer
1[ suggested to build TCs as a
collection of test frames and test scripts, yet these two terms were not precisely de-
fined although TCs were perceived to be measureable by their size. Weyuker [12]
brought a quite different approach when she maintained that TCs are formed by deci-
sion statements, and recognized that the more the number of decision statements in
the tested code, the more complex is the TC, recommending to limit the average
]