The Testing Environment - Software Testing: Testing Across the Entire Software Development Life Cycle - page 151

10.2 SIMULATIONS
Simulations have been used in development shops for at least 40 years. The idea
is simple but the activity is complex and work intensive. Simulations are based
on the correct assumption that the behavior of hardware is measurable and pre-
dictable. Simulations are mathematical models of hardware behavior that can be
linked together to approximate the production environment of a new application
or system under ideal conditions . The argument for using simulation predictions
is that they are much less expensive than acquiring the hardware just for testing
purposes. This assertion tends to discount the enormous effort necessary to col-
lect the detailed data needed to make the simulation results better than an expe-
rienced systems programmer's best guess. How many corporate executives would
be willing to risk a $100 million revenue forecast based on a programmer's best
guess? If you hear someone behind you in class mumble “none,” then you have just
experienced why simulation as a testing technique fell into disfavor with corporate
IT departments.
In the last 5 years, simulations have experienced resurgence in popularity in
the role of design validation. Simulations are now used to identify and discard
designs that truly will not meet the design goals, allowing designers to focus on
the designs that have the best prospects (not guarantees) of meeting the design
goals.
10.3 BENCHMARKING
Benchmarking techniques have existed about as long as simulations. Like
simulations, they became very popular in the 1960s. Also like simulations,
they fell into disuse. Unlike simulations, they have not experienced a real re-
surgence. A benchmark is the execution of a specific application or system
under very controlled circumstances that are thoroughly measured and well
documented.
The original impetus behind benchmarks was to have a public comparison of
products for prospective buyers. For example, if you are interested in buying an
industrial strength database management system, you might shop DB2, Oracle, and
Sybase. If the brochure features of all three database management systems meet your
purchase criteria, then you look for benchmark results from each vendor that refl ect
circumstances closest to the way you intend to use the product. The more dissimilar
the benchmark circumstances are from your needs, the less valuable the benchmark
results are in comparison shopping.
The benchmark owes its popularity in part to the fact that the benchmark cost
is not born by the prospective customer. It is part of the vendor's marketing cost.
Unfortunately, public benchmarks are seldom run exactly as the customer intends
to use the product; furthermore, if the new application or system is not off-the-shelf
software, then its benchmark is probably not cost justifi ed.