Web Testing for Reliability Improvement - Advances in Computers - page 199

There are three generic methods for information gathering and OP construction, in
decreasing accuracy: actual measurement of usage at customer installations, survey
of target customers, and usage estimation based on expert opinions . Fortunately, the
availability of existing web logs offers us the opportunity to collect usage information
for OP construction without incurring much additional cost. For new web sites or new
web-based applications, similar information about the “intended” customer usage
can be obtained by surveying potential customers or from experts.
Once an OP is constructed, it can be used to support statistical testing by some
random sampling procedure to select test cases according to the probability distrib-
ution and execute them. Essentially, each operation in the OP corresponds to certain
test cases specifically constructed or selected from existing ones to test a specific
system operation.
The actual test runs are sampled from these test cases according to the probability
of associated operations. The number of test runs for each operation in the OP is pro-
portional to its probability. Under most circumstances, these test cases and associated
runs can be prepared ahead of time, so that some test procedure can be employed to
sequence the multiple test runs according to various criteria. In some cases, truly
random sampling can be used, to dynamically select test cases to run next. How-
ever, such dynamic random sampling will slow down test execution and the system
performance because of the overhead involved in managing the test case selection
in addition to monitoring the test runs. Therefore, unless absolutely necessary, we
should prepare the test cases and test procedures ahead of time to reduce the impact
of testing overhead on normal system operations.
In addition to or in place of proportional sampling, progressive testing is often
used with the help of OPs. For example, at the beginning of testing, higher proba-
bility threshold can be used to select a few very important or highly used operations
for testing. As testing progresses, the threshold can be lowered to allow testing of
less frequently used operations so that a wide variety of different operations can be
covered. In a sense, the use of OPs can help us prioritize and organize our testing
effort so that important or highly used areas are tested first, and other areas are pro-
gressively tested to ensure good coverage.
5 . 3 U n i fi e d M a r k o v M o d e l s ( U M M s ) f o r U s a g e - B a s e d Te s t i n g
As mentioned in Section 4 , the primary limitation of FSM-based testing for the
web is its inability to handle large number of states or individual web pages. We
can augment FSMs with probabilistic usage information to focus our testing effort
on usage scenarios and navigation sequences commonly used by target customers,
while reduce or eliminate testing of less frequently used ones. The use of this ap-
proach would help us ensure and maximize product reliability from a customer's