Web Testing for Reliability Improvement - Advances in Computers - page 212

modified,” therefore no need to resend or reload). Since the total number of entries
with missing bytes information represented about 15% of the total number of entries
(hits), we simply used the rest to calculate the number of bytes transferred.
Workload measurements associated with periods shorter than calendar days, such
as by the hour or by some short bursts, can also be used in reliability analysis. With
time-stamped individual hits and related information available in the access log, such
measurements can be easily obtained. As for reliability analysis, data with less vari-
ability, usually through data grouping or clustering, are generally preferable, because
they typically produce more stable models that fit the observations better and provide
better reliability assessments and predictions [41,47] . In our case, daily data have
much less variability than hourly ones, yet give us enough data points for statisti-
cal analyses and model fitting. Consequently, we only use daily data in subsequent
reliability analyses.
6 . 3 A n a l y z i n g O p e r a t i o n a l R e l i a b i l i t y
As mentioned in Section 2 , a workload profile with considerable variability, such
as the web traffic for our two web sites studied here, is a clear indication that mea-
suring failures alone over calendar time is not suitable for reliability analysis. The
number of problems encountered per day is expected to be closely related to the
usage intensity. When we combine the measurement results for the web failures, in
this case type E errors extracted from the error log, and workload measured by the
number of users, sessions, hits, and bytes transferred, we can perform analyses to
evaluate web software reliability.
As observed earlier, the peaks and valleys in errors generally coincide with the
peaks and valleys in workload. This close relationship between usage time and fail-
ure count can be graphically examined as in Fig. 10 , plotting cumulative errors vs.
cumulative bytes transferred over the observation period. An essentially linear rela-
tion can be detected between the two. Similar observations can be obtained if we plot
cumulative errors vs. cumulative hits, users, or sessions.
This relationship can also be characterized by the daily failure rate, as defined
by the number of errors divided by the workload measured by bytes transferred,
hits, users, or sessions for each day. These daily failure rates also characterize web
software reliability, and can be interpreted as applying the Nelson model [36] men-
tioned in Section 2 to daily snapshots. Table IV gives the range (min to max),
the mean, and the standard deviation (std.dev.), for each daily error rates defined
above. Because these rates are defined for different workload measurement units
and have different magnitude, we used the relative standard error, or rse , defined as:
rse = std . dev ./ mean , to compare their relative spread in Table IV . We also included
the daily error count for comparison. All these daily error rates fall into tighter spread