Biology Reference
In-Depth Information
2.4 Evaluating Simulation Effectiveness
A crucial component of using simulation to mimic authentic data is verifying
that the simulated data retain the key characteristics of the original data.
This is done by testing whether the simulated data come from the same
distribution as the original authentic data. If they come from the same distri-
bution, then the simulation method should be trustworthy and provide valid
results; if not, then the differences between the original and simulated data
can provide distorted and unrealistic results.
Of course, given a finite amount of original data, there exist an infinite
number of distributions that could generate that data. The distribution tests
used here merely attempt to confirm that the simulation method is within
that space of possible models, specifically those that have a reasonable
chance of generating the data. We must use domain knowledge (such as our
awareness of which characteristics are relevant) to further constrain the pos-
sible simulation models. Goodness-of-fit tests of the simulated data should
be considered as relative measures of consistency; it is known that distribu-
tional tests become extremely sensitive with large amounts of data, and so
may reject even the most useful simulations.
In addition, it should be considered that a mimic method will only be
useful if it accurately captures the randomness of the underlying distribu-
tion. If a mimic is simply a duplicate of the original data, it is clearly not
a good additional test, nor does it avoid any privacy concerns. Similarly, a
mimic that merely adds random noise to the original is not providing a new
authentic set of possible data; it is simply providing the original data with
extra variation.
2.4.1 univariate
χ
2
Tests
The first method for evaluating the closeness between distribution of
authentic and mimic data is a series of simple χ
2
tests. To test a mimic
against its original dataset, we take each univariate data series and split it
by day of week. The values for a single day of week are then formed into
bins; an example of the binning process is given in Figure 2.1. The width of
the bin varies by density, determined such that there are at least 10 obser-
vations in each bin. The original data are split and binned in the same
fashion, and these two sets of counts (mimicked and original) are tested
for distributional equality using a χ
2
test (with degrees of freedom equal
to
k
− 1, where
k
= the number of bins). An FDR (Benjamini and Hochberg
1995) significance correction is used to account for multiple testing across
multiple series. The χ
2
tests can also be repeated for each DOW separately
with FDR correction, not only to inform us whether there are issues with
our simulation but also to point us toward the reasons for those issues.
