Information Technology Reference
In-Depth Information
6
Conclusions
In this paper, we have addressed the estimation of statistical properties. We have pre-
sented two approaches: one for classification if a development of observed p values is
expected to lead to a statistical significant result and another one for the prediction of
needed sample sizes, also by taking into account previous samples.
The comparison of the significance classifier with a threshold-based classification
leads to significantly better results in most cases. Especially in the experiments with
randomly generated distributions, a better performance could be observed. For sam-
ples where the mean values of the distributions are not too close, high classification
accuracies (almost 90%) can be reached even if only five p values are used.
The experiments with the replication prediction do not exhibit that clear results. The
power-based predictor leads to lower average error rates for the setting with a greater
difference of the mean values as well as in the cases where many p values are used. In
some settings, the regression-based approach leads to better results, e.g., if only 5 or 10
p values are used for the closer distribution pairs.
It should be at least mentioned that the approaches presented here - multiple sta-
tistical tests with increasing sample sizes - are violating regular statistical procedures
where the setting should be clear before experiments are performed and multiple tests
with the same data should be avoided or at least taken into account by using adapted
significance levels. For exploration-based studies such approaches might be acceptable
in order to filter out certain variants or if one is aware of the statistical statement.
The current significance classifier uses a rather small set of straight-forward features.
It would be interesting to investigate if further features can lead to an improvement of
the classifier's accuracy. The prediction of the needed number of replications has not
been addressed deeply within this study. In this case, an investigation of further sta-
tistical or time series prediction methods should be performed. Further experiments are
needed in order to make statements in what situations adequate results are expected. An-
other topic for future work is the application of the approaches to simulation systems.
In this context, relevant research questions are how the approaches perform if other dis-
tributions (than normal distributions) are present and what the underlying distributions
of certain observation variables of simulation models are.
Acknowledgements.
The content of this paper is a partial result of the AssistSim
project (Hessen Agentur Project No.: 185/09-15) which is funded by the European
Union (European Regional Development Fund - ERDF) as well as the German State
Hesse in context of the
Hessen ModellProjekte
. We would like to thank our AssistSim
project
partners
for
interesting
discussions
on
the
automation
of
simulation
experiments.
References
1. Bianchi, L., Dorigo, M., Gambardella, L.M., Gutjahr, W.J.: A survey on metaheuristics for
stochastic combinatorial optimization. Natural Computing: An International Journal 8(2),
239-287 (2009)
