Information Technology Reference
In-Depth Information
The paper is structured as follows: In Section 2, we discuss some approaches related
to ours. The context of the work and the framework of automated operation and control
of simulation experiments is presented in Section 3. In Section 4 we introduce our
approach to significance estimation. Experimental results are presented in Section 5. A
conclusion as well as ideas for further works are discussed in Section 6.
2
Related Work
The automation of (simulation) experiments as well as the application of data
mining approaches to experimental settings and results has been addressed by vari-
ous researchers. Explora is a knowledge discovery assistant system for multipattern
and multistrategy discovery (e.g., [8,9]). Klosgen lists four analysis tasks that can be
aimed at in such a setting [8]: single-variant analysis (e.g., influence of predefined fac-
tors on output variables), comparison of variants, analysis of whole space of variants,
and optimization. Klosgen reports that the discovery approach “can constitute a valu-
able approach also in an area where the analyst has already a lot of knowledge on
the domain”. Referring to Klosgen three paradigms are fundamental in order to sup-
port data exploration: search, visualization, and navigation, and KDD should combine
these three paradigms in a semi-automatic process [9]. The Explora system “constructs
hierarchical spaces of hypotheses, organizes and controls the search for interesting in-
stances in these spaces, verifies and evaluates the instances in data, and supports the
presentation and management of the discovery findings” [9, p. 250]. Different facets of
interestingness are also discussed in this paper: evidence, redundancy, usefulness, nov-
elty, simplicity, and generality. The application of Explora to simulation experiments in
practical political planning is presented in [8].
King et al. [6] address the “automation of science”; they present the development of
the robot scientist “Adam” who autonomously generates functional genomics hypothe-
ses and tests these hypotheses using laboratory automation. An ontology and logical
language has been developed to describe the research performed by the robot. The au-
tomated conclusions have been confirmed through manually performed experiments.
In earlier work, King et al. present genomic hypothesis generation with their “robot
scientist” [7]. Experiments and hypothesis generation are performed in a loop where
experimental results are evaluated and machine learning (with access to background
knowledge) is applied. The output of this step is used in order to select experiments for
the next cycle.
Huber et al. apply decision tree learning (ID3) in order to extract knowledge from
simulation runs in model optimization [5]. They set up a classification task where the
relation between input and output of simulation runs is learned. The result of the learn-
ing phase is a decision tree indicating which attributes are important and what attribute
values lead to “good” or “bad” behavior. In their paper, they apply the approach to find
the range of configuration and workload parameters to optimize the performance for
a multiprocessor system. Referring to Huber et al. this qualitative information of the
system behavior can be helpful for interpretation of the optimization results.
Burl et al. [2] present an approach to automated knowledge discovery from simu-
lators. They address the “landscape characterization problem” with the aim to identify
