Information Technology Reference
In-Depth Information
SMock — A Test Platform for Monitoring Tools
Christian Colombo, Ruth Mizzi, and Gordon J. Pace
Department of Computer Science, University of Malta
{christian.colombo,rmiz0015,gordon.pace}@um.edu.mt
Abstract.
In the absence of a test framework for runtime verification tools, the
evaluation and testing of such tools is an onerous task. In this paper we present the
tool SMock; an easily and highly configurable mock system based on a domain-
specific language providing profiling reports and enabling behaviour replayabil-
ity, and specifically built to support the testing and evaluation of runtime verifi-
cation tools.
1
Introduction
Two of the major challenges in runtime verification, which are crucial for its adoption
in industry, are those of the management of overheads induced through the monitoring
and the ensuring the correctness of the reported results. State-of-the-art runtime verifi-
cation tools such as Java-MOP [7] and tracematches [1] have been tested on the DaCapo
benchmark
1
, but the kind of properties in these case studies were typically rather low
level, contrasting with our experience with industrial partners who are more interested
in checking business logic properties (see e.g., [4,3]). Whilst we had the chance to test
our tool Larva [5] on industrial case studies, such case studies are usually available
for small periods of time and in limited ways due to confidentiality concerns. Relying
solely on such case studies can be detrimental for the development of new tools which
need substantial testing and analysis before being of any use.
To address this lack, we have built a configurable framework which may be used to
mock transaction
2
systems under di
erent loads and usage patterns. The key feature of
the framework is the ease with which one can define di
ff
erent types of transactions and
farm out concurrent instances of such transactions through user-specified distributions.
Profiling the overheads induced by di
ff
erent runtime verification tools, thus, enables
easier benchmarking and testing for correctness of di
ff
ff
erent tools and techniques under
di
erent environment conditions. Although not a replacement of industrial case studies,
this enables better scientific evaluation of runtime verification systems.
SMock allows straightforward scripting of case studies, giving control over trans-
action behaviour, timing issues, load buildup, usage patterns, etc., which can be used
to benchmark the correctness and performance of di
ff
erent runtime verification tools.
A major issue SMock attempts to address is that of repeatability of the experiments,
ff
The research work disclosed in this publication has been partially funded by the University of
Malta Research Fund 2012, grant number CPSRP04-02.
1
http://www.dacapobench.org/
2
We use
transaction
to refer to an independent, highly replicated unit of processing.
