An Overview of Large-Scale Stream Processing Engines - Large Scale and Big Data: Processing and Management - page 395

1. Local monitor (LM) runs at each site and produces a collection of local
statistics, which it forwards periodically to the end-point monitor (EM).
LM maintains various box- and site-level statistics regarding utilization and
queuing delays for various resources including CPU, disk, bandwidth, and
power (only relevant to sensor proxies).
2. Endpoint monitor (EM) runs at every site that produces Borealis outputs.
EM evaluates QoS for every output message and keeps statistics on QoS for
all outputs for the site.
In addition, there are three levels of collaborating optimizers:
1. Local optimizer runs at every site and is responsible for scheduling mes-
sages to be processed as well as deciding where in the locally running dia-
gram to shed load, if required.
2. Neighborhood optimizer runs at every site and is primarily responsible for
load balancing the resources at a site with those of its immediate neighbors.
3. Global optimizer is responsible for accepting information from the end-
point monitors and making global optimization decisions.
Monitoring components run continuously and trigger optimizer(s) when they
detect problems (e.g., resource overload) or optimization opportunities (e.g., neigh-
bor with significantly lower load). The local monitor triggers the local optimizer
or neighborhood optimizer while the end-point the monitors will trigger the global
optimizer. Each optimizer tries to resolve the situation itself. If it cannot achieve this
within a predefined period, monitors trigger the optimizer at the higher level. This
approach strives to handle problems locally when possible because in general, local
decisions are cheaper to make and realize with the added benefit of being less disrup-
tive. Another implication is that transient problems are dealt with locally, whereas
more persistent problems potentially require global intervention.
12.4 IBM SYSTEM S AND IBM SPADE
The IBM System S [3,4,20] is a large-scale, distributed data stream processing
middleware that supports structured as well as unstructured data stream process-
ing and can be scaled to a large number of compute nodes. The System S runtime
can execute a large number of long-running queries that take the form of data-flow
graphs. A data-flow graph consists of a set of processing elements (PEs) connected
by streams, where each stream carries a series of tuples. The PEs implement data
stream analytics and are basic execution containers that are distributed over the
compute nodes. The compute nodes are organized as a shared-nothing cluster of
workstations (COW). The PEs communicate with each other via their input and out-
put ports, connected by streams. The PE ports as well as the streams connecting
them are typed. PEs can be explicitly connected using hard-coded links or through
implicit links that rely on type compatibility. System S also provides several other
services, such as fault tolerance, scheduling, and placement optimization, distributed
job management, storage services, and security.