An Overview of Large-Scale Stream Processing Engines - Large Scale and Big Data: Processing and Management - page 402

• registerStream(description) -> SID : Registers an external data stream

and returns the generated internal SID for this stream.

• registerQuery(name, query, input SIDs) -> SID : Adds a query with the

parameters name, query, and a set of one or more input SIDs. It returns the
SID of the query output stream.

• registerOutput(SID, target) : Registers an output destination that receives

the data from the given SID. The target parameter contains the host address
and port of the output location.

• pushEvents(SID, events) : Pushes a set of one or more events with the given

input stream SID to the system.

Figure 12.5 illustrates the distributed stream processing in Stormy. Multiple cli-
ents can register streams, queries, and output destinations; either independently of
each other, or they may even share queries or output destinations. Stormy uses a
consistent hashing mechanism [14] to distribute queries among the available nodes
where the output range of a hash function can be treated as a fixed circular space,
or “ring.” Each node in the system gets assigned a random value that defines its
position on this ring. If a new item is inserted into the system, the hash value of its
key is calculated and the item is assigned to the first node reached by following the
ring clockwise starting at the location of the item's hash value. In other words, every
node becomes responsible for the key space between its own position and the posi-
tion of its predecessor. The mapping of ranges to nodes is maintained using a DHT
where every node knows the mapping of keys to nodes and can therefore forward an
incoming request to the responsible node. If the DHT changes, the new version of the
mapping is propagated through the system using a gossip protocol. A gossip protocol
distributes new information to all nodes in the system, typically with some time
delay. Therefore, it might happen that a node with outdated mapping information
forwards the request to the wrong node. However, the request will eventually arrive
at the correct node as the new mapping information is gossiped through the system.

Q2
Q1

Input

Input E:S1

N 1 N 2

E:S7

E:S7(S11)

E:S11

E:S2 Q4

N 4

N 3

E:S8

Q3

N 5

Output E:S12

Output

FIGURE 12.5 Execution of distributed stream processing in Stormy. (From S. Loesing et
al., Stormy: An elastic and highly available streaming service in the cloud, in EDBT/ICDT
Workshops , pages 55-60, 2012.)