Databases Reference
In-Depth Information
Figure 4-2. Direct connection hashing
In the previous example, you connected the spout to a well-known device. You can use
the same approach to connect to unknown devices using a coordinating system to
maintain the device list. The coordinator detects changes to the list and creates and
destroys connections. For example, when collecting log files from web servers, the list
of web servers may change over time. When a web server is added, the coordinator
detects the change and creates a new spout for it. See
Figure 4-3
.
It's recommended to create connections from spouts to message emit-
ters, rather than the other way around. If the machine on which a spout
is running goes down, Storm will restart it on another machine, so it's
easier for the spout to locate the message emitter than for the message
emitter to keep track of which machine the spout is on.
Enqueued Messages
The second approach is to connect your spouts to a
queue system
that will receive the
messages from the message emitters and will leave the messages available for con-
sumption by the spouts. The advantage of using a queue system is that it can serve as
middleware between the spouts and data source; in many cases, you can use the queue
to be reliable using the capability of replay messages of many queue systems. This means
