Spouts - Getting Started with Storm - page 35

Figure 4-3. Direct connection coordinator
you don't need to know anything about message emitters, and the process of adding
and removing emitters will be easier than with direct connection. The problem with
this architecture is that the queue will be your point of failure, and you'll be adding a
new layer to your processing flow.
Figure 4-4 shows the architecture schema.
Figure 4-4. Using a queue system
You can use round-robin pull or hashing queues (divide the queue mes-
sages by hash to send it to the spouts or create many queues) to paral-
lelize the processing through queues, dividing the messages between
many spouts.
You'll create an example using Redis as your queue system and their Java library,
Jedis . In this example, you'll create a log processor to collect logs from an unknown
source using the command lpush to insert messages into the queue and blpop to allow
you to wait for a message. If you have many processes, using blpop will let you receive
the messages in round-robin fashion.