Database Reference
In-Depth Information
Scaling the Storm cluster - adding new
supervisor nodes
In production, one of the most common scenarios one can run into is when the processing
need outgrows the size of the cluster. Scaling then becomes necessary; there are two op-
tions: we can perform vertical scaling wherein we can add more compute capability, or we
can use horizontal scaling where we add more nodes. The latter is more cost-effective and
also makes the cluster more robust.
Here are the steps to be executed to add a new node to the Storm cluster:
1. Download and install the 0.9.2 version of Storm as it is used in the rest of the
cluster by extracting the downloaded ZIP file.
2. Create the required directories:
sudo mkdir -p /usr/local/storm/tmp
3. All Storm nodes, the Nimbus nodes, and the supervisor require a location on to
store a small amount of data related to configurations on the local disk. Please en-
sure you create the directory and assign read/write permissions on all Storm nodes.
4. Create the required directories for the logs, as follows:
sudo mkdir -p /mnt/app_logs/storm/storm_logs
5. Update the storm.yaml file with necessary changes for Nimbus and Zookeeper:
#storm.zookeeper.servers: This is a list of the hosts
in the Zookeeper cluster for Storm cluster
storm.zookeeper.servers:
- "<IP_ADDRESS_OF_ZOOKEEPER_ENSEMBLE_NODE_1>"
- "<IP_ADDRESS_OF_ZOOKEEPER_ENSEMBLE_NODE_2>"
#storm.zookeeper.port: Port on which zookeeper cluster
is running.
storm.zookeeper.port: 2182
#For our installation, we are going to create this
directory in /usr/local/storm/tmp location.
storm.local.dir: "/usr/local/storm/tmp"
#nimbus.host: The nodes need to know which machine is
the #master in order to download topology jars and
Search WWH ::




Custom Search