Database Reference
In-Depth Information
Health Check
There are a few basic things that you'll want to look for to ensure that nodes in your cluster are
healthy:
▪ Check the MBean for
org.apache.cassandra.concurrent.ROW_MUTATION_STAGE
.
You're looking to see that the completed task counts are increasing, which means that writes
(inserts, updates, and deletes) are coming through and are completing successfully.
▪ Check the MBean for
org.apache.cassandra.concurrent.ROW_READ_STAGE
. For this,
you're also ensuring that the completed task counts are increasing, which means that read
operations are coming through and are completing successfully.
▪ Make sure for both of these MBeans that the number in the
PendingTasks
attribute is not
growing too much. Even if it's not a very low number, it should be relatively stable. A grow-
ing set of pending tasks means that Cassandra is having trouble keeping up with the load. As
is usual with databases, once this problem starts, it will only continue in a downward spiral.
Three things that can improve this situation are a decreased load, scaling up (adding more
hardware), or scaling out (adding another node and rebalancing).
▪ As usual, check the logs to make sure that nothing is reporting at
ERROR
level.
Summary
In this chapter we looked at ways you can monitor and manage your Cassandra cluster. In par-
ticular, we went into some detail on JMX and learned the rich variety of operations Cassandra
makes available to the MBean server. We saw how to use JConsole to view what's happening in
your Cassandra cluster, and we performed basic maintenance tasks. Using just a little Java code,
we saw how you could easily use MBeans yourself to JMX-enable a new aspect of the source
code if you wish.
We also gained some insight into how to use tools that come with the Java Development Kit to
help us understand the runtime object graph within Cassandra. Using JMap and the Java Heap
Analysis Tool, we took a snapshot of the program's memory and then queried it using Object
Query Language to find objects with properties we might be concerned about.
There are a variety of other operations monitoring tools that are more robust, but this will get
you started. Your organization might use one already that you can hook into, such as OpenNMS
(see
http://www.opennms.org
), which has JMX hooks. Nagios, for example, is open source, free,
and rather straightforward. It also has the advantage of readily connecting to JMX, and it's what
they use at Mahalo to monitor their Cassandra ring.
