Database Reference
In-Depth Information
drastically impair the performance of Cassandra and should therefore be avoided
when possible.
Clock Drift
Clock drift refers to the phenomenon where one clock does not run at the exact
same speed as another clock. It is especially important to be aware of this if you
are running in a virtualized environment as drift from the hypervisor can be much
more prevalent than on regular iron. The system clock is incredibly important to
Cassandra's write and reconciliation architecture. Most writes are serialized by
timestamp. In other words, if two writes come in for the same column at almost
the same time, the determining factor for which value wins is which timestamp is
higher. If the system clocks in the ring are not all in sync, you are probably going
to see some really strange behavior.
One of the ways to deal with that is to monitor the clock drift using NTP. NTP,
or Network Time Protocol, is the most commonly used time synchronization sys-
tem on the Internet. It also comes with a binary for telling you the offset (drift)
from its synchronizing time server. You obviously want to minimize the amount
of drift your system experiences. But there will invariably be some that you have
to deal with. Monitoring is the way you know if the NTP daemon isn't doing the
job it is supposed to be doing and keeping your clocks in sync. Being alerted to
a problem with the clocks in a distributed environment that relies heavily on time
for decision making could save a lot of time tracking down weird problems later
on.
Ping Times
It is also a good idea to check the ping time responses from each of the Cassandra
nodes being monitored. There are any number of reasons that these responses can
begin to come back slowly. A few examples include the following:
A machine that is doing too much work and running short of CPU cycles
to respond quickly
I/O saturation, too high an await (average wait) time, and the machine
cannot respond quickly to the request
Network saturation due to unthrottled streaming on a high-speed network
link
Whatever the reason is, it is good to know if there is network congestion of
which you should be aware. When a node is slow to receive packets (which is the
Search WWH ::




Custom Search