Database Reference
In-Depth Information
10.
Troubleshooting
Knowing a little about how a system works is one of the keys to troubleshooting
it. In this chapter, we will discuss some of the ways to troubleshoot Cassandra and
the tools involved.
Toolkit
Most of the tools that are used for troubleshooting are basic *nix tools. We'll go
over some of the key command-line switches and what to look for when examin-
ing a system.
iostat
iostat is a commonly used *nix tool that shows metrics about the input/output of
a system. It does not come installed out of the box on most *nix distributions. It
is commonly found as part of the
sysstat
package. The following is taken from
the
iostat
man page that comes with Linux:
The
iostat
command is used for monitoring system input/output
device loading by observing the time the devices are active in relation
to their average transfer rates.
One of the common first signs that your cluster is having I/O troubles is the
await (average wait) time. Average wait time (measured by
iostat
in milli-
seconds) is the length of time for I/O requests issued to a device to be served. This
includes the time spent by the requests in queue and the time spent servicing them.
Listing 10.1
shows what a normal but active system's device average wait time for
I/O looks like via
iostat
.
Listing 10.1
Normal
iostat
# iostat -dx 2
Device:
rrqm/s
wrqm/s
r/s
w/s rsec/
s wsec/s avgrq-sz avgqu-sz
xvdb
0.00
0.00 12.50 1.60
362.10 114.20
33.16
0.11
await
svctm
%util
