Database Reference
In-Depth Information
This basic form of Paxos is concerned with updating, essentially, the entire
state. It can be used to update individual elements of a distributed state
representation, but this tends to introduce a lot of overhead leading to the
development of Multi-Paxos.
The Multi-Paxos is often the algorithm that is actually implemented in the
real world. It depends on the fact that processes are relatively stable and
that a change in the Leader of a cluster is going to be rare. This allows
the algorithm to dispense with the Prepare-Promise phase of the
communication protocol after a process has been declared a Leader. When a
new Leader is elected it must repeat the Prepare-Promise cycle again, so the
worst-case scenario is the basic Paxos algorithm.
Althoughthesealgorithms appear toberelatively straightforward, theyhave
proven notoriously difficult to implement properly. Thus, it is
recommended to use systems like those described in this chapter rather
than attempting an implementation. This chapter introduces two of these
systems: ZooKeeper and etcd. The ZooKeeper project is a distributed
configuration and coordination tool used to coordinate many of the other
distributed systems found in this topic. Etcd is a new system that provides
many of the same features of ZooKeeper through an HTTP-based interface
and using a new consensus algorithm called Raft, which was designed
explicitly to deal with the complexities of Paxos.
Apache ZooKeeper
The ZooKeeper project was developed at Yahoo! with the mission of
providing exactly the primitives described in the last section. Originally
developed to tame the menagerie of services being implemented in Yahoo!'s
internal Hadoop environments, it has been open-sourced and contributed
to the Apache Foundation.
It has since proven itself to be a valuable infrastructural component for the
development of distributed systems. In fact, it is used to coordinate many
of the applications used later in this topic. The Kafka data motion system
uses it to manage metadata for both its servers and clients. It is also a key
component of the Storm data processing framework's server management
features and plays a similar role for the Samza data processing framework.
Rather than imposing a particular set of coordination or configuration
features, ZooKeeper provides a low-level API for implementing
