Design Patterns for Scaling - The Practice of Cloud System Administration - page 112

gram to take the next one and process it. The kernel limits how many waiting connections
are permitted, so if there is a flood of new connections, some will be dropped.
Finally, in a multi-core machine, the single thread will be bound to a single CPU, leav-
ing the other cores idle. Multithreaded code can take advantage of all cores, thereby mak-
ing maximum use of a machine.
In multithreading, a main thread receives new requests. For each request, it creates a
new thread, called a worker thread , to do the actual work and send the reply. Since thread
creation is fast, the main thread can keep up with a flood of new requests and none will
be dropped. Throughput is improved because requests are processed in parallel, multiple
CPUs are utilized, and head of line blocking is reduced or eliminated.
That said, multithreading is more difficult to implement. If multiple threads have to ac-
cess the same resources in memory, locks or signaling flags (semaphores) are needed to
prevent resource collision. Locking is complex and error prone.
There are limits to the number of threads a machine can handle, based on RAM and
CPU core limits. If any one core becomes overloaded, performance will quickly drop for
that core. Connection floods can still cause dropped requests, but the number of connec-
tions being handled is increased.
5.7 Queueing
Another way that data can be processed differently to achieve better scale is called
queuing . A queue is a data structure that holds requests until the software is ready to pro-
cess them. Most queues release elements in the order that they were received, called first
in, first out (FIFO) processing.
Queueingissimilartomultithreadinginthatthereisamasterthreadandworkerthreads.
The master thread collects requests and places them in the queue. There is usually a fixed
number of worker threads. Each one takes a request from the queue, processes it, sends a
reply, and then takes another item from the queue. This workflow is called feeding from a
queue .
5.7.1 Benefits
Queueingsharesmanyoftheadvantages anddisadvantages ofmultithreading. Atthesame
time, it has several advantages over basic multithreading.
With queueing, you are less likely to overload the machine since the number of worker
threads is fixed and remains constant. There is also an advantage in retaining the same
threads to service multiple requests. This avoids the overhead associated with new thread
creation. Thread creation is lightweight, but on a massive scale the overhead can add up.