Design Patterns for Resiliency - The Practice of Cloud System Administration - page 139

timeline of actions that should be undertaken to mitigate problems or extend the deadlines.
Low-priority updates might become a high priority after a certain amount of time. If many
systems are turned off due to load shedding, it might be possible to enable them, one at a
time, to let each catch up.
To be able to manage such situations one must have visibility into the system so that
prioritization decisions can be made. For example, knowing the age of a task (how long it
has been delayed), predicting how long it will take to process, and indicating how close it
is to a deadline will permit operations personnel to gauge when delayed items should be
continued.
Delayed Work Reduces Quality
AnoldversionofGoogleWebSearchhadtwoparts:theuser-facingwebfront-end
and the system that received and processed updates to the search index (corpus).
These updates arrived in large chunks that had to be distributed to each frontend.
Thequalityofthesearchsystemwasmeasuredintermsofhowfreshthecorpus
was across all the web frontends.
The monitoring dashboard displayed the freshness of shards in each frontend.
It listed how many shards were in each freshness bucket: up-to-date, 1 hour old,
2 hours old, 4 hours old, and so on. With this visibility, operations staff could see
whensomethingwaswrongandgainaninsightintowhichfrontendswerethemost
out of date.
If the system was overloaded, the updater system was paused to free up re-
sourcesforhandlingtheadditionalload.Thedashboardenabledoperationsstaffto
understand the effects of the pause. They could unpause high-priority updates to
maintain at least minimal freshness.
6.7.2 DoS and DDoS Attacks
A denial-of-service (DoS) attack is an attempt to bring down a service by sending a large
volume of queries. A distributed denial-of-service (DDoS) attack occurs when many
computersaroundtheInternetareusedinacoordinatedfashiontocreateanextremelylarge
DoS attack. DDoS attacks are commonly initiated from botnets , which are large collec-
tions of computers around the world that have been successfully infiltrated and are now
controlled centrally, without the knowledge of their owners.
Blocking the requests is usually not a successful defense against a DDoS attack. At-
tackers can forge packets in a way that obscures where the attack is coming from, thereby
making it impossible for you to construct filters that would block the attack without block-