Designing in a Distributed World - The Practice of Cloud System Administration - page 24

We talk with the team that operates our authentication system and they recommend
budgeting 3 ms for authentication.
Formatting the data takes very little time—less than the slop in our other estimates—so
we can ignore it.
Thisleaves147msforthemessagetoberetrievedfromstorage.Ifatypicalindexlook-
up requires 3 disk seeks (10 ms each) and reads about 1 megabyte of information (30 ms),
that is 60 ms. Reading the message itself might require 4 disk seeks and reading about 2
megabytes of information (100 ms). The total is 160 ms, which is more than our 147 ms
remaining budget.
How Did We Know That?
Howdidweknowthatitwilltake3diskseekstoreadtheindex?Itrequiresknow-
ledge of the inner workings of the UNIX file system: how files are looked up in
a directory to find an inode and how inodes are used to look up the data blocks.
This is why understanding the internals of the operating system you use is key to
being able to design and operate distributed systems. The internals of UNIX and
UNIX-likeoperatingsystemsarewelldocumented,thusgivingthemanadvantage
over other systems.
While disappointed that our design did not meet the design parameters, we are happy
that disaster has been averted. Better to know now than to find out when it is too late.
It seems like 60 ms for an index lookup is a long time. We could improve that consid-
erably. What if the index was held in RAM? Is this possible? Some quick calculations es-
timate that the lookup tree would have to be 3 levels deep to fan out to enough machines
to span this much data. To go up and down the tree is 5 packets, or about 2.5 ms if they are
all within the same datacenter. The new total (150 ms+3 ms+2.5 ms+100 ms = 255.5 ms)
is less than our total 300 ms budget.
We would repeat this process for other requests that are time sensitive. For example, we
send email messages less frequently than we read them, so the time to send an email mes-
sage may not be considered time critical. In contrast, deleting a message happens almost
as often reading messages. We might repeat this calculation for a few deletion methods to
compare their efficiency.
Onedesignmightcontacttheserveranddeletethemessagefromthestoragesystemand
the index. Another design might have the storage system simply mark the message as de-
leted in the index. This would be considerably faster but would require a new element that