Designing in a Distributed World - The Practice of Cloud System Administration - page 21

Case Study: Emulation before Improvements
When Tom was at Cibernet, he was involved in a project to replace an older sys-
tem. Because it was a financial system, the new system had to prove it was bug-
for-bug compatible before it could be deployed.
The old system was built on obsolete, pre-web technology and had become so
complex and calcified that it was impossible to add new features. The new system
wasbuilt onnewer,better technology and,being acleaner design, wasmore easily
able to accommodate new functionality. The systems were run in parallel and res-
ults were compared.
Atthatpointengineersfoundabugintheoldsystem.Currencyconversionwas
being done in a way that was non-standard and the results were slightly off. To
make the results between the two systems comparable, the developers reverse-en-
gineered the bug and emulated it in the new system.
Now the results in the old and new systems matched down to the penny. With
the company having gained confidence in the new system's ability to be bug-for-
bugcompatible,itwasactivatedastheprimarysystemandtheoldsystemwasdis-
abled.
Atthispoint,newfeaturesandimprovementscouldbemadetothesystem.The
first improvement, unsurprisingly, was to remove the code that emulated the cur-
rency conversion bug.
1.7 Speed
So far we have elaborated on many of the considerations involved in designing large dis-
tributed systems. For web and other interactive services, one item may be the most import-
ant: speed. It takes time to get information, store information, compute and transform in-
formation, and transmit information. Nothing happens instantly.
An interactive system requires fast response times. Users tend to perceive anything
faster than 200 ms to be instant. They also prefer fast over slow. Studies have documented
sharp drops in revenue when delays as little as 50 ms were artificially added to web sites.
Time is also important in batch and non-interactive systems where the total throughput
must meet or exceed the incoming flow of work.
The general strategy for designing a system that is performant is to design a system us-
ing our best estimates of how quickly it will be able to process a request and then to build
prototypes to test our assumptions. If we are wrong, we go back to step one; at least the
next iteration will be informed by what we have learned. As we build the system, we are