Designing for Operations - The Practice of Cloud System Administration - page 29

A typical way to achieve all of these goals is for the configuration to take the form of a
text file with a well-defined format. Automated systems can easily generate such a file.
Text files are easy to parse and therefore auditable. They can be archived easily. They can
alsobestoredinasourcecoderepositoryandanalyzedwithstandardtextcomparisontools
such as UNIX diff .
In some systems the configuration is dynamically updated as the system runs. This
“state”maybereflectedbackintotheprimaryconfigurationfileormaybeaseparateentity.
In this case there are additional requirements.
There must be a way for automation to read and update the state. This step may be car-
ried out through an API or by reading and updating a configuration file. If a file is used, a
locking protocol must exist to prevent both the service and external automation from read-
ing the file in an incomplete state and to prevent update collisions. Tools should be avail-
able for doing sanity checks on configurations that do not involve activating the configura-
tion on a live system.
An inferior option would be a configuration file that is an opaque binary blob. Such a
file is not human readable. In these types of systems it is impossible to keep a history of
the configuration and see change over time. Often strange problems are debugged by ana-
lyzing changes to a configuration file, where a change may be too small to be remembered
butjustbigenoughtocauseaproblem.Thistypeofanalysisisnotpossibleifthefileisnot
plain text.
We have been burned by systems that provide an API for extracting the entire config-
uration but where the result turns out not to actually represent the entire configuration. All
too often, the omission is found only during a disaster recovery exercise or emergency. For
that reason each new release should be tested to verify that the configuration data does not
omit anything.
From an operational perspective, the ideal is for the configuration to consist of one or
more plain text files that can be easily examined, archived, and compared.
Some systems read their configuration directly from a source code repository, which is
convenient and highly recommended. However, it must also be possible to disable this fea-
ture and provide configurations directly. Such an approach may be used in an emergency,
when the source code repository is down, and for experimentation. The use of this feature
must be exposed in a way that the monitoring system can detect it. Other users can then
be made aware that it is happening—for example, by showing this status in dashboards. It
may also be an alertable event, in which case alerts can be generated if this feature is used
on production systems. Alerting if this feature is disabled for more than a certain amount
of time assures that temporary fixes are not forgotten.