Monitoring Fundamentals - The Practice of Cloud System Administration - page 340

• System Logs: The operating system kernel, devices, and system services contrib-
ute to the system log. This is often useful for tracking hardware problems and sys-
tem changes.
• Application Logs: Each service or server generates logs of actions. This is useful
for studying errors and debugging problems, as well as providing business inform-
ation such as which features are used the most.
• Application Debug Logs: Applications often generate debugging information in a
separate log. This type of log is often more verbose but is retained for a shorter
amount of time. Such logs are used for debugging problems by developers and op-
erations staff.
16.6.1 Approach
While logs do not directly fit the Dickson model, if they did they would be (R+, L-, D-)
because they are high resolution (generally one log entry per action), high latency (often
processed much later in batches), and low diversity (usually collected about a particular
topic, such a web server hits).
Thearchitectureofalogprocessingsystemissimilartothemonitoringarchitecturedis-
cussed in the next chapter. Generally logs are collected from machines and services and
kept in a central place for storage and analysis. Retention is often a legal issue, as logging
data often contains personal information that is regulated.
Themarketspaceforloganalysistoolshasbeengrowingovertheyearsasnewanalysis
methods are invented. Weblogs can beanalyzed todetermine the path auser takes through
thesystem.Asaresultonecanfinduserinterface“deadends”thatleaveusersbaffled.Ap-
plication logs can beused tofindflaws insales processes ortoidentify high-value custom-
ers who otherwise would not have been discovered. System logs can be analyzed to find
anomalies and even predict hardware failures.
The consolidation of logging data can be rather complex, as different systems generate
logsindifferentformats.It'sbesttoestablishasinglelogformatforallsystems.Byprovid-
ingasoftwarelibrarythatgenerateslogsthatconformtotheformat,youcanmakethepath
of least resistance (using the library) be the path that has the desired behavior (conforms to
the standard).
16.6.2 Timestamps
Record timestamps the same way in all systems to make it easier to combine and compare
logs. Inparticular,all machines should use NTP orthe equivalent to keep clocks synchron-
ized, and timestamps should be stored in UTC rather than local time zones.