Databases Reference
In-Depth Information
Figure
8-7
is an example of the type and amount of valuable information generated
by a single website visitor that is recorded. A typical e-commerce site serving thousands
of users a day will generate gigabytes of machine data that can be used to provide
significant insights into the IT infrastructure and business operations.
How do we process machine-generated data?
Let's consider the scenario of a retailer. The retailer is using around forty applications
hosted on a multitude of servers (200+) in different data centers to manage their business
processes. A single business process involves several business applications, workflows,
and associated data. Each application creates log messages that are stored as text files
in the local file system. There are multiple web applications where customers browse
products and offerings and select their preferences. Once an order is placed, the request
is sent to a central order-processing application. This application performs the following
steps in order: checks for availability in the inventory management application, performs
the payment in the credit-card-processing application and initiate the shipment. Each of
the involved applications runs on different servers and produces log files.
The log messages usually consist of some fixed fields, like for example a timestamp,
a logging level, or the name of the logging component or application in addition to key
information in unformatted plain text. There are different possibilities for storing these
messages. In a relational database, you might just record the whole of the message in
a CLOB field, or you might try to store the message into a pre-defined schema. While
CLOB will be an easy way to store the data, retrieving information from CLOB has
additional challenges, as you will have to develop a full text search index to be able to
find a message in the CLOB field. The approach with the pre-defined schema also has
several disadvantages, leading to problems when adding a new application that creates
log messages with a different format. Messages might also be truncated when they do not
fit into the schema.
Figure
8-8
shows an example of a log message. Each message consists of a couple
of fields like timestamp, class, method, and log level, followed by the log message itself
consisting of unstructured text. Different applications can have different sets of logging
fields.
Figure 8-8.
Log message example
