Database Reference
In-Depth Information
Real-Time Architecture Components
This section describes the components often found in a modern real-time
architecture. Noteverysystemwillhaveeverycomponent, especiallyatfirst.
The goal should be to build a system appropriate to an initial application,
with the ability to expand to incorporate any missing components in the
future.
After the first application is established, other applications of the
infrastructure will inevitably follow with their own requirements. In fact, it
is quite likely that some components will have multiple implementations. It
happens most often in the storage component, where there is a wide range
of performance characteristics among the different products available.
This section covers each of the components and some aspects of their
development and future path. In general, the specifics of each component
will be covered in more depth in their relevant chapters in the remainder of
Part I.
Collection
The most common environment for real-time applications discussed in this
topic is handled over TCP/IP-based networks. As a result, the collection of
data happens through connections over a TCP/IP network, probably using a
common protocol such as HTTP. The most obvious example of this form of
collection is a large latency-sensitive website that geo-distributes its servers
(known as edge servers) to improve end-user latency.
With websites being one of the original use cases for large-scale data
collection, it is not surprising that the log formats used by web servers
dominated the formatting of log files. Unfortunately, most web servers
adhered to the National Center for Supercomputing Applications (NCSA)
Common Log Format and the later World Wide Web Consortium (W3C)
Extended Log File Format, which was never intended to support the rich
data payloads (and does so poorly).
Newer systems now log in a variety of formats, with JavaScript Object
Notation (JSON) being one of the most popular. It has gained popularity
by being able to represent rich data structures in a way that enables it
to be easily extended. It is also relatively easy to parse, and libraries for
it are widely available. It is used to transfer data between client-side web
applications and the server side.
