Database Reference
In-Depth Information
The “exactly once” delivery mechanism, of course, sacrifices the raw
performance of the “at most once” delivery mechanism in an effort to
provide this safety whether or not it is required. The “at least once” delivery
system tries to balance these two extremes by providing reliable message
delivery by pushing the handling semantics to the consumer. By doing this,
the consumer can use the delivered data stream to construct the semantics
required by the application without affecting other consumers of the data
stream. For example, a consumer requiring “exactly once” processing for
something such as financial transactions between two accounts can
implement a de-duplication procedure to ensure that messages are not
reprocessed. Another consumer who cares only about tracking something
like the number of unique account transaction pairs does not need “exactly
once” delivery because it uses idempotent set operations, which means it
does not need to pay the management penalty. The general theory is that
appropriate message handling, beyond ensuring that they are delivered at
least once, is dependent on the application logic and should be handled at
that level.
The “n+1” Problem
In “traditional” log processing systems, the architecture is essentially a
funnel.Thedatafromapotentiallylargenumberofedgesystemsiscollected
to a small number of central locations (usually one, but legal or physical
restrictionssometimenecessitatemorethanoneprocessinglocation).These
processing locations then manipulate the data and perhaps move it along to
another location, such as a data warehouse, for further analysis.
After this first funnel is in place, the inevitable happens: A second data
consumerneedstobeadded,resultingintheconstructionofanotherfunnel.
After that, perhaps another front-end service is introduced that has its own
data collection mechanism. Eventually, every time a new service or
processing mechanism is added it must integrate with each of the other
systems and, even worse, they must integrate with it. It may seem like
an exaggeration, but this is actually a fairly common antipattern in the
industry.
In the Enterprise space, this problem manifested itself in the form of the
service-oriented architecture (SOA) and saw the development of the idea of
theenterprise service bus(ESB).Theidea wasthatthebuswouldhandle the
interaction with various services, which were implemented independently
