Advanced Deployments - Implementing Splunk: Big Data Reporting and Development for Operational Intelligence - page 337

This setup is complicated but very resilient as only a large network failure will cause
loss of events.
Receiving syslog with a Splunk forwarder
It is also possible to use Splunk instances to receive the syslog events directly, which
then forward the forwarders to Splunk indexers. This setup might look somewhat
like the following figure:
These interim Splunk forwarder processes can be configured with a large input
buffer using the queueSize and persistentQueueSize settings in inputs.conf .
Note that these interim forwarders cannot be light forwarders. There are a few
advantages to this approach that I can think of:
• If these Splunk forwarder processes are in the data center with the device
producing the events, the forwarder process will set the time zone of the
events. If you have devices in data centers in multiple time zones, this
can be very helpful.
• The work of parsing the events will be handled at this stage, offloading
some work from the indexers.
One disadvantage is that any parsing rules that are relevant to events parsed
by these interim forwarders must be installed at this layer, which may require
a restart when there are changes.
Consuming logs from a database
Some applications are built to store their logs in a database. This has the advantage
that the logs are centralized, but the disadvantage that it is difficult to scale beyond
the limits of the database server. If the logs are pulled into Splunk, it is possible to
take advantage of the Splunk interface and correlate these events with other logs.