Advanced Deployments - Implementing Splunk: Big Data Reporting and Development for Operational Intelligence - page 336

Since the configuration of the native syslog process is simple and unlikely to change,
simply using another process on your single Splunk instance will add some level
of protection from losing messages. A slow disk, high CPU load, or memory pressure
can still cause problems, but you at least won't have to worry about restarting the
Splunk process.
The next level of protection would be to use separate hardware to receive the syslog
events and to use a Splunk forwarder to send the events to one or more Splunk
indexers. That setup looks like the following figure:
This single machine is still a single point of failure, but it has the advantage that the
Splunk server holding the indexes can be restarted at will and will not affect the
instance receiving the syslog events.
The next level of protection is to use a load balancer or a dynamic DNS scheme to
spread the syslog data across multiple machines receiving the syslog events, which
then forward the events to one or more Splunk indexers. That setup looks somewhat
like the following figure: