Advanced Deployments - Implementing Splunk: Big Data Reporting and Development for Operational Intelligence - page 334

Sometimes this is the only approach possible, and in those cases, you should follow
a few rules:
• Only copy complete logs to the watched directory.
• If possible, use batch stanzas in inputs.conf , instead of monitor stanzas,
so that Splunk can delete files after indexing them.
• If possible, copy sets of logs to different Splunk servers, either to multiple
forwarders that then spread the logs across multiple indexers, or possibly
directly to watched directories on the indexers. Be sure to not copy the
same log to multiple machines as Splunk has no mechanism for sharing
file position information across instances.
Receiving syslog events
Another common source of data is syslog , usually from devices that have no
filesystem or no support for installing software. These sources are usually devices or
appliances, and usually send those events using UDP packets. Syslog management
deserves a topic of its own, so we will only discuss how to integrate syslog with
Splunk at a high level.
Receiving events directly on the Splunk indexer
For very small installations, it may be acceptable to have your Splunk server listen
directly for syslog events. This installation looks essentially like the following figure:
On the Splunk indexer, you would create an input for syslog , listening on udp or
tcp . The inputs.conf configuration would look like:
[udp://514]
sourcetype = syslog
The advantage of this approach is its simplicity. The major caveat is that, if the
Splunk process is down or busy for some reason, you will lose messages. Reasons
for dropped events could include a heavy system load, large queries, a slow disk,
network problems, or a system upgrade.