Configuring Splunk - Implementing Splunk: Big Data Reporting and Development for Operational Intelligence - page 276

Never make changes in $SPLUNK_HOME/etc/system/default
as your changes will be lost when you upgrade Splunk.

2. $SPLUNK_HOME/etc/apps/*/default : Configurations are "overlaid" in
reverse ASCII order by app directory name. a beats z .

3.

$SPLUNK_HOME/etc/apps/*/local

4. $SPLUNK_HOME/etc/system/local

° The configurations in this directory are applied last.
° Outside of search, these configurations cannot be overridden
by an app configuration. Apps are a very convenient way to
compartmentalize control and distribute configurations. This
is particularly relevant if you use the deployment server, which
we will cover in Chapter 11 , Advanced Deployments .

Do not edit configurations in $SPLUNK_HOME/etc/system/local
unless you have a very specific reason. An app is almost always the
correct place for configuration.

A little pseudo code to describe this process might look like this:

$conf = new Configuration('$SPLUNK_HOME/etc/')

$conf.merge( 'system/default/$conf_name' )

for $this_app in reverse(sort(@all_apps)):
$conf.merge( 'apps/$this_app/default/$conf_name' )

for $this_app in reverse(sort(@all_apps)):
$conf.merge( 'apps/$this_app/local/$conf_name' )

$conf.merge( 'system/local/$conf_name' )

Merging order when searching
When searching, configuration merging is slightly more complicated. When running
a search, there is always an active user and app, and they come into play. The logical
order looks like this:

1. $SPLUNK_HOME/etc/system/default
2.

$SPLUNK_HOME/etc/system/local