Advanced Deployments - Implementing Splunk: Big Data Reporting and Development for Operational Intelligence - page 370

That should be it. The hardest part is usually convincing the web server to
both authenticate and proxy. Use the /debug/sso page to help diagnose what
is happening.
There can also be issues with punctuation in the header fieldname. If it's possible,
removing any punctuation in the header name may eliminate unexpected problems.
Load balancers and Splunk
Some organizations that have invested heavily in load balancers like to use them
whenever possible to centralize network management. There are three services
Splunk typically exposes, mentioned in the following sections:
web
Usually on port 8000, the Splunk web server can be load balanced when configured
with search head pooling . The load balancer should be configured to be "sticky",
as the web server will still rely on user sessions tied to the web server the user
started on.
See the Multiple search heads section for more information.
splunktcp
Usually on port 9997, splunktcp is itself stateless. Splunk auto load balancing is
very well tested and very efficient but does not support complicated logic. For
instance, you could use a load balancer to prefer connections to indexers in the
same data center, only using indexers in another data center as a last resort.
The problem is that when only one address is provided to a Splunk forwarder,
the forwarder will open one connection and keep it open indefinitely. This means
that when an indexer is restarted, it will never receive a connection until forwarders
are restarted.
The easy solution is to expose two addresses on your load balancer and list both of
these addresses in outputs.conf . The two addresses must be either two different
ports or two different IP addresses. Two different CNAMEs on the same port will
not work, as Splunk resolves the addresses and collapses the list of IP addresses.