We had fun this week in our Seattle office setting up clustering for Splunk on Windows on a pure-IPv6 network. IPv6 has been gaining acceptance more outside the US than within for quite a number of years now and I am one of those optimists that expects that we will reach the tipping point soon where IPv6 adoption becomes the norm rather than the exception.
We had a set of four systems. On our indexer tier were a set of three indexers – one cluster master and two cluster slaves. We also had a separate search head. Each of these systems was running Windows Server 2008R2 and had the latest version of Splunk Enterpise 6 installed. The requirement was that this lab environment be able to communicate on a pure-IPv6 environment. We aren’t going to go into the details of how to configure your environment for IPv6 here as there are whole books devoted to the subject, but Microsoft TechNet is a good starting point.
Let’s first of all talk about the systems themselves. You need both the IPv4 and IPv6 stacks installed on the servers. The frontend web servers and backend splunkd processes talk on 127.0.0.1, so IPv4 is needed just for the local loopback. We don’t need to talk to the physical network using IPv4 so it doesn’t need to be configured, but it does have to be there.
Secondly, we needed to configure splunkweb to listen on the IPv6 port. We do this by editing the web.conf file on each server as follows:
[settings] listenOnIPv6 = yes
You can place this setting in the $SPLUNK_HOME\etc\system\local\web.conf file – it will not get overwritten on upgrade then. We need to do something similar to make splunkd listen on the IPv6 port. We do this by editing the server.conf file on each server:
[general] serverName = my-server-name listenOnIPv6 = yes
Once you restart everything, you will be able to access each individual machine by it’s IPv6 address or DNS name. However, don’t restart just yet because we want to set up clustering as well. We do the cluster set up within server.conf as well. Let’s start with the cluster master:
[clustering] mode = master replication_factor = 2 search_factor = 2 pass4SymmKey = changeme
You will need to understand the replication factor and search factor parameters as you would for any clustering requirement and you should definitely refer to the documentation or your friendly local Splunk expert for a good discussion on that subject. Now we need to set up the other indexers. Here is the addition to the server.conf:
[replication_port://9887] [clustering] master_uri = https://[2001:xxxx:xxxx::1]:8089 mode = slave pass4SymmKey = changeme
Note that the IPv6 address I am using in the master_uri is the globally routable IPv6 address of the cluster master. You can get the IPv6 addresses of the cluster master using ipconfig on that server.
Finally, our search heads will require a slightly different version:
[clustering] master_uri = https://[2001:xxxx:xxxx::1]:8089 mode = searchhead pass4SymmKey = changeme
This will allow the search head to redirect the request to the right cluster members based on their status.
I have a final word on the Windows Firewall. Turn it off unless you have a really good reason to have it on. It takes up resources that could be doing Splunk work. If, however, you need to have the Windows Firewall on, ensure you open port 8089 and 9887 for connections from the Splunk Enterprise servers and port 8000 generally for your web traffic, plus any port that you are listening for connections on (such as 9097 for connections from your Universal Forwarders).
By the way, this process will also work handily on other operating systems running Splunk Enterprise. Just ensure you have the IPv4 localhost enabled.