Just found out about another cool feature that apparently has been in the product a while. By default, the Universal Forwarder can only load balance between indexers when it is safe for us to cut over the data stream. Meaning, to avoid half an event going to one server and the other half going to another server, we have to wait for a safe place to break. When reading a file this can only be done when we hit EOF. For TCP, it’s when we don’t get data on a port for 10 seconds. Over the long run, all the indexers end up with approximately the same amount of events, so not being able to split the stream is not huge issue. However, when loading large amounts of historical data, it would be nice to spread the data over all your indexers. Enter the forceTimebasedAutoLB setting for outputs.conf. Using this setting Splunk can break the data stream and reconstruct the event properly on the indexer.
Of course I had to prove this to myself and here are my results.
I used 3 Windows VMs for my test. One UF and two indexers. Here is my outputs.conf on the UF:
Simple enough to setup.
Test 1: Reading a file with 2 million events with no forceTimebasedAutoLB:
Notice all the events are on one server.
Test 2: Reading the same file with the forceTimebasedAutoLB setting on:
Notice that the file has been spread among the two indexer. AWESOME!!!!1!!
Test 3: TCP data. I used netcat from my laptop to send over 10 million events to the UF:
w1n!
Thanks Jag for adding this sweet feature.