Deploying Splunk Securely with Ansible Config Management – Part 1

Automation Johnny

Intro

More times than not I have seen corporations struggle with config management and it is key for concise mitigation and remediation plan. Interfacing with a variety of Splunk customers the corporations whom do implement a config management system usually have a different tactic on how to manage Splunk while doing it in a secure fashion. In this series of blog posts which will hopefully walk you through a simple deployment of Ansible all the way to the most complex use-cases I have seen. I will first be covering how Ansible can be leverage to manage a simple Splunk deployment on your own hosts. Part 2 we will cover how this can be done in a larger scale with EC2 utilizing dynamically changing inventory of hosts for deployments whom need to scale in a cloud environment. Finally part 3 will cover how to manage a giant deployment of Splunk with multi-tenecy requirements where there are various “customers” or business units with different Splunk config needs. The idea is to embark the necessary knowledge to not only deploy Splunk but anything else using Ansible as your config management system.

Why Ansible?

There are a few config management system. The most common ones I have seen deployed are Chef and Puppet. In my previous job we evaluated a few of them, including Puppet and Chef and we ended up choosing Ansible. Here are the reasons why I have seen organizations make that choice:

• No Agent Required – this is awesome! Personally, I think the less agents a system has the easier it is to manage and the smaller your exploit landscape needs to be. Ansible also has the ability to deploy an agent specially on a scenario where your host are pulling configs instead of a master server pushing them.
• Using SSH as Transport – No need to deal with custom communication protocols, everything is encrypted natively. Keys are used instead of passwords. Also SSH is pretty ubiquitous across the varied linux distributions. Most times than not you are already talking to your servers via SSH.
• Ease to pickup – I found that the learning curve in Ansible is tremendously smaller any otherconfig management system. This is primarily due to the fact that play books read easily. They are YAML base and the project has great documentation
• Low overhead and scales to huge deployments – There is no need to run a dedicated Ansible master server; the application has very low resource requirements. Ansible is also shown to scale, I have seen deployments as large as 100,000+ endpoints(link here).
• Python Base – do not like something, want to integrate with something else? Ansible is python based and very easily extendable. If you can code in python this is the CMDB for you, they speak your language!

Ansible is not for organizations whom their large server base is windows. Although they are working on Windows clients.

Installing Ansible

I suggest you grab Ansible from their stable repo in github instead of your distributions repository. The stable version in github has welcomed updates and additions like ansible-vault which we will cover later in further detail.

cd /opt/
git clone https://github.com/ansible/ansible

Now lets install the playbooks and configuration files we will use for deployment.

cd /etc/
git clone https://github.com/divious1/ansible-splunk-simple
mv ansible-splunk-simple /etc/ansible

Now you have a copy of the ansible application under /opt/ansible and a collection of configuration files under /etc/ansible. Below is a logical diagram which represents a high level Ansible application structure.

Ansible Structure


Explanation of what the different pieces are:

• ansible-playbook: Ansible executable which runs the playbooks etc..
• hosts: INI file which contains the role/group and host mapping
• playbooks: Ties in Roles, host groups and task together to create orchestrated actions on target hosts
• roles: contains the actions each group will complete (this is where the deployment logic lives).

Roles in details

Lets walk through the structure of a role in details. I will start with the common role. The common role should be ran no matter what kind of role the host has as it performs common functions that you would want on every host. If we look at the main.yml under tasks for this role we can see all the tasks it performs.

ansible$cat roles/common/tasks/main.yml
---
# This playbook contains common tasks in this role
- include: apt.yml
- include: users.yml
- include: files.yml
- include: cron.yml
- include: time.yml

The role contains 5 play books. Lets figure out what these are what what they do. Lets look at apt.yml in detail:

ansible$cat roles/common/tasks/apt.yml
---
# This playbook install the apps required in a server
- name: install security controls
tags:
- configuration
- security
apt: name={{ item }} state=present
with_items:
- chkrootkit
- rkhunter
- clamav
- fail2ban
- name: install basic utilities
tags:
- configuration
apt: name={{ item }} state=present
with_items:
- vim
- screen
- iotop
- htop
- ioping
- ntp


The description of this is at the top as a comment. Using the “apt” Ansible module (follow the link for information on modules) we install a variety of software on the server. The first batch of software is tagged as “configuration” and “security” and the last are just configurations. The first bath install chkrootkit, rkhunter, clamav, fail2ban.

Running Ansible for the First Time

Requirements

• Ansible Installed
• splunk-admin user updated with your keys under /playbooks/splunk_creds/splunk-admin.pub
• Root password of hosts to run Ansible in
• Make sure you have ssh keys generated for root
• Hosts inventory updated

Before running Ansible make sure that your environment is set correctly. Run . /opt/ansible/hacking/env-setup in order to have the necessary paths and environment activated for Ansible to run. Verify it worked by running
which ansible-playbook
/opt/ansible/bin/ansible-playbook
This can also be added to bash profile as well.

To build a splunk server from scratch just run:
/etc/ansible#ansible-playbook /etc/ansible/playbook/search_heads.yml

Make sure that you have hosts defined under hosts
If the host you are running is virgin (nothing has been done to it) you will want to run Ansible with -k so it prompts for the password during the first run. You will only need this once as Ansible has a task under roles common to copy the ssh key of your user to that host.

You should see a similar output to this:


/etc/ansible#ansible-playbook playbooks/search_heads.yml
PLAY [apply common configuration to all nodes] ********************************
GATHERING FACTS ***************************************************************
ok: [162.243.231.42]
TASK: [common | install security controls] ************************************
ok: [162.243.231.42] => (item=chkrootkit,rkhunter,clamav,fail2ban)
TASK: [common | install basic utilities] **************************************
ok: [162.243.231.42] => (item=vim,screen,iotop,htop,ioping,ntp)
TASK: [common | create splunk-admin] ******************************************
ok: [162.243.231.42]
TASK: [common | copy splunk-admin bash_profile] *******************************
ok: [162.243.231.42]


Closing Thoughts

This should have armed you with the basic tools to do config management and orchestration on Splunk hosts as well as the rest of your infrastructure.

I encourage you to walk through each roles task to verify what it is doing. Also the github read me will have more information as to how Splunk is configured using the shipped playbooks. Feel free to drop me a line with any questions or if something is not working at jose at splunk.

Thank you Mike Regan, Mike, Baldwin and the rest of the Splunk Cloud team for your contributions and working with me on the Playbooks.