UPDATE 9/24/14 (evening): I changed the script a little bit to include platform information in the output by using the uname command and bash version information in the output with –version. This should work on Linux and OSX.
UPDATE 9/25/14: The first script below is specific to find the original shellshock: CVE-2014-6271. The second shellshock vulnerability, CVE-2014-7169, requires a different test. See the script later in the post to cover this.
UPDATE 9/26/14: A whole bunch of useful comments have been added to this post. I have added information at the end of the post in response. I have further updated the scripts. Also, I should point out – if you are looking for information about how Splunk products are affected (or not) by this vulnerability, the official source is, and will always be, our Product Security Portal.
I knew that eventually, a vulnerability would find its way into the wild with its own theme song.
Today, David Millis, one of our fine Client Architects, sent out this link concerning a new vulnerability (CVE-2014-6271) in bash, found installed on, oh, just about every modern UNIX variant. Read about it here and here and about 4,000 other places by simply typing “shellshock” into Google News. And basically – stop reading Splunk blogs and go patch bash. Now.
I’ll let others tell you how you could use Splunk to search through your various logs for evidence that evildoers are trying to exploit this in your environment. But how can we use the trusty Splunk forwarders installed on your servers to ensure that, after your sysadmins patch bash everywhere, you don’t have systems that are still vulnerable? How about a scripted input?
Create a shell script that looks like this, somewhere on your system you want to test for vulnerable bash. A good place to put it could be right alongside the Splunk_TA_nix scripts that already gather information from your UNIX systems. I created it there, and called it shellshock.sh, and made it executable. You can always use Splunk’s deployment server to get this script out to your systems.
#!/bin/bash
HOSTNAME=$(/bin/hostname)
RUNNING=$(/bin/date)
BASHVERSION=$(/bin/bash --version | head -1)
if [ -f /bin/uname ]; then UNAME=$(/bin/uname -srvmpio); else UNAME=$(/usr/bin/uname -v); fi
THECHECK=$(env='() { :;}; echo status=VULNERABLE' bash -c "ls -al /bin/bash" 2>&1 /dev/null)
if [[ $THECHECK == *VULNERABLE* ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-6271 status=VULNERABLE version=$BASHVERSION"; else echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-6271 status=NOTVULNERABLE version=$BASHVERSION"; fi
Yeah, it looks funky above, but copy and paste it, all will be well. Now, create an entry in inputs.conf like this (this runs the check every hour…):
[script://./bin/shellshock.sh]
sourcetype = shellshock
source = shellshock
interval = 3600
index = os
disabled = 0And restart your forwarder.
You can now search for any hosts with sourcetype=shellshock, and of course use the field “status” to determine if the host is vulnerable or not, and with the “platform” field and “version” field you can filter on various OS versions and levels (you may need to adjust the field extractions depending on what you end up with in your versions and platforms…). The middle screenshot below shows that I’ve patched a Linux box and an OSX box in my lab, but one remains unpatched (because showing all OK is never fun for examples!)
I have tested this on recent Linux distributions (CentOS) and OSX, and it appears functional on both. It’s pretty simple – my guess is it will work on a lot of other *nix flavors. However, your mileage may vary. Please let me know how this works out for you, and improve as you see fit!
2014-7169 UPDATE: Here’s a modification of the above script to check for CVE-2014-7169. It’s a little more complex as there’s a need to capture standard error to assess the vulnerability, so a temp file is used. Tested on CentOS and OSX, but again, your mileage may vary. See Matt Qualls comment below for variations for Solaris and AIX. If you have both of these running as scripted inputs, you can simply use Splunk to filter on the “cve” field to display results from one or both vulnerabilities.
#!/bin/bash
HOSTNAME=$(/bin/hostname)
RUNNING=$(/bin/date)
BASHVERSION=$(/bin/bash --version | head -1)
TEMPFILE=$(mktemp -t 7169check.XXXXXX)
MATCH="^bash: findmyvuln: command not found$"
if [ -f /bin/uname ]; then UNAME=$(/bin/uname -srvmpio); else UNAME=$(/usr/bin/uname -v); fi
env X='() { (a)=>\' bash -c "echo findmyvuln > $TEMPFILE" 2>> $TEMPFILE; if [ -f echo ]; then cat echo; fi
THECHECK=$(cat $TEMPFILE)
rm $TEMPFILE
if [ -f echo ]; then rm echo; fi
if [[ $THECHECK =~ $MATCH ]] ; then echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-7169 status=VULNERABLE version=$BASHVERSION"; else echo "$RUNNING hostname=$HOSTNAME platform=$UNAME cve=2014-7169 status=NOTVULNERABLE version=$BASHVERSION"; fi
Search output from BOTH scripts above, showing that the freshly-compiled bash on my test OSX host is not vulnerable to 2014-6271, but is still for vulnerability 2014-7169:
UPDATE: 9/26/2014 (evening)
Thanks to everyone below for the many useful comments. I’ve specifically taken the improvements that Dan and Drew mentioned, and a few that Matt Qualls included in his version, and one from William Triest, and updated the two scripts above to incorporate.
The complete versions that Flint and William have provided are worth a look. In particular, the comments from William concerning default shells on a system are worth considering if you know that bash isn’t your default shell, because the script will fail if /bin/sh actually is linked to something else that won’t work with bash-style regex evaluation strings. I have updated the scripts to use /bin/bash as default.
My colleague David Cavuto has provided content that’s worthy of its own blog post. There, he details how to use Splunk specifically to find evidence of attempted bash exploit. I personally know of three major Splunk customers using this approach, today, to help with Shellshock remediation.
The second script above – the one that checks for 2014-7169 – has gone through several updates over the past few days. It turns out that the original version I posted was not operating correctly on OSX (on Linux, it was fine) – this version does, by trying to execute a command that doesn’t exist (findmyvuln) and then seeing if bash tried to actually execute it (which means that it is vulnerable). A patched bash will simply echo the string “findmyvuln” – POSSIBLY along with a few errors. The original version I posted was looking for the errors such as “syntax error near unexpected token” – which isn’t an entirely valid test, as the patched bash on OSX outputs errors but is not vulnerable. Now, the script looks for signs of execution of the bogus command – if those exist, then “VULNERABLE” is returned.
Finally, to Matt Qualls – your updates are very helpful as I personally don’t have any AIX (or Solaris, anymore) platforms to test against. I’d love to see some (sanitized, of course) output of the tracking you’re doing with that search.
Keep the updates coming, folks!