(Hi all–welcome to the latest installment in the series of technical blog posts from members of the SplunkTrust, our Community MVP program. We’re very proud to have such a fantastic group of community MVPs, and are excited to see what you’ll do with what you learn from them over the coming months and years.
–rachel perkins, Sr. Director, Splunk Community)
Hi, I’m Mark Runals, Lead Security Engineer at The Ohio State University, and member of the SplunkTrust.
If your experience is anything like mine, there have been times when you’ve put together a query that has found events of interest to you–only to have to spend extra time scanning back and forth within the results to make sure you’re looking at the same related events. For example, let’s say you have identified a series of suspect accounts that have logged in from a series of IP address and you’ve sorted the results based on a combination of time and IP. As you are scanning down the time column as it relates back to the user list, it can be easy to miss if the IP has changed or vice versa. It would be nice to add a visual break to quickly inform the reader where a block of events stops or starts. While this example won’t use a query as dramatic as the above scenario, hopefully the methodology is presented clearly enough that you can adapt it as needed.
Let’s start with a base set of data. In this case, I’m going to get the top 3 longest running searches from my Splunk instance. That query might look something like this:
index=_audit action=search info=completed | sort -total_run_time | dedup 3 user | table _time user total_run_time scan_count event_count | sort user -_time
The screenshot is trimmed to only the first three users and there are only three results to review. Hopefully though you can see how (because the user names are similar and the results include several columns that just contain numbers) as your eye roves around the data you might not immediately pick up on key result set elements like time and user shift if dedup or other limiting commands had not been used.
One key to introducing a break is to decide how you will end up sorting the data. I will continue with sorting by user and then time with the most recent events, by user, first. The command we’re going to use here is appendpipe. Under ‘normal’ use this command can be used for things like providing a subtotal in a running list. It is similar in usage to eventstats but unlike eventstats will only provide 1 result vs having the result attached to each applicable event.
In our case what we want to do is place the line break between users, so I will be taking the oldest event and subtracting some additional time from it so that when sorted it will be the last entry per user. Remember that data in the _time field is really stored in epoch, so manipulating it can be done with simple mathematical equations. In our case, my query now looks like this:
index=_audit action=search info=completed | sort -total_run_time | dedup 3 user | table _time user total_run_time scan_count event_count | appendpipe [ stats min(_time) as _time by user | eval _time = _time - 10 ] | sort user -_time
Note: I’ve moved the sorting command after the appendpipe and the ‘blank’ event line is 10 seconds after the last real event per user.
You could stop there but I’d like to add something like a short string with all dashes to really break one result set group from the other. To make things easier I’m going to use the foreach command. This will allow me to loop over all of my columns at once instead of an eval command for each:
index=_audit action=search info=completed | sort -total_run_time | dedup 3 user | table _time user total_run_time scan_count event_count | appendpipe [ stats min(_time) as _time by user | eval _time = _time - 10 ] | sort user -_time | foreach * [ eval <<FIELD>> = if(len(<<FIELD>>) >= 1, <<FIELD>>, "----") ]
To really clean up the results you’d likely want to take care of the _time and user fields. You could do this with one foreach command and add the dash line string or maybe get a little fancy and in the _time field say something like “New User”. A lot of this depends on personal taste and the data itself. Remember, part of this effort is to let the eye quickly pass over the event break line and more quickly identify the blocks of related events. You don’t want to add too much visual clutter. Either way the end result might look something like this:
index=_audit action=search info=completed | sort -total_run_time | dedup 3 user | table _time user total_run_time scan_count event_count | appendpipe [ stats min(_time) as _time by user | eval _time = _time - 10 ] | sort user -_time | foreach * [ eval <<FIELD>> = if(len(<<FIELD>>) >= 1, <<FIELD>>, "----") ] | eval user = if(total_run_time = "----", "----", user ) | eval _time = if(total_run_time = "----", "---- New User ----", _time )
Hopefully you can see the advantage to leveraging this capability and with the use of the foreach command you really aren’t adding a whole lot of length to your base query. Have you come up with other ways to introduce breaks in your data? Let us know!