One of the great features of the Splunk App for Microsoft Exchange is that you can track messages to the edge. It doesn’t matter what type of devices we go through, we get to see the messages and what hops they go through. Doing that requires some knowledge of the data flow and the construction of appropriate searches.
Let’s take an example of the inbound message flow. To track an inbound message, we use a macro – msgtrack-inbound-messages. The comments in the macros.conf file tell us that we need to have a table that has the date/time, message-id, cs-ip, sender, sender-domain, recipient-count, list-of-recipients and message-size. It then goes on to show off the Microsoft Exchange version. How would we alter this for including Cisco ESA data? Cisco Email Security Appliance is a very common, industry leading security appliance for anti-malware protection. This is one of those we would have to include.
The best way to include Cisco ESA (or for that matter any email security) data is to construct a search that provides the table for just that data stream. Then you can include the data rather easily in the overall definition. Cisco ESA logs are problematic because you don’t get just one event per message like you would with, say, a firewall. You get many events. In addition, there isn’t just one identifying ID that you can correlate on – you can package multiple messages inside of an inbound connection, handle processing across multiple IDs and then send on to the Exchange system with multiple connections. That’s a lot of IDs to handle. The logs look like this sample:
Wed Feb 12 19:48:37 2014 Info: New SMTP ICID 1000000201 interface Data2 (192.0.2.44) address 203.0.113.83 reverse dns host unknown verified no
Wed Feb 12 19:48:37 2014 Info: ICID 1000000201 ACCEPT SG UNKNOWNLIST match sbrs[0.0:10.0] SBRS 5.1
Wed Feb 12 19:48:37 2014 Info: ICID 1000000201 TLS success protocol TLSv1 cipher AES128-SHA
Wed Feb 12 19:48:37 2014 Info: Start MID 500000014 ICID 1000000201
Wed Feb 12 19:48:37 2014 Info: MID 500000014 ICID 1000000201 From:
Wed Feb 12 19:48:37 2014 Info: MID 500000014 ICID 1000000201 RID 0 To:
Wed Feb 12 19:48:37 2014 Info: MID 500000014 ICID 1000000201 RID 1 To:
Wed Feb 12 19:48:38 2014 Info: MID 500000014 Message-ID ''
Wed Feb 12 19:48:38 2014 Info: MID 500000014 Subject 'FW: Some Subject Matter'
Wed Feb 12 19:48:38 2014 Info: MID 500000014 ready 52076 bytes from
Wed Feb 12 19:48:38 2014 Info: LDAP: Masquerade query Rewrites-Inbound MID 500000014 address bill.jones@example.com to bill.jones@example.com
Wed Feb 12 19:48:38 2014 Info: LDAP: Masquerade query Rewrites-Inbound MID 500000014 address reginald.brown@example.com to reginald.brown@example.com
Wed Feb 12 19:48:38 2014 Info: LDAP: Masquerade query Rewrites-Inbound MID 500000014 address amy.johnson@example.com to amy.johnson@example.com
Wed Feb 12 19:48:38 2014 Info: MID 500000014 rewritten to MID 500000031 by LDAP rewrite
Wed Feb 12 19:48:38 2014 Info: MID 500000031 ICID 0 From:
Wed Feb 12 19:48:38 2014 Info: LDAP: Reroute query AD.routing MID 500000014 RID 0 address bill.jones@example.com to [('bill.jones@internal.example.com', '')]
Wed Feb 12 19:48:38 2014 Info: MID 500000031 ICID 0 RID 0 To:
Wed Feb 12 19:48:38 2014 Info: LDAP: Reroute query AD.routing MID 500000014 RID 1 address amy.johnson@example.com to [('amy.johnson@internal.example.com', '')]
Wed Feb 12 19:48:38 2014 Info: MID 500000031 ICID 0 RID 1 To:
Wed Feb 12 19:48:38 2014 Info: Message finished MID 500000014 done
Wed Feb 12 19:48:38 2014 Info: MID 500000031 attachment 'image001.jpg'
Wed Feb 12 19:48:38 2014 Info: MID 500000031 attachment 'image002.jpg'
Wed Feb 12 19:48:38 2014 Info: MID 500000031 Custom Log Entry: Attachment Names: image001.jpg, image002.jpg
Wed Feb 12 19:48:38 2014 Info: MID 500000031 Custom Log Entry: Attachment Sizes: 1736, 1525
Wed Feb 12 19:48:38 2014 Info: MID 500000031 Custom Log Entry: Attachment Types: image/jpeg, image/jpeg
Wed Feb 12 19:48:38 2014 Info: ICID 1000000201 close
Wed Feb 12 19:48:38 2014 Info: New SMTP DCID 70000094 interface 192.0.2.44 address 192.0.2.89 port 25
Wed Feb 12 19:48:38 2014 Info: DCID 70000094 STARTTLS command not supported
Wed Feb 12 19:48:38 2014 Info: Delivery start DCID 70000094 MID 500000027 to RID [0]
Wed Feb 12 19:48:38 2014 Info: Message done DCID 70000094 MID 500000027 to RID [0]
Wed Feb 12 19:48:38 2014 Info: MID 500000031 matched all recipients for per-recipient policy DEFAULT in the inbound table
Wed Feb 12 19:48:39 2014 Info: Delivery start DCID 70000094 MID 500000026 to RID [0]
Wed Feb 12 19:48:39 2014 Info: Message done DCID 70000094 MID 500000026 to RID [0]
Wed Feb 12 19:48:39 2014 Info: Delivery start DCID 70000094 MID 500000033 to RID [0]
Wed Feb 12 19:48:39 2014 Info: Message done DCID 70000094 MID 500000033 to RID [0]
Wed Feb 12 19:48:40 2014 Info: MID 500000031 interim verdict using engine: CASE spam negative
Wed Feb 12 19:48:40 2014 Info: MID 500000031 using engine: CASE spam negative
Wed Feb 12 19:48:40 2014 Info: Delivery start DCID 70000094 MID 500000049 to RID [0]
Wed Feb 12 19:48:40 2014 Info: MID 500000031 interim AV verdict using Sophos CLEAN
Wed Feb 12 19:48:40 2014 Info: MID 500000031 antivirus negative
Wed Feb 12 19:48:40 2014 Info: MID 500000031 Outbreak Filters: verdict negative
Wed Feb 12 19:48:40 2014 Info: MID 500000031 queued for delivery
Wed Feb 12 19:48:40 2014 Info: Message done DCID 70000094 MID 500000049 to RID [0]
Wed Feb 12 19:48:40 2014 Info: Delivery start DCID 70000094 MID 500000031 to RID [0, 1]
Wed Feb 12 19:48:40 2014 Info: Message done DCID 70000094 MID 500000031 to RID [0, 1]
Wed Feb 12 19:48:40 2014 Info: MID 500000031 RID [0, 1] Response '2.6.0 Queued mail for delivery'
Wed Feb 12 19:48:40 2014 Info: Message finished MID 500000031 done
Wed Feb 12 19:48:40 2014 Info: DCID 70000094 close
In this example, you can see one Inbound Connection ID (ICID), one Direct Connection ID (DCID), multiple Recipient ID (RID) and two Message IDs (MIDs) and a clear conversion from one MID to the other in the middle of the transaction. Let’s assume for a moment that you have used MV_ADD in transforms.conf to extract ICID, DCID and MID as multi-value fields. In addition, you will note (I’ve left them in) that there are other messages interleaved in the process – if there are queued messages waiting to go, the Cisco ESA will transmit those down the same DCID to the Exchange environment. Let’s take a look at my search:
sourcetype=cisco:esa
| eventstats values(cs_ip) AS cs_ip BY icid
| eventstats values(ss_ip) AS ss_ip BY dcid
| stats values(mid) AS tmpMID
values(icid) AS icid
values(sender) AS sender
values(recipient) AS recipient
values(total_bytes) AS total_bytes
values(message_id) AS message_id
values(cs_ip) AS cs_ip
values(ss_ip) AS ss_ip
values(dcid) AS dcid BY mid
| eval recipient_count=mvcount(recipient)
| eval mid=tmpMID
| mvexpand mid
| eventstats values(tmpMID) AS tmp BY mid
| eval t=mvjoin(tmp, " ")
| rex field=sender "@(?.*)"
| stats values(icid) AS icid
values(sender) AS sender
values(sender_domain) AS sender_domain
values(recipient) AS recipient
max(total_bytes) AS total_bytes
max(recipient_count) AS recipient_count
values(message_id) AS message_id
values(dcid) AS dcid
values(tmp) as mid
values(cs_ip) AS cs_ip
values(ss_ip) AS ss_ip BY msg
I’m betting most of you would have reached for the transaction command here. In concept, that would be the most expedient way of doing this. Transaction (and others like append and join) are commonly used by folks new to Splunk Search because they work in a similar way to SQL commands, which many people have dealt with in the past. Events are streams of data and really need to be treated differently. Transaction is one of a handful of commands that are not streamable. A streamable command will get pushed down to the Splunk Indexer whereas a non-streamable command is run on the Search Head. In large scale environments, you will have multiple indexers but you will only be interacting with one search head. This means that transaction is a big performance destroyer. There are other non-streamable commands and some non-streamable eval statements. You can expect appends and joins to be big performance killers, for example.
The search itself is mostly simple eval/stats. However, the major magic here is in eventstats. Eventstats is an underutilized streaming command that alters the event stream as it passes. In this case, the first eventstats command adds an accumulator for any IP addresses used by the inbound connection and the second eventstats command does the same for the IP addresses used on the outbound connection. The third eventstats is where our magic happens. It accumulates all the Internal Message IDs related to this single message. We then concatenate those all together to give us a unique field on which to generate our final table.
When looking at a search that I’ve been given, I always go back to the pipeline. I’ll start from the very first command and slowly add commands to the pipeline, re-running the resulting pipeline at each step and adding the fields being generated along the way to my view. This allows me to really understand what the pipeline is doing.
You can extend these ideas such that all the big transaction macros in the Splunk App for Microsoft Exchange are streamable. This will result in major performance gains when you have complicated environments. Don’t let transaction kill your performance.