Splunk 6.2 Feature Overview: XML Event Logs

We’ve been (rightly) criticized for a couple of things in recent years. Firstly, when you configure a Windows Event Log, it’s too big. This is because we combine the event log object with the message from the locale-specific DLL and that includes a bunch of common explanatory text. I don’t really need to know what a login really means (to the tune of 1K of data ingest) every time someone logs in, especially when these events are happening hundreds of times a minute. Secondly, our event log extractions are for US/English only. Got German Windows? Sorry – our extractions don’t work for that. Finally, we discard the additional data that is provided in the event log object. A primary example of this is the Logon Performance Report, which Jason Conger has written about. In addition, we occasionally lost contact with the locale-specific message DLL and that caused us to not write the event at all, losing even more data.

The locale-specific message DLL is the culprit here. It changes depending on the locale of Windows you have installed, causes the message bloat and throws away data that we are interested in. What if we didn’t use the locale-specific message DLL? What if we stored the event logs in XML?

With Splunk 6.2, that’s exactly what you can do. The effort does not come for free, however. We’ve already done the hard work of updating the Splunk_TA_Windows to include CIM compliant extractions for the Security Event Log. If you have done your own extractions (or you are using an alternate TA that contains extractions), then these need to be updated to support the XML format. However, converting to the XML format solves all the problems I mentioned.

So, how does it work? First of all, you need to update your inputs.conf to render the XML. We’ve designed it so that nothing changes on upgrade (backwards compatibility FTW!) You need to add a line to your WinEventLog definitions like this:

[WinEventLog://Security]
index=security
current_only=1
evt_resolve_ad_obj=0
renderXML=1
disabled=0

Once you have restarted your universal forwarder, you will see XML coming in with a different source type (XmlWinEventLog:Security, for example).

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
 <System>
  <Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/>
  <EventID>4672</EventID>
  <Version>0</Version>
  <Level>0</Level>
  <Task>12548</Task>
  <Opcode>0</Opcode>
  <Keywords>0x8020000000000000</Keywords>
  <TimeCreated SystemTime='2014-04-29T22:15:03.280843700Z'/>
  <EventRecordID>2756</EventRecordID>
  <Correlation/>
  <Execution ProcessID='540' ThreadID='372'/>
  <Channel>Security</Channel>
  <Computer>sacreblue</Computer>
  <Security/>
 </System>
 <EventData>
  <Data Name='SubjectUserSid'>AUTORITE NT\Système</Data>
  <Data Name='SubjectUserName'>Système</Data>
  <Data Name='SubjectDomainName'>AUTORITE NT</Data>
  <Data Name='SubjectLogonId'>0x3e7</Data>
  <Data Name='PrivilegeList'>SeAssignPrimaryTokenPrivilege
                             SeTcbPrivilege
                             SeSecurityPrivilege
                             SeTakeOwnershipPrivilege
                             SeLoadDriverPrivilege
                             SeBackupPrivilege
                             SeRestorePrivilege
                             SeDebugPrivilege
                             SeAuditPrivilege
                             SeSystemEnvironmentPrivilege
                             SeImpersonatePrivilege</Data>
 </EventData>
</Event>

Note that the keys are always rendered in English, irrespective of the system locale of the machine that is generating the events. This event was generated from a machine running with French as the system locale – note the field names.

The effect of this on your data ingest rate is rather dramatic. Firstly, the size of the Windows Security Event Log is reduced by almost 70%. That means that a typical Active Directory Monitoring solution will occupy just over 1Gb of data per 1,000 users (instead of the 4Gb of data that it would take up in the old format). In addition, you don’t have to write custom code to get at those extra fields – the Logon timing report discussed above is now possible with just a simple dashboard.

For those of you running Enterprise Security, you will be happy to hear that, as of Splunk_TA_Windows 4.7.0, the XmlWinEventLog format is recognized within the TA and all the same field extractions are performed on the new format. That means you can take advantage of the data ingest savings straight away.

You might be wondering if there is any reason to not use the XmlWinEventLog format as your default now? Unfortunately, yes, there is. When we wrote the Windows Infrastructure and Exchange apps, we used the old format. The field extractions for these apps will be updated in the next release. Some dashboards will not render correctly if you change your data format right now.