Let’s face it – most of our applications run on hypervisors – Microsoft Hyper-V, VMware or Citrix XenServer seem to be the top contenders. This makes our technology stacks that much more complex since we have added a layer of abstraction between the application and the bare metal. Instead of a stack that includes Compute, Storage, OS, and Application, we’ve added Hypervisor to the mix. How do we correlate what is happening on the compute platform to what is happening on the application level? How do we understand which other applications are running on the same hypervisor? One common instance, for instance, is in memory management. An application runs out of memory, but the hypervisor has memory that has not been allocated to the guest because the memory metric from the hypervisor perspective doesn’t reflect the fact that the application is under memory stress (which is generally because the hypervisor has no visibility into how guests use the allocated memory, so they can’t see the difference between cache memory, paged memory, and pooled memory).
The key to all of this is understanding our correlation points. In the case of hypervisors, the most obvious correlation points are the MAC address of the guest OS and the type of Hypervisor that the guest is running on. For this work, we will turn to my favorite data input workhorse, the SA-ModularInput-PowerShell addon. With this addon, we can write small PowerShell scripts that run on a regular basis to capture the information. Since the SA-ModularInput-PowerShell is based on PowerShell 3.0, we have a couple of thousand PowerShell cmdlets to choose from. Normally, we will be monitoring the guest OS, so let’s get this correlation information from there.
Let’s start with getting the MAC address of the guest OS. One of the many cmdlets in the PowerShell 3.0 set is the Get-NetAdapter cmdlet. This returns an object per “real” interface. The command I use is:
Get-NetAdapter | Select-Object -Property Name,MacAddress,LinkSpeed
Here is an example output from my VMware server:
Name MacAddress LinkSpeed ---- ---------- --------- Ethernet 8 8A-AF-38-2E-D8-D1 1 Gbps Ethernet 5 0A-75-BB-0D-CF-D7 1 Gbps Ethernet 7 9A-0F-47-69-CD-D8 1 Gbps Ethernet 6 EE-2D-D1-D5-58-75 1 Gbps
This is all good information that we will need to accomplish the first task in our list. If you want a correlation between the IP address and the network adapter, then you can add ifIndex to the list of properties and use the following command to get the list of IP addresses:
Get-NetIPAddress | Where-Object PrefixOrigin -ne "WellKnown"
Our network adapter information does not include the hypervisor information. For this, we need WMI information – in this case, the Win32_ComputerSystem class. This has a property called Manufacturer that follows a standard format:
Get-WmiObject –query ‘select * from Win32_ComputerSystem’ Domain : bd.splunk.com Manufacturer : Xen Model : HVM domU Name : BD-XD7-01 PrimaryOwnerName : Windows User TotalPhysicalMemory : 1069137920
This gives you a bunch of useful information, so much so that I do this query on all my Windows systems. For our purposes, I will note that Manufacturer line. This is a standard value:
| Manufacturer Value | Hypervisor |
|---|---|
| Xen | Citrix XenServer |
| VMware, Inc. | VMware ESXi |
| Microsoft Hyper-V | Microsoft Hyper-V |
If the host is not housed on a Hypervisor, then the manufacturer will be a PC manufacturer like “Lenovo”, “Dell, Inc.” or “Hewlett-Packard”. Now that we have that, we can add the hypervisor information to our network adapter information to get a combined lookup:
Get-NetAdapter | `
Select-Object Name,MacAddress,LinkSpeed | `
Add-Member -PassThru -Name HWManufacturer -Value (gwmi -query 'Select * From Win32_ComputerSystem').Manufacturer
Even better, we can correlate the hypervisor, IP Address and Mac Address together for a great correlation lookup:
Get-NetIPAddress | Where Prefix-Origin -ne "WellKnown" | `
Select IPAddress,AddressFamily, `
@{n='MacAddress';e={(Get-NetAdapter -InterfaceIndex $_.ifIndex).MacAddress}}, `
@{n='Manufacturer';e={(Get-WmiObject -query 'SELECT * FROM Win32_ComputerSystem').Manufacturer}}
This syntax may be a little unusual to the PowerShell novice. It is known as a computed property and allows you to use the results of other cmdlets (or indeed any PowerShell script) as a value in the object that is created.
Now that we have our little script ready, we can run this on a regular basis – say, at 2am each day – by adding it to an inputs.conf file:
script = Get-NetIPAddress | Where Prefix-Origin -ne "WellKnown" | Select IPAddress,AddressFamily, @{n='MacAddress';e={(Get-NetAdapter -InterfaceIndex $_.ifIndex).MacAddress}}, @{n='Manufacturer';e={(Get-WmiObject -query 'SELECT * FROM Win32_ComputerSystem').Manufacturer}}
schedule = 0 0 2 * ? *
sourcetype = PowerShell:NetAdapter
Yes – that script line needs to be typed all on the same line. You will get four fields in each event – an IP address, address family (IPv4 or IPv6), a MAC address and a manufacturer. Now you can create a lookup within Splunk for easy correlations:
sourcetype = PowerShell:NetAdapter | stats values(MacAddress) as MacAddress, values(Manufacturer) by as Manufacturer by host,IPAddress | outputlookup HostIPInformation
Turn this search into a saved search and run it every 24 hours to get the right information. Finally, we need to use this information. Let’s say you have a search that outputs an IP address and you want to know if it’s on a hypervisor, how about something like this:
`mysearch` | lookup HostIPInformation src_ip as IPAddress OUTPUT Manufacturer,MacAddress | eval IsHypervisor=case(Manufacturer=="*VMware*",true,Manufacturer=="*Xen*",true,Manufacturer=="*Hyper-V*",true,host=*,false)
You can use this information to correlate the applications running on the guest OS to the hypervisor it is in by using the Splunk App for VMware or the Splunk App for Server Virtualization.