Let’s proceed in the order, we want first to get Splunk ready to receive Windows performance data.<\/strong><\/p>\n This is quite simple and\u00a0relies on deploying the Windows technical add-on built by Splunk:<\/p>\n https:\/\/splunkbase.splunk.com\/app\/742\/<\/a><\/p>\n Depending on your Splunk architecture, ensure to deploy the technical add-on everywhere it is required: The add-on\u00a0has no inputs activated by default, at this point there are no modification required.<\/p>\n Indexes creation:<\/strong><\/p>\n The Windows technical add-on contains the embedded definition for a few indexes:<\/p>\n If you are running a standalone server then you have nothing else to do as the indexes have created for you by the add-on.<\/p>\n If \u00a0you are running Splunk with clustered indexers, be sure to declare those indexes properly before continuing the setup.<\/p>\n For the demonstration purpose, we\u00a0will assume that:<\/strong><\/p>\n Deploy the technical add-on as usual and continue the setup.<\/strong><\/p>\n Let’s have a look at the default “inputs.conf” file provided within the technical add-on, since we focus\u00a0on performance metric, we are only interested for now in the “perfmon” stanzas.<\/p>\n For the the demonstration purposes, let’s have a look at CPU and memory metrics:<\/strong><\/p>\n Splunk_TA_windows\/default\/inputs.conf<\/em><\/p>\n Things you will (should) probably want to customise:<\/strong><\/p>\n Let’s\u00a0with the following configuration, as always do never modify a default file, create a local file and copy only the stanzas you are interested in:<\/strong><\/p>\n Splunk_TA_windows\/local\/inputs.conf<\/em><\/p>\n Deploy this configuration to your Windows servers, and if you use Splunk deployment server, ensure you check “restart splunkd”.<\/strong><\/p>\n Next step, let’s check for some data coming in:<\/strong><\/p>\n Depending on the mode (multikv or not), the data will be available in:<\/strong><\/p>\n CPU statistics:<\/em><\/p>\n Memory statistics:<\/em><\/p>\n In this article, we will go will the multikv mode.<\/strong><\/p>\n Let’s get some CPU statistics:<\/strong><\/p>\n Per host average CPU usage over time:\u00a0<\/em><\/p>\n Very simple.<\/p>\n If you are “like me”, when looking at memory statistics, the first (and potentially the only) metric you want to be able to retrieve is the percentage of memory being used, or eventually memory free.<\/p>\n So what’s the problem then ? Well, “as it” although we have dozens of various metrics, the percentage of utilisation\u00a0is not available with perfmon data.<\/p>\n What ???<\/strong><\/p>\n Hopefully, we can calculate it ! Using Splunk power and features, we can correlate between the inventory data which contains the amount of physical memory available, and the memory metrics available in\u00a0perfmon.<\/p>\n The following search reports the amount of physical memory in KB:<\/strong><\/p>\n Notes:<\/strong><\/p>\n This requires the input “OperatingSystem” to be activated in your deployment, using:<\/p>\n For the demonstration, let’s store this result in\u00a0a temporarily lookup csv file:<\/strong><\/p>\n Then, looking at the memory statistics, we have the amount of currently used volume of memory in KB, let’s map this with the inventory data and use some easy calculation:<\/strong><\/p>\n<\/div>\n There you go!<\/p>\n Resilient solution:<\/strong><\/p>\n Understanding a system CPU load requires knowing\u00a0what and when the processes consumes resources, the perfmon provides processes related data with the “[perfmon:\/\/Process]” stanza.<\/p>\n However, for some reasons the perfmon data is not accurate on multi core systems, a nice article gave me the answer I was looking for:<\/strong><\/p>\n Windows CPU monitoring with Splunk<\/a><\/p>\n Based on this great article, let’s add our WMI input to generate accurate processes CPU statistics: (caution: this is a “wmi.conf” and not “inputs.conf”)<\/strong><\/p>\n Splunk_TA_windows\/local\/wmi.conf<\/em><\/p>\n Once deployed, let’s use some magic searches and start analysing processes activity:<\/strong><\/p>\n
\n<\/b><\/p>\n\n
\n
DEPLOYING AND CONFIGURING THE ADD-ON<\/span><\/h2>\n
\n
COLLECTING PERFORMANCE DATA<\/span><\/h2>\n
## CPU\r\n[perfmon:\/\/CPU]\r\ncounters = % Processor Time; % User Time; % Privileged Time; Interrupts\/sec; % DPC Time; % Interrupt Time; DPCs Queued\/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions\/sec; C2 Transitions\/sec; C3 Transitions\/sec\r\ndisabled = 1\r\ninstances = *\r\ninterval = 10\r\nobject = Processor\r\nuseEnglishOnly=true\r\nindex = perfmon\r\n\r\n## Memory\r\n[perfmon:\/\/Memory]\r\ncounters = Page Faults\/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies\/sec; Transition Faults\/sec; Cache Faults\/sec; Demand Zero Faults\/sec; Pages\/sec; Pages Input\/sec; Page Reads\/sec; Pages Output\/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes\/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed\/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)\r\ndisabled = 1\r\ninterval = 10\r\nobject = Memory\r\nuseEnglishOnly=true\r\nindex = perfmon\r\n<\/pre>\n
\n
## CPU\r\n[perfmon:\/\/CPU]\r\ncounters = % Processor Time; % User Time; % Privileged Time; Interrupts\/sec; % DPC Time; % Interrupt Time; DPCs Queued\/sec; DPC Rate; % Idle Time; % C1 Time; % C2 Time; % C3 Time; C1 Transitions\/sec; C2 Transitions\/sec; C3 Transitions\/sec\r\ndisabled = 0\r\ninstances = *\r\ninterval = 30\r\nobject = Processor\r\nuseEnglishOnly=true\r\nindex = perfmon\r\nmode = multikv\r\n\r\n## Memory\r\n[perfmon:\/\/Memory]\r\ncounters = Page Faults\/sec; Available Bytes; Committed Bytes; Commit Limit; Write Copies\/sec; Transition Faults\/sec; Cache Faults\/sec; Demand Zero Faults\/sec; Pages\/sec; Pages Input\/sec; Page Reads\/sec; Pages Output\/sec; Pool Paged Bytes; Pool Nonpaged Bytes; Page Writes\/sec; Pool Paged Allocs; Pool Nonpaged Allocs; Free System Page Table Entries; Cache Bytes; Cache Bytes Peak; Pool Paged Resident Bytes; System Code Total Bytes; System Code Resident Bytes; System Driver Total Bytes; System Driver Resident Bytes; System Cache Resident Bytes; % Committed Bytes In Use; Available KBytes; Available MBytes; Transition Pages RePurposed\/sec; Free & Zero Page List Bytes; Modified Page List Bytes; Standby Cache Reserve Bytes; Standby Cache Normal Priority Bytes; Standby Cache Core Bytes; Long-Term Average Standby Cache Lifetime (s)\r\ndisabled = 0\r\ninterval = 30\r\nobject = Memory\r\nuseEnglishOnly=true\r\nindex = perfmon\r\nmode = multikv\r\n<\/pre>\n
CHECKING DATA COMING IN<\/span><\/h2>\n
index=perfmon<\/pre>\n
![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img1.png\")
<\/a><\/p>\n\n
\n
ANALYSING CPU STATISTICS<\/span><\/h2>\n
index=perfmon sourcetype=\"PerfmonMk:CPU\" instance=_Total\r\n| timechart avg(%_Processor_Time) as cpu_usage by host<\/pre>\n
![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img2.png\")
<\/a><\/p>\nWHERE IS MY PERCENTAGE OF MEMORY UTILISATION ?<\/span><\/h2>\n
![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img3-212x300.jpg\")
<\/a><\/p>\nindex=windows sourcetype=WinHostMon\r\n| stats latest(TotalPhysicalMemoryKB) as TotalPhysicalMemoryKB, latest(TotalVirtualMemoryKB) as TotalVirtualMemoryKB by host | sort 0 host\r\n<\/pre>\n<\/div>\n
[WinHostMon:\/\/OperatingSystem]\r\ninterval = 600\r\ndisabled = 1\r\ntype = OperatingSystem\r\nindex = windows\r\n<\/pre>\n
index=windows sourcetype=WinHostMon\r\n| stats latest(TotalPhysicalMemoryKB) as TotalPhysicalMemoryKB, latest(TotalVirtualMemoryKB) as TotalVirtualMemoryKB by host | sort 0 host\r\n| outputlookup windows_memory_inventory.csv<\/pre>\n
index=perfmon sourcetype=\"PerfmonMk:Memory\"\r\n| eval used_memory_KB=coalesce('Available_KBytes', Value)\r\n| lookup windows_memory_inventory.csv host as host OUTPUTNEW TotalPhysicalMemoryKB\r\n| eval free_memory_pct=((used_memory_KB\/TotalPhysicalMemoryKB)*100), used_memory_pct=(100-free_memory_pct)\r\n| timechart avg(used_memory_pct) as used_memory_pct by host\r\n<\/pre>\n![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img3.png\")
<\/a><\/p>\n\n
WHAT ABOUT PROCESSES ?<\/span><\/h2>\n
[WMI:process]\r\nindex = windows\r\ndisabled = 0\r\ninterval = 30\r\nwql = Select IDProcess,Name,PercentProcessorTime,TimeStamp_Sys100NS from Win32_PerfRawData_PerfProc_Process\r\n<\/pre>\n
index=windows sourcetype=\"WMI:process\" Name!=_Total Name!=Idle\r\n| reverse | streamstats current=f last(PercentProcessorTime) as last_PercentProcessorTime last(Timestamp_Sys100NS) as last_Timestamp_Sys100NS by Name\r\n| eval cputime = 100 * (PercentProcessorTime - last_PercentProcessorTime) \/ (Timestamp_Sys100NS - last_Timestamp_Sys100NS)\r\n| search cputime > 0\r\n| timechart limit=50 useother=f avg(cputime) by Name\r\n<\/pre>\n
![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img4.png\")
<\/a><\/p>\n![\"\"](\"https:\/\/51.68.196.81\/octamis-blog\/wp-content\/uploads\/2017\/05\/img5.png\")
<\/a><\/p>\n