======================================== Deployment of the TA-metricator-for-nmon ======================================== .. _standalone_technical_Add-on: ------------ Requirements ------------ Operating system ^^^^^^^^^^^^^^^^ **The Technical Add-on is compatible with:** - Linux OS X86 in 32/64 bits, PowerPC (PowerLinux), s390x (ZLinux), ARM - IBM AIX 7.1 and 7.2 - Oracle Solaris 11 Third party software and libraries ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ To operate as expected, the Technical Add-on requires a Python **or** a Perl environment available on the server: **Python environment: used in priority** .. hint:: Python 3 support - From the release 1.1.0 of the Add-ons, Python 3.x is required (unless using Perl) - The last release supporting Python 2.x is the release 1.0.11 +--------------------------------------------+----------------------+ | Requirement | Version | | | | +============================================+======================+ | Python interpreter | 3.x | +--------------------------------------------+----------------------+ **Perl environment: used only in fallback** +--------------------------------------------+----------------------+ | Requirement | Version | | | | +============================================+======================+ | Perl interpreter | 5.x | +--------------------------------------------+----------------------+ | Time::HiRes module | any | +--------------------------------------------+----------------------+ | Text::CSV or Text::CSV_XS module | any | +--------------------------------------------+----------------------+ **Notes:** - IBM AIX does not generally contain Python. Nevertheless, Perl is available as a standard and the Technical Add-on has the Perl "Text::CSV" module built-in. More, Time::HiRes is part of Perl core modules. - Modern Linux distribution generally have Python available and do not require any further action. - Linux distributions lacking Python will fallback to Perl and must satisfy the Perl modules requirements. - If running on a full Splunk instance (any Splunk dedicated machine running Splunk Enterprise), the Technical Add-on uses Splunk built-in Python interpreter. ---------- Deployment ---------- **The TA-metricator-for-nmon can be deployed to any full Splunk instance or Universal Forwarder instances.** The technical Add-on should be deployed to the regular Splunk directory for application: :: $SPLUNK_HOME/etc/apps *where $SPLUNK_HOME refers to the root directory of the Splunk installation* The Technical Add-on uses relative paths referring to $SPLUNK_HOME, as such it is fully compatible with any deployment where $SPLUNK_HOME refers to a custom directory for your installation. Deployment by Splunk deployment server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **The TA-metricator-for-nmon can be deployed by any Splunk deployment server:** *Upload the tgz archive on your deployment server in a temporary directory, example:* :: cd /tmp/ *The Support Add-on tgz archive must be uncompressed and installed in $SPLUNK_HOME/etc/deployment-server:* :: cd /opt/splunk/etc/deployment-server/ tar -xvzf /tmp/TA-metricator-for-nmon_*.tar.gz *If you have any customization required, create a local directory and configure your settings in local/ configuration files.* **Finally, create a serverclass or add the TA-metricator-for-nmon application into existing serverclass, required parameters are:** - Enable App - Restart Splunkd **There are no additional configuration actions required, the monitoring inputs are activated by default and the Technical Add-on will start as soon as it is deployed and splunkd has been restarted** Deployment by any configuration management solution ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The Technical Add-on can be deployed by any configuration management product such as Ansible, Chef or Pupet. Steps are the same than for a deployment by Splunk deployment server and the configuration management solution must ensure to issue a proper restart of the Splunk instance after the Technical Add-on deployment. What happens once the Technical Add-on has been deployed ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Once the technical Add-on has been deployed, and the Splunk instance restarted, the following actions are taken automatically:** Fifo reader processes and Nmon processes startup ------------------------------------------------ **At startup time, Splunk will automatically trigger the execution of the "bin/metricator_helper.sh" script.** **This script does several actions, such as:** - Identifying the operating system and its sub-version - For Linux OS, locally extracting the "bin/linux.tgz" archive if existing and if first deployment/upgrade - starting the fifo_reader processes - starting the nmon binary according to the guest Operating System and configuration settings **The script activity is available in:** - standard output: :: eventtype=nmon:collect host= - error output: :: index=_internal sourcetype=splunkd host= error metricator_helper.sh Running processes in machine ---------------------------- Several processes can be found in machine, at initial run you will find fifo_reader processes (output might differ specially for paths): *Using Python interpreter: (Universal Forwarder example)* :: python /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_reader.py --fifo fifo1 /bin/sh -c /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_reader.sh /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo /bin/sh /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_reader.sh /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo *Using Perl interpreter: (Universal Forwarder example)* :: /usr/bin/perl /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_reader.pl --fifo fifo1 /bin/sh /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon/bin/metricator_reader.sh /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo *The startup operation will be visible by a message logged:* :: eventtype=nmon:collect starting fifo_reader *Example:* :: 12-02-2018 05:12:14, sys-91371.dal-ebis.ihost.com INFO: starting the fifo_reader fifo1 In addition, you will find an nmon binary instance running, example: (output will differ depending on operating systems and settings) :: /opt/splunkforwarder/var/log/metricator/bin/linux/rhel/nmon_power_64_rhel6_be -F /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo -T -s 60 -c 1440 -d 1500 -g auto -D -p *The startup operation will be visible by a message logged:* :: eventtype=nmon:collect starting nmon *Example:* :: 12-02-2018 05:12:15, sys-91371.dal-ebis.ihost.com INFO: starting nmon : /opt/splunkforwarder/var/log/metricator/bin/linux/sles/nmon_power_64_sles12_le -F /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1/nmon.fifo -T -s 60 -c 1440 -d 1500 -g auto -D -p in /opt/splunkforwarder/var/log/metricator/var/nmon_repository/fifo1 Nmon data processing -------------------- **The Nmon data processing is achieved every minute by the script "metricator_consumer.sh"** Its activity is indexed in Splunk, and available via the following search: :: eventtype=nmon:processing host= *Example:* :: 12-02-2018 09:50:02 Reading NMON data: 440 lines 26766 bytes Splunk Root Directory ($SPLUNK_HOME): /opt/splunkforwarder Add-on type: /opt/splunkforwarder/etc/apps/TA-metricator-for-nmon Add-on version: 1.0.0 nmonparser version: 2.0.0 Guest Operating System: linux Python version: 2.7.5 HOSTNAME: sys-91367.dal-ebis.ihost.com NMON VERSION: 16f TIME of Nmon Data: 05:11.54 DATE of Nmon data: 12-FEB-2018 INTERVAL: 60 SNAPSHOTS: 1440 logical_cpus: 1 NMON OStype: Linux virtual_cpus: 1 SerialNumber: PPD-Linux NMON ID: 12-FEB-2018:05:11.54,sys-91367.dal-ebis.ihost.com,PPD-Linux,26766,1518430314,1518446953 ANALYSIS: Enforcing fifo mode using --mode option Starting_epochtime: 1518430314 Ending_epochtime: 1518446953 last known epoch time: 0 CONFIG section: will not be extracted (time delta of 66282 seconds is inferior to 86400 seconds) Output mode is configured to run in minimal mode using the --silent option Elapsed time was: 0.188985 seconds Splunk indexing --------------- **Once the data processing steps have been achieved, several csv flow files are generated and consumed by Splunk in batch mode. (index and delete)** The traces of these activities are visible in Splunk using the following search: :: index=_internal sourcetype=splunkd host= batch input *Example:* :: 02-12-2018 10:01:09.073 -0500 INFO TailReader - Batch input finished reading file='/opt/splunkforwarder/var/log/metricator/var/csv_repository/sys-91367.dal-ebis.ihost.com_01_DGBACKLOG.metrics.csv' 02-12-2018 10:01:09.073 -0500 INFO TailReader - Batch input finished reading file='/opt/splunkforwarder/var/log/metricator/var/csv_repository/sys-91367.dal-ebis.ihost.com_01_DGIOTIME.metrics.csv' 02-12-2018 10:01:09.072 -0500 INFO TailReader - Batch input finished reading file='/opt/splunkforwarder/var/log/metricator/var/csv_repository/sys-91367.dal-ebis.ihost.com_01_DGWRITESERV.metrics.csv' 02-12-2018 10:01:09.072 -0500 INFO TailReader - Batch input finished reading file='/opt/splunkforwarder/var/log/metricator/var/csv_repository/sys-91367.dal-ebis.ihost.com_01_DGWRITEMERGE.metrics.csv' 02-12-2018 10:01:09.071 -0500 INFO TailReader - Batch input finished reading file='/opt/splunkforwarder/var/log/metricator/var/csv_repository/sys-91367.dal-ebis.ihost.com_01_DGWRITES.metrics.csv' Immediately after the files consumption, the metrics and events are available in Splunk. For trouble shooting and any advanced configuration purposes, please consult the different pages of this documentation.