.. _indexes_management: ##################################### Indexes customizations and management ##################################### ===================== Default indexes model ===================== **The application stack uses by default a 4 layer indexes models:** +--------------------------------+-------------------------------------------------------------------------------------+ | Index name | Purpose | +================================+=====================================================================================+ | os-unix-nmon-metrics | metric store index where metrics data is being stored | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-events | nmon data stored as regular events | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-config | nmon configuration data | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-internal | internal data generated by the application for various processing tasks | +--------------------------------+-------------------------------------------------------------------------------------+ ++++++++++++ indexes.conf ++++++++++++ **The indexes are defined in the Support Addon SA-metricator-for-nmon:** :: SA-metricator-for-nmon/default/indexes.conf +++++++++++ macros.conf +++++++++++ **The front-end application metricator-for-nmon uses macro definitions for the indexes to be used by the application:** :: metricator-for-nmon/default/macros.conf **List of indexes macros:** +--------------------------------+-------------------------------------------------------------------------------------+ | macro name | indexes target | +================================+=====================================================================================+ | nmon_index | index=os-unix-nmon-* | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon_idx | idx=os-unix-nmon-* | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon_events_index | index=os-unix-nmon-events* | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon_metrics_index | index=os-unix-nmon-metrics* | +--------------------------------+-------------------------------------------------------------------------------------+ *Wildcards are automatically replaced by any suite of characters.* +++++++++++++++ eventtypes.conf +++++++++++++++ **The front-end application metricator-for-nmon uses events definition in eventtypes.conf:** +--------------------------------+-------------------------------------------------------------------------------------+ | eventtype | indexes target | +================================+=====================================================================================+ | nmon | index=os-unix-nmon-events* OR index=os-unix-nmon-internal* | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon:events | index=os-unix-nmon-events* sourcetype=nmon_data | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon:config | index=os-unix-nmon-config* sourcetype=nmon_config | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon:collect | index=os-unix-nmon-internal* sourcetype=nmon_collect | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon:processing | index=os-unix-nmon-internal* sourcetype=nmon_processing | +--------------------------------+-------------------------------------------------------------------------------------+ | nmon:clean | index=os-unix-nmon-internal* sourcetype=nmon_clean | +--------------------------------+-------------------------------------------------------------------------------------+ | uptime | index=os-unix-nmon-events* sourcetype=nmon_data type=UPTIME | +--------------------------------+-------------------------------------------------------------------------------------+ | inventory | index=os-unix-nmon-config* sourcetype=nmon_config | +--------------------------------+-------------------------------------------------------------------------------------+ +++++++++++ inputs.conf +++++++++++ **The Technical Addon contains a default definition of indexes to be used for data indexing:** :: TA-metricator-for-nmon/default/inputs.conf **With following inputs stanza specifying their target index:** :: [script://./bin/metricator_helper.sh] index = os-unix-nmon-internal [script://./bin/metricator_consumer.sh] index = os-unix-nmon-internal [batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv] index = os-unix-nmon-metrics [batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv] index = os-unix-nmon-events [monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv] index = os-unix-nmon-config [script://./bin/metricator_cleaner.sh --cleancsv] index = os-unix-nmon-internal ======================================================== Customizing indexes names and multi-tenant configuration ======================================================== **For various reasons and purposes, you may need to customize the indexes names:** * You have your own naming convention and you wish to be able to use for the application deployment as well * You are a large company with multiple regions across the globe and you want a geographical or per environment segregation * You want to use access control based indexes segregation to allow specific populations to access specific indexes * Any combination of these The applications stack is compatible with any kind of customization. **Recommended customization scenario:** As we use a naming convention which matches indexes using wildcards by default, you are free to use any convention starting by the index names, such as: +--------------------------------+-------------------------------------------------------------------------------------+ | Default Index name | Automatic target | +================================+=====================================================================================+ | os-unix-nmon-metrics | os-unix-nmon-metrics* | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-events | os-unix-nmon-events* | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-config | os-unix-nmon-config* | +--------------------------------+-------------------------------------------------------------------------------------+ | os-unix-nmon-internal | os-unix-nmon-internal* | +--------------------------------+-------------------------------------------------------------------------------------+ +++++++++++++++++++++++++++ Using our naming convention +++++++++++++++++++++++++++ **If you are using our naming built-in naming convention then good news for you, you have absolutely nothing else to do than creating your own indexes.** Example of configuration: per environment segregation ----------------------------------------------------- Let's take the example of having 3 environments where you are deploying the application stack: - Engineering (-eng) - UAT (-uat) - Production (-prod) Because you may want to use a much higher retention for Production systems rather than Eng and UAT and have a larger period of metrics available, this is valuable to split the ingestion into multiple-indexes stacks. **You proceed the following way:** - For each of your indexer layer, prepare the stack of indexes and apply your custom indexes - Create a local version of the inputs.conf shipped with the Technical Addon and publish it to the instances managed by the deployment servers - Respect Splunk best practices in every step! **Based on the example above::** Indexing layer preparation: ^^^^^^^^^^^^^^^^^^^^^^^^^^^ **Production configuration, assuming clustered indexers.** *On cluster master:* :: cd /opt/splunk/etc/master-apps/SA-metricator-for-nmon if [ ! -d local ]; then mkdir local; fi cp -p default/indexes.conf local/ *Then achieve the following configuration, adapt if using Splunk volumes or for any other need:* :: # nmon data ingested as metrics [os-unix-nmon-metrics-prod] disabled = false coldPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/colddb datatype = metric homePath = $SPLUNK_DB/os-unix-nmon-metrics-prod/db splitByIndexKeys = metric_name,host thawedPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/thaweddb repFactor = auto # nmon data ingested as regular events [os-unix-nmon-events-prod] disabled = false coldPath = $SPLUNK_DB/os-unix-nmon-events-prod/colddb homePath = $SPLUNK_DB/os-unix-nmon-events-prod/db thawedPath = $SPLUNK_DB/os-unix-nmon-events-prod/thaweddb repFactor = auto # nmon config ingested as regular events [os-unix-nmon-config-prod] disabled = false coldPath = $SPLUNK_DB/os-unix-nmon-config-prod/colddb homePath = $SPLUNK_DB/os-unix-nmon-config-prod/db thawedPath = $SPLUNK_DB/os-unix-nmon-config-prod/thaweddb repFactor = auto # nmon internal data [os-unix-nmon-internal-prod] disabled = false coldPath = $SPLUNK_DB/os-unix-nmon-internal-prod/colddb homePath = $SPLUNK_DB/os-unix-nmon-internal-prod/db thawedPath = $SPLUNK_DB/os-unix-nmon-internal-prod/thaweddb repFactor = auto *Publish the cluster bundle* Technical Addon configuration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ *On Deployment Servers:* :: cd /opt/splunk/etc/deployment-apps/TA-metricator-for-nmon if [ ! -d local ]; then mkdir local; fi *Add the following configuration to local/inputs.conf (create the file)* :: [script://./bin/metricator_helper.sh] index = os-unix-nmon-internal-prod [script://./bin/metricator_consumer.sh] index = os-unix-nmon-internal-prod [batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv] index = os-unix-nmon-metrics-prod [batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv] index = os-unix-nmon-events-prod [monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv] index = os-unix-nmon-config-prod [script://./bin/metricator_cleaner.sh --cleancsv] index = os-unix-nmon-internal-prod *Achieve the same configuration on any server running the TA-metricator-for-nmon and not managed by Deployment Servers (cluster master, Deployment Servers...)* *Reload or restart the Deployment Servers to publish this new configuration to managed servers* **And finally repeat the operation for each environment that needs to be managed.** **It is not required to achieve any modification on the front-end user application since the default configuration takes in charge any index respecting the naming convention using wildcards definitions.** +++++++++++++++++++++++++++++++++++ Using a different naming convention +++++++++++++++++++++++++++++++++++ If the builtin indexes naming convention is not option for you, it is still possible to customize the different applications to match your needs. However for easy of use and maintenance, we would recommend using the builtin naming convention when possible to avoid additional steps in your deployments. **You will proceed as following:** - Customize indexes in local/indexes.conf (Indexer layer in SA-metricator-for-nmon) - Customize macros in local/macros.conf (Search Head layer in metricator-for-nmon) - Customize eventtypes in local/eventtypes.conf (Search Head layer in metricator-for-nmon) - Customize inputs.conf in local/inputs.conf (Technical Addon in TA-metricator-for-nmon)