Indexes customizations and management

Default indexes model

The application stack uses by default a 4 layer indexes models:

Index name

Purpose

os-unix-nmon-metrics

metric store index where metrics data is being stored

os-unix-nmon-events

nmon data stored as regular events

os-unix-nmon-config

nmon configuration data

os-unix-nmon-internal

internal data generated by the application for various processing tasks

indexes.conf

The indexes are defined in the Support Addon SA-metricator-for-nmon:

SA-metricator-for-nmon/default/indexes.conf

macros.conf

The front-end application metricator-for-nmon uses macro definitions for the indexes to be used by the application:

metricator-for-nmon/default/macros.conf

List of indexes macros:

macro name

indexes target

nmon_index

index=os-unix-nmon-*

nmon_idx

idx=os-unix-nmon-*

nmon_events_index

index=os-unix-nmon-events*

nmon_metrics_index

index=os-unix-nmon-metrics*

Wildcards are automatically replaced by any suite of characters.

eventtypes.conf

The front-end application metricator-for-nmon uses events definition in eventtypes.conf:

eventtype

indexes target

nmon

index=os-unix-nmon-events* OR index=os-unix-nmon-internal*

nmon:events

index=os-unix-nmon-events* sourcetype=nmon_data

nmon:config

index=os-unix-nmon-config* sourcetype=nmon_config

nmon:collect

index=os-unix-nmon-internal* sourcetype=nmon_collect

nmon:processing

index=os-unix-nmon-internal* sourcetype=nmon_processing

nmon:clean

index=os-unix-nmon-internal* sourcetype=nmon_clean

uptime

index=os-unix-nmon-events* sourcetype=nmon_data type=UPTIME

inventory

index=os-unix-nmon-config* sourcetype=nmon_config

inputs.conf

The Technical Addon contains a default definition of indexes to be used for data indexing:

TA-metricator-for-nmon/default/inputs.conf

With following inputs stanza specifying their target index:

[script://./bin/metricator_helper.sh]
index = os-unix-nmon-internal

[script://./bin/metricator_consumer.sh]
index = os-unix-nmon-internal

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv]
index = os-unix-nmon-metrics

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv]
index = os-unix-nmon-events

[monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv]
index = os-unix-nmon-config

[script://./bin/metricator_cleaner.sh --cleancsv]
index = os-unix-nmon-internal

Customizing indexes names and multi-tenant configuration

For various reasons and purposes, you may need to customize the indexes names:

  • You have your own naming convention and you wish to be able to use for the application deployment as well

  • You are a large company with multiple regions across the globe and you want a geographical or per environment segregation

  • You want to use access control based indexes segregation to allow specific populations to access specific indexes

  • Any combination of these

The applications stack is compatible with any kind of customization.

Recommended customization scenario:

As we use a naming convention which matches indexes using wildcards by default, you are free to use any convention starting by the index names, such as:

Default Index name

Automatic target

os-unix-nmon-metrics

os-unix-nmon-metrics*

os-unix-nmon-events

os-unix-nmon-events*

os-unix-nmon-config

os-unix-nmon-config*

os-unix-nmon-internal

os-unix-nmon-internal*

Using our naming convention

If you are using our naming built-in naming convention then good news for you, you have absolutely nothing else to do than creating your own indexes.

Example of configuration: per environment segregation

Let’s take the example of having 3 environments where you are deploying the application stack:

  • Engineering (-eng)

  • UAT (-uat)

  • Production (-prod)

Because you may want to use a much higher retention for Production systems rather than Eng and UAT and have a larger period of metrics available, this is valuable to split the ingestion into multiple-indexes stacks.

You proceed the following way:

  • For each of your indexer layer, prepare the stack of indexes and apply your custom indexes

  • Create a local version of the inputs.conf shipped with the Technical Addon and publish it to the instances managed by the deployment servers

  • Respect Splunk best practices in every step!

Based on the example above::

Indexing layer preparation:

Production configuration, assuming clustered indexers.

On cluster master:

cd /opt/splunk/etc/master-apps/SA-metricator-for-nmon
if [ ! -d local ]; then mkdir local; fi
cp -p default/indexes.conf local/

Then achieve the following configuration, adapt if using Splunk volumes or for any other need:

# nmon data ingested as metrics
[os-unix-nmon-metrics-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/colddb
datatype = metric
homePath = $SPLUNK_DB/os-unix-nmon-metrics-prod/db
splitByIndexKeys = metric_name,host
thawedPath = $SPLUNK_DB/os-unix-nmon-metrics-prod/thaweddb
repFactor = auto

# nmon data ingested as regular events
[os-unix-nmon-events-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-events-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-events-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-events-prod/thaweddb
repFactor = auto

# nmon config ingested as regular events
[os-unix-nmon-config-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-config-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-config-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-config-prod/thaweddb
repFactor = auto

# nmon internal data
[os-unix-nmon-internal-prod]
disabled = false
coldPath = $SPLUNK_DB/os-unix-nmon-internal-prod/colddb
homePath = $SPLUNK_DB/os-unix-nmon-internal-prod/db
thawedPath = $SPLUNK_DB/os-unix-nmon-internal-prod/thaweddb
repFactor = auto

Publish the cluster bundle

Technical Addon configuration

On Deployment Servers:

cd /opt/splunk/etc/deployment-apps/TA-metricator-for-nmon
if [ ! -d local ]; then mkdir local; fi

Add the following configuration to local/inputs.conf (create the file)

[script://./bin/metricator_helper.sh]
index = os-unix-nmon-internal-prod

[script://./bin/metricator_consumer.sh]
index = os-unix-nmon-internal-prod

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*metrics.csv]
index = os-unix-nmon-metrics-prod

[batch://$SPLUNK_HOME/var/log/metricator/var/csv_repository/*events.csv]
index = os-unix-nmon-events-prod

[monitor://$SPLUNK_HOME/var/log/metricator/var/config_repository/*events.csv]
index = os-unix-nmon-config-prod

[script://./bin/metricator_cleaner.sh --cleancsv]
index = os-unix-nmon-internal-prod

Achieve the same configuration on any server running the TA-metricator-for-nmon and not managed by Deployment Servers (cluster master, Deployment Servers…)

Reload or restart the Deployment Servers to publish this new configuration to managed servers

And finally repeat the operation for each environment that needs to be managed.

It is not required to achieve any modification on the front-end user application since the default configuration takes in charge any index respecting the naming convention using wildcards definitions.

Using a different naming convention

If the builtin indexes naming convention is not option for you, it is still possible to customize the different applications to match your needs.

However for easy of use and maintenance, we would recommend using the builtin naming convention when possible to avoid additional steps in your deployments.

You will proceed as following:

  • Customize indexes in local/indexes.conf (Indexer layer in SA-metricator-for-nmon)

  • Customize macros in local/macros.conf (Search Head layer in metricator-for-nmon)

  • Customize eventtypes in local/eventtypes.conf (Search Head layer in metricator-for-nmon)

  • Customize inputs.conf in local/inputs.conf (Technical Addon in TA-metricator-for-nmon)