{"id":520,"date":"2025-09-12T13:20:58","date_gmt":"2025-09-12T12:20:58","guid":{"rendered":"https:\/\/www.octamis.com\/octamis-blog\/?p=520"},"modified":"2025-09-15T08:44:46","modified_gmt":"2025-09-15T07:44:46","slug":"splunk-v10-remote-upgrader","status":"publish","type":"post","link":"https:\/\/www.octamis.com\/octamis-blog\/splunk-v10-remote-upgrader\/","title":{"rendered":"Splunk v10 Remote Upgrader"},"content":{"rendered":"\n<ol>\n<li><strong>Introduction<\/strong>\n<ul>\n<li>Limitations<\/li>\n\n\n\n<li>New naming convention<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Installation \/ Configuration<\/strong>\n<ul>\n<li>Addon installation<\/li>\n\n\n\n<li>Prepare and upgrade\n<ul>\n<li>Upgrade (or downgrade) from one version to another<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>How does the Remote Upgrader work ?<\/strong>\n<ul>\n<li>Upgrade \/ Downgrade tests<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Conclusion<\/strong><\/li>\n<\/ol>\n\n\n\n<h2 class=\"wp-block-heading\">Introduction<\/h2>\n\n\n\n<p>Here is a quick review of the Remote Upgrader Feature provided with the brand new Splunk version 10, that was released in July.<\/p>\n\n\n\n<p>This new feature allows you to upgrade (or downgrade) your Agents distributed across your Splunk infrastructure managed by your Agent Management, formerly known as Deployment Server.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Limitations<\/h3>\n\n\n\n<p>There are not a lot of limitations, but still:<\/p>\n\n\n\n<ul>\n<li>it is limited to Splunk Universal Forwarder, version 8 to 10<\/li>\n\n\n\n<li>supported only on Linux platforms<\/li>\n\n\n\n<li>needs a specific addon and a specific configuration<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">New Naming Convention<\/h3>\n\n\n\n<p>A good to know information, if you didn&#8217;t notice it: the <strong>Deployment Server <\/strong>and <strong>Clients <\/strong>are now renamed as <strong>Agent Management<\/strong> and <strong>Agents<\/strong>.<\/p>\n\n\n\n<p>You can find this information here: <a href=\"https:\/\/help.splunk.com\/en\/splunk-enterprise\/administer\/update-your-deployment\/10.0\/agent-management\/about-agent-management\">https:\/\/help.splunk.com\/en\/splunk-enterprise\/administer\/update-your-deployment\/10.0\/agent-management\/about-agent-management<\/a><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Installation \/ Configuration<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Addon installation<\/h3>\n\n\n\n<p>First of all, as this feature is depending on an addon, you have to download it from Splunkbase:<\/p>\n\n\n\n<ol start=\"1\">\n<li>Download and untar the Remote Upgrader for Linux Universal Forwarders on&nbsp;<a href=\"https:\/\/splunkbase.splunk.com\/app\/7699\" target=\"_blank\" rel=\"noreferrer noopener\">splunkbase.com<\/a>.<\/li>\n\n\n\n<li>On the Agent Management, copy the package from the default folder <br><code><strong>\/opt\/splunk\/etc\/deployment-apps\/splunk_app_uf_remote_upgrade_linux\/default\/packages\/splunk-upgrader-{version}.tgz<\/strong><\/code> and\u00a0<br><code><strong>\/opt\/splunk\/etc\/deployment-apps\/splunk_app_uf_remote_upgrade_linux\/default\/packages\/splunk-upgrader-{version}.tgz.sig<\/strong><\/code>\u00a0<br>to the local folder\u00a0<br><code><strong>\/opt\/splunk\/etc\/deployment-apps\/splunk_app_uf_remote_upgrade_linux\/local\/packages\/<\/strong><\/code><\/li>\n\n\n\n<li>Deploy the app by affecting it to a Server Class.<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-verse\"><strong>NOTE:<\/strong> the remote upgrader for Linux Universal Forwarder <strong>is not a Splunk add-on<\/strong>. It's a mechanism you use to deliver the Agent package and\/or the Remote Upgrader package to remote Agent boxes. It runs as a separate Linux service outside of the Splunk home directory. <br><strong>It always requires ROOT to install<\/strong>, so Agent and Agent Management cannot install the remote upgrader.<\/pre>\n\n\n\n<p>You can install the Remote Upgrader for Linux as an existing user or by creating a new one.<\/p>\n\n\n\n<p>You have to untar the the app you previously copied to the local\/packages directory (in step 2):<\/p>\n\n\n\n<p><code><strong>tar zxvf \/opt\/splunkforwarder\/etc\/apps\/splunk_app_uf_remote_upgrade_linux\/local\/packages\/splunk-upgrader-linux-102.tgz -C \/var\/tmp<\/strong><\/code><\/p>\n\n\n\n<p>You can untar in the directory you want, here <code><strong>\/var\/tmp<\/strong><\/code>, then go to that directory and run the following command:<\/p>\n\n\n\n<p><strong><code>\/var\/tmp#  sudo .\/bin\/install.sh --accept-license --create-user<\/code><br><code>\/var\/tmp# sudo systemctl start splunk-upgrader<\/code><\/strong><\/p>\n\n\n\n<p>You can verify the status:&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"1002\" height=\"243\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-29.png\" alt=\"\" class=\"wp-image-543\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-29.png 1002w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-29-300x73.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-29-768x186.png 768w\" sizes=\"(max-width: 1002px) 100vw, 1002px\" \/><\/figure>\n\n\n\n<p>The setup is then finsihed.<\/p>\n\n\n\n<pre class=\"wp-block-verse\">NOTE for Docker users: this addon cannot be tested in <a href=\"https:\/\/github.com\/splunk\/docker-splunk\/tree\/develop\" target=\"_blank\" rel=\"noreferrer noopener\">Splunk dockers<\/a> as it requires systemctl. If you want to test with Docker, I advise you to use a Ubuntu docker for the forwarder, and configure it to use systemctl, then install a Splunk Agent on it and connect it to the Agent Management.<\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Prepare and upgrade<\/h3>\n\n\n\n<p>You can set up delivery using an Agent Management or a third party delivery product as follows:<\/p>\n\n\n\n<p><a href=\"https:\/\/www.splunk.com\/en_us\/download\/previous-releases-universal-forwarder.html\" target=\"_blank\" rel=\"noreferrer noopener\">Download the universal forwarder package and signature<\/a> and insert them into the delivery app at <code>.\/local\/packages\/<\/code> dir, then deploy it with the <s>Deployment Server<\/s> Agent Management.<\/p>\n\n\n\n<ul>\n<li>Update your <code>local_config<\/code> and deploy it using the Agent Management. See\u00a0<a href=\"https:\/\/help.splunk.com\/splunk-cloud-platform\/forward-and-process-data\/splunk-remote-upgrader-for-linux-universal-forwarders\/9.0\/configure\/modify-remote-upgrader-using-the-configuration-files#id_88f73335_233d_4949_8d15_394d4046b6f0__Modify_remote_upgrader_using_the_configuration_files\" target=\"_blank\" rel=\"noreferrer noopener\">Modify remote upgrader using the configuration files<\/a>\u00a0for more information.<\/li>\n<\/ul>\n\n\n\n<ul>\n<li>Use the Agent Management to deliver the Splunk Remote Upgrader for Linux package.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Upgrade (or downgrade) from one version to another<\/h4>\n\n\n\n<p>You have to download from Splunk Website the Universal Forwarder package. you also need to download the x509 signature for the package you choose.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"206\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-20.png\" alt=\"\" class=\"wp-image-523\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-20.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-20-300x65.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-20-768x167.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>Downlad the X509 certificate that you will rename as the name of the packet +.sig at the end. If you download the splunkforwarder-9.2.6-bfd122d7f8fc-linux-2.6-amd64.deb package, then create the splunkforwarder-9.2.6-bfd122d7f8fc-linux-2.6-amd64.deb.sig file and put the X509 certificate inside. You can get the Certificate on the same page as your Splunk Agent download, by clicking on the \u201cMore\u201d button.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">How does the Remote Upgrader work ?<\/h3>\n\n\n\n<p>Place these 2 files under <code>local\/packages<\/code> of the splunk_app_upgrader_delivery (under <code>\/opt\/splunk\/etc\/deployment-apps<\/code>).<\/p>\n\n\n\n<p>Then to start the upgrade, just trigger a new delivery of the app to the Agent.<\/p>\n\n\n\n<p><code><strong>\/opt\/splunk\/bin\/splunk reload deploy-server<\/strong><\/code><\/p>\n\n\n\n<p>Then, behind the scenes, Splunk copies the new Agent package\u00a0to the <code>\/tmp\/SPLUNK_UPDATER_MONITORED_DIR<\/code> directory and creates a file named \u201c<code>start_uf_upgrade<\/code>\u201d which triggers the update of the Agent.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Upgrade\/downgrade tests<\/h3>\n\n\n\n<p>Example of the <code>\/tmp\/SPLUNK_UPDATER_MONITORED_DIR<\/code> directory during an upgrade\/downgrade (here version 9.2.4):<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"150\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-21.png\" alt=\"\" class=\"wp-image-525\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-21.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-21-300x48.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-21-768x122.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>If you have 2 different versions of the Agent package, apparently only the last one is considered (to be verified).<\/p>\n\n\n\n<p>The Agent is then updated and some minutes after you have your Agent version upgraded in the Agent Management (Deployment Server):<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"79\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-22.png\" alt=\"\" class=\"wp-image-526\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-22.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-22-300x25.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-22-768x64.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>You can also downgrade your Agent by putting a previous version in the local\/packages directory. On the previous screenshot, the version 9.2.6 was just upgraded from 9.2.4.<\/p>\n\n\n\n<p>For testing purposes, I replaced the 9.2.6 with the 9.2.4 in the <code>local\/packages<\/code> and triggered a new deployment with <code>splunk reload deploy-server<\/code> on the Agent Management.<\/p>\n\n\n\n<p><strong>Downgrade :<\/strong><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"291\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-23.png\" alt=\"\" class=\"wp-image-527\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-23.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-23-300x92.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-23-768x236.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-full is-resized\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"100\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-24.png\" alt=\"\" class=\"wp-image-528\" style=\"width:756px;height:auto\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-24.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-24-300x32.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-24-768x81.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>The version was replaced, but pending. It took time but finally got it back to OK.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"81\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-25.png\" alt=\"\" class=\"wp-image-529\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-25.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-25-300x26.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-25-768x66.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p><strong>New upgrade<\/strong><\/p>\n\n\n\n<p>I decided to upgrade again, to version 9.2.7.<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"98\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-26.png\" alt=\"\" class=\"wp-image-531\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-26.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-26-300x31.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-26-768x80.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>I put the right packages in <code>local\/packages<\/code> then triggered a new upgrade (with the tgz file this time, instead of .deb package). But I got the following error message in the logs:<\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"208\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-27.png\" alt=\"\" class=\"wp-image-532\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-27.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-27-300x66.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-27-768x169.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<p>So it is mandatory to<strong> keep the same type of package<\/strong> you used to upgrade the Agent at the beginning. <span style=\"font-size: revert\">All logs about the upgrade is available with that query:<\/span><\/p>\n\n\n\n<figure class=\"wp-block-image size-full\"><img loading=\"lazy\" decoding=\"async\" width=\"945\" height=\"49\" src=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-28.png\" alt=\"\" class=\"wp-image-533\" srcset=\"https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-28.png 945w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-28-300x16.png 300w, https:\/\/www.octamis.com\/octamis-blog\/wp-content\/uploads\/2025\/09\/image-28-768x40.png 768w\" sizes=\"(max-width: 945px) 100vw, 945px\" \/><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>This Remote Upgrader is working well. It requires ROOT permissions to be installed and a specific or existing user to operate. But it is working well. You can find additional information on the <a href=\"https:\/\/help.splunk.com\/en\/splunk-cloud-platform\/forward-and-process-data\/splunk-remote-upgrader-for-linux-universal-forwarders\/9.0\/about-the-splunk-remote-upgrader-for-linux-universal-forwarders\/about-the-splunk-remote-upgrader-for-linux-universal-forwarders\" target=\"_blank\" rel=\"noreferrer noopener\">Remote Upgrader for Linux Universal Forwarders page<\/a>. Preparing the Agent packages is a manual thing to do, and still needs a way to push the Remote Upgrader addon to the Agent, and requires an access on the Agent to install it.<br><br>The time for the agent to get back to an OK status can be strange at some point. I only tested with one Agent, so upgrading several or a high number of Linux Agent can be different from the experience described in this article. Please test in DEV\/PREPROD environments before going to production (as always).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Introduction Here is a quick review of the Remote Upgrader Feature provided with the brand new Splunk version 10, that was released in July. This new feature allows you to upgrade (or downgrade) your Agents distributed across your Splunk infrastructure managed by your Agent Management, formerly known as Deployment Server. Limitations There are not a [&hellip;]<\/p>\n","protected":false},"author":10,"featured_media":538,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[2],"tags":[35,34],"_links":{"self":[{"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/posts\/520"}],"collection":[{"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/users\/10"}],"replies":[{"embeddable":true,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/comments?post=520"}],"version-history":[{"count":16,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/posts\/520\/revisions"}],"predecessor-version":[{"id":556,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/posts\/520\/revisions\/556"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/media\/538"}],"wp:attachment":[{"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/media?parent=520"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/categories?post=520"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.octamis.com\/octamis-blog\/wp-json\/wp\/v2\/tags?post=520"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}