When we release a new version of Security Onion, we update the Release Notes section and publish a blog post to https://blog.securityonion.net. You’ll want to review these for any relevant information about the individual updates.
Once you’re ready to install an update, use the
Please note that
soup only updates Security Onion components and does NOT update the underlying operating system (OS). There is an option during Configuration to automatically update the OS packages. If you did not enable this option, then you will want to manually update the OS periodically and you will want to ensure that the OS is fully updated before running
soup, you may see errors like:
local: Data failed to compile: ---------- Rendering SLS 'base:common' failed: Jinja variable 'list object' has no attribute 'values'
Status: Downloaded newer image for quay.io/securityonion/so-acng:2.3.30 quay.io/securityonion/so-acng:2.3.30 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 543 100 543 0 0 1412 0 --:--:-- --:--:-- --:--:-- 1414 There is a problem downloading the so-acng:2.3.30 image. Details: gpg: Signature made Thu 18 Feb 2021 02:26:10 PM UTC using RSA key ID FE507013 gpg: BAD signature from "Security Onion Solutions, LLC <email@example.com>"
If you see these errors, it most likely means that a salt highstate process was already running when
soup began. You can wait a few minutes and then try
soup again. Alternatively, you can run
sudo salt-call state.highstate and wait for it to complete before running
soup will check your Elasticsearch
log_size_limit values to see if they are over the recommended values. If so, it will ask you to update the values in
/opt/so/saltstack/local/pillar/minions/. When updating these files, please update any and all instances of
log_size_limit as it may exist as
If you have a distributed deployment with a manager node and separate sensor nodes and/or search nodes, you only need to run
soup on the manager. Once
soup has completed, other nodes should update themselves at the next Salt highstate (typically within 15 minutes).
Just because the update completed on the manager does NOT mean the upgrade is complete on other nodes in the grid. Do not manually restart anything until you know that all the search/heavy nodes in your deployment are updated. This is especially important if you are using true clustering for Elasticsearch.
Each minion is on a random 15 minute check-in period and things like network bandwidth can be a factor in how long the actual upgrade takes. If you have a heavy node on a slow link, it is going to take a while to get the containers to it. Depending on what changes happened between the versions, Elasticsearch might not be able to talk to said heavy node until the update is complete. This will definitely be the case when upgrading to 2.3.40 because of the way the basic license SSL works.
If it looks like you’re missing data after the upgrade, please avoid restarting services and instead make sure at least one search node has completed its upgrade. The best way to do this is to run
sudo salt-call state.highstate from a search node and make sure there are no errors. Typically if it works on one node it will work on the rest. Forward nodes are less complex and will update as they check in so you can monitor those from the Grid section of Security Onion Console (SOC).
When you run
soup on the manager, it does the following:
- Checks to see if it is running on a manager.
- Checks to see if the grid is in Airgap mode. If so, it will then ask for the location of the ISO or mount point.
- Checks to see if we’re running the latest version of
soup. If not, it will put the latest in the correct place and ask you to re-run
- Compares the installed version with what is available on github or the ISO image.
- Checks to see if Salt needs to be updated (more on this later).
- Downloads the new Docker images or, if airgap, copies them from the ISO image.
- Stops the Salt master and minion and restarts it in a restricted mode. This mode only allows the manager to connect to it so that we make sure the manager is done before any of the minions are updated.
- Updates Salt if necessary. This will cause the master and minion services to restart but still in restricted mode.
- Makes any changes to pillars that are needed such as adding new settings or renaming values. This varies from release to release.
- If the grid is in Airgap mode, then it copies the latest ET Open rules and yara rules to the manager.
- The new Salt code is put into place on the manager.
- If Fleet is enabled, then it generates new osquery packages.
- Runs a highstate on the manager which is the actual upgrade where it will use the new Salt code and Docker containers.
- Unlocks the Salt master service and allows minions to connect again.
- Issues a command to all minions to update Salt if necessary. This is important to note as it takes time to to update the Salt minion on all minions. If the minion doesn’t respond for whatever reason, it will not be upgraded at this time. This is not an issue because the first thing that gets checked when a minion talks to the master is if Salt needs to be updated and will apply the update if it does.
- Nodes connect back to the manager and actually perform the upgrade to the new version.