Managing Alerts¶
Security Onion generates a lot of valuable information for you the second you plug it into a TAP or SPAN port. Between Zeek logs, alert data from Suricata, and full packet capture from Stenographer, you have enough information to begin identifying areas of interest and making positive changes to your security stance.
Note
Network Security Monitoring, as a practice, is not a solution you can plug into your network, make sure you see blinking lights and tell people you are “secure.” It requires active intervention from an analyst to qualify the quantity of information presented. One of those regular interventions is to ensure that you are tuning properly and proactively attempting to reach an acceptable level of signal to noise.
Alerting Engines & Severity¶
There are three alerting engines within Security Onion: Suricata, Wazuh and Playbook (Sigma). Though each engine uses its own severity level system, Security Onion converts that to a standardized alert severity:
event.severity
: 4
==> event.severity_label
: critical
event.severity
: 3
==> event.severity_label
: high
event.severity
: 2
==> event.severity_label
: medium
event.severity
: 1
==> event.severity_label
: low
All alerts are viewable in Alerts, Dashboards, Hunt, and Kibana.
NIDS Testing¶
The easiest way to test that our NIDS is working as expected might be to simply access http://testmynids.org/uid/index.html from a machine that is being monitored by Security Onion. You can do so via the command line using curl
:
curl testmynids.org/uid/index.html
Alternatively, you could also test for additional hits with a utility called tmNIDS
, running the tool in interactive mode:
curl -sSL https://raw.githubusercontent.com/0xtf/testmynids.org/master/tmNIDS -o /tmp/tmNIDS && chmod +x /tmp/tmNIDS && /tmp/tmNIDS
If everything is working correctly, you should see a corresponding alert (GPL ATTACK_RESPONSE id check returned root
) in Alerts, Dashboards, Hunt, or Kibana. If you do not see this alert, try checking to see if the rule is enabled in /opt/so/rules/nids/all.rules
:
grep 2100498 /opt/so/rules/nids/all.rules
You can also test using so-test.
Identifying rule categories¶
Rulesets come with a large number of rules enabled (over 20,000 by default). You should only run the rules necessary for your environment, so you may want to disable entire categories of rules that don’t apply to you. Run the following command to get a listing of categories and the number of rules in each:
cut -d\" -f2 /opt/so/rules/nids/all.rules | grep -v "^$" | grep -v "^#" | awk '{print $1, $2}'|sort |uniq -c |sort -nr
So what’s next?¶
In tuning your sensor, you must first understand whether or not taking corrective actions on this signature will lower your overall security stance. For some alerts, your understanding of your own network and the business being transacted across it will be the deciding factor. For example, if you don’t care that users are accessing Facebook, then you can silence the policy-based signatures for Facebook access.
Another consideration is whether or not the traffic is being generated by a misconfigured piece of equipment. If it is, then the most expedient measure may be to resolve the misconfiguration and then reinvestigate tuning.
There are multiple ways to handle overly productive signatures and we’ll try to cover as many as we can without producing a full novel on the subject.
Note
Check out our NIDS tuning video at https://youtu.be/1jEkFIEUCuI!
so-rule¶
so-rule
allows you to disable, enable, or modify NIDS rules. Run so-rule
without any options to see the help output:
so-rule
usage: so-rule [-h] ...
optional arguments:
-h, --help show this help message and exit
commands:
disabled Manage and list disabled rules (add, remove, list)
enabled Manage and list enabled rules (add, remove, list)
modify Manage and list modified rules (add, remove, list)
Disable the SID¶
We can use so-rule
to modify an existing NIDS rule. For example, suppose we want to disable SID 2100498. We can start by listing any currently disabled rules:
sudo so-rule disabled list
No rules disabled.
Next, let’s disable SID 2100498:
sudo so-rule disabled add 2100498
Configuration updated. Would you like to apply your changes now? (y/N) y
Applying idstools state...
Once that completes, we can then verify that 2100498 is now disabled with so-rule disabled list
:
sudo so-rule disabled list
Disabled rules:
- 2100498
Finally, we can check that 2100498 is commented out in /opt/so/rules/nids/all.rules
:
grep 2100498 /opt/so/rules/nids/all.rules
# alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)
If you can’t run so-rule
, then you can modify configuration manually. Security Onion uses idstools
to download new signatures every night and process them against a set list of user generated configurations. To enable or disable SIDs for Suricata, the Salt idstools
pillar can be used in the minion pillar file (/opt/so/saltstack/local/pillar/minions/<managername>_<role>.sls
). In a distributed Security Onion environment, you only need to change the configuration in the manager pillar and then all other nodes will get the updated rules automatically.
If SID 4321 is noisy, you can disable it as follows:
idstools:
sids:
disabled:
- 4321
From the manager, run the following to update the config:
sudo salt $SENSORNAME_$ROLE state.apply idstools
If you want to disable multiple rules at one time, you can use a regular expression, but make sure you enclose the full entry in single quotes like this:
idstools:
sids:
disabled:
- 're:heartbleed'
Modify the SID¶
We can use so-rule
to modify an existing NIDS rule. For example, suppose that we want to modify SID 2100498 and replace any instances of “returned root” with “returned root test”. We can start by listing any rules that are currently modified:
sudo so-rule modify list
No rules currently modified.
Let’s first check the syntax for the add
option:
sudo so-rule modify add -h
usage: so-rule modify add [-h] [--apply] SID|REGEX SEARCH_TERM REPLACE_TERM
positional arguments:
SID|REGEX A valid SID (ex: "4321") or regular expression pattern (ex:
"re:heartbleed|spectre")
SEARCH_TERM A quoted regex search term (ex: "\\\$EXTERNAL_NET")
REPLACE_TERM The text to replace the search term with
optional arguments:
-h, --help show this help message and exit
--apply After updating rule configuration, apply the idstools state.
Now that we understand the syntax, let’s add our modification:
sudo so-rule modify add 2100498 "returned root" "returned root test"
Configuration updated. Would you like to apply your changes now? (y/N) y
Applying idstools state...
Once the command completes, we can verify that our modification has been added:
sudo so-rule modify list
Modified rules + modifications:
- 2100498 "returned root" "returned root test"
Finally, we can check the modified rule in /opt/so/rules/nids/all.rules
:
grep 2100498 /opt/so/rules/nids/all.rules
alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root test"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)
To include an escaped $
character in the regex pattern you’ll need to make sure it’s properly escaped. For example, if you want to modify SID 2009582 and change $EXTERNAL_NET
to $HOME_NET
:
sudo so-rule modify add 2009582 "\\\$EXTERNAL_NET" "\$HOME_NET"
The first string is a regex pattern, while the second is just a raw value. You’ll need to ensure the first of the two properly escapes any characters that would be interpreted by regex. The second only needs the $
character escaped to prevent bash from treating that as a variable.
If you can’t run so-rule
, you can modify the configuration manually in the manager pillar file at /opt/so/saltstack/local/pillar/minions/<managername>_<role>.sls
(where <role>
is manager
, managersearch
, standalone
, or eval
depending on the manager type that was chosen during install). In this file, the idstools
section has a modify
sub-section where you can add your modifications. For example:
idstools:
sids:
modify:
- '2019401 "seconds \d+" "seconds 3600"'
From the manager, run the following to update the config:
sudo salt $SENSORNAME_$ROLE state.apply idstools
If you need to modify a part of a rule that contains a special character, such as a $
in variable names, the special character needs to be escaped in the search
part of the modify string. For example:
idstools:
sids:
modify:
- '2826931 "\$EXTERNAL_NET" "!$HOME_NET"'
Rewrite the signature¶
In some cases, you may not want to use the modify option above, but instead create a local copy of the rule as shown in the Adding Local Rules section and then disable the original rule.
Edit the
/opt/so/saltstack/local/salt/idstools/local.rules
file usingvi
or your favorite text editor:sudo vi /opt/so/saltstack/local/salt/idstools/local.rules
Paste the rule. You may want to bump the SID into the 90,000,000 range and set the revision to 1.
Now that we have a signature that will generate alerts a little more selectively, we need to disable the original signature. As shown above, we edit the minion pillar and add the SID to the
idstools - sids - disabled
section.Finally, from the manager, update the config on the remote node:
salt $SENSORNAME_$ROLE state.highstate
Threshold¶
You can manage threshold entries for Suricata using Salt pillars. The format of the pillar file can be seen below, as well as in /opt/so/saltstack/default/pillar/thresholding/pillar.usage
and /opt/so/saltstack/default/pillar/thresholding/pillar.example
Note
The signature id (SID) must be unique. If you have multiple entries for the same SID, it will cause an error in salt resulting in all of the nodes in your grid to error out when checking in.
Usage:
thresholding:
sids:
<signature id>:
- threshold:
gen_id: <generator id>
type: <threshold | limit | both>
track: <by_src | by_dst>
count: <count>
seconds: <seconds>
- rate_filter:
gen_id: <generator id>
track: <by_src | by_dst | by_rule | by_both>
count: <count>
seconds: <seconds>
new_action: <alert | pass>
timeout: <seconds>
- suppress:
gen_id: <generator id>
track: <by_src | by_dst | by_either>
ip: <ip | subnet>
Example:
thresholding:
sids:
8675309:
- threshold:
gen_id: 1
type: threshold
track: by_src
count: 10
seconds: 10
- threshold:
gen_id: 1
type: limit
track: by_dst
count: 100
seconds: 30
- rate_filter:
gen_id: 1
track: by_rule
count: 50
seconds: 30
new_action: alert
timeout: 30
- suppress:
gen_id: 1
track: by_either
ip: 10.10.3.7
11223344:
- threshold:
gen_id: 1
type: limit
track: by_dst
count: 10
seconds: 10
- rate_filter:
gen_id: 1
track: by_src
count: 50
seconds: 20
new_action: pass
timeout: 60
- suppress:
gen_id: 1
track: by_src
ip: 10.10.3.0/24
In order to apply the threshold to all nodes, place the pillar in /opt/so/saltstack/local/pillar/global.sls
. If you want to apply the threshold to a single node, place the pillar in /opt/so/saltstack/local/pillar/minions/<MINION_ID>.sls
Warning
Please note that Suricata 6 has a 64-character limitation on the IP field in a threshold. You can read more about this at https://redmine.openinfosecfoundation.org/issues/4377.
For example, the following threshold IP exceeds the 64-character limit:
thresholding:
sids:
2012454:
- suppress:
gen_id: 1
track: by_dst
ip: 1.1.1.1,2.2.2.2,3.3.3.3,4.4.4.4,5.5.5.5,6.6.6.6,7.7.7.7,8.8.8.8,9.9.9.9,10.10.10.10,11.11.11.11
This results in the following error in the Suricata log:
<Error> - [ERRCODE: SC_ERR_PCRE_COPY_SUBSTRING(325)] - pcre_copy_substring failed
The solution is to break the ip
field into multiple entries like this:
thresholding:
sids:
2012454:
- suppress:
gen_id: 1
track: by_dst
ip: 1.1.1.1,2.2.2.2,3.3.3.3,4.4.4.4,5.5.5.5,6.6.6.6,7.7.7.7,8.8.8.8
- suppress:
gen_id: 1
track: by_dst
ip: 9.9.9.9,10.10.10.10,11.11.11.11
Suppressions¶
A suppression rule allows you to make some finer grained decisions about certain rules without the onus of rewriting them. With this functionality we can suppress rules based on their signature, the source or destination address and even the IP or full CIDR network block. This way, you still have the basic ruleset, but the situations in which they fire are altered. It’s important to note that with this functionality, care should be given to the suppressions being written to make sure they do not suppress legitimate alerts. See above for suppress
examples.
Flowbits¶
idstools
may seem like it is ignoring your disabled rules request if you try to disable a rule that has flowbits set.
Note
For a quick primer on flowbits, see https://blog.snort.org/2011/05/resolving-flowbit-dependancies.html.
For example, consider the following rules that reference the ET.MSSQL
flowbit.
First rule:
alert tcp $HOME_NET any -> $EXTERNAL_NET !1433 (msg:"ET POLICY Outbound MSSQL Connection to Non-Standard Port - Likely Malware"; flow:to_server,established; content:"|12 01 00|"; depth:3; content:"|00 00 00 00 00 00 15 00 06 01 00 1b 00 01 02 00 1c 00|"; distance:1; within:18; content:"|03 00|"; distance:1; within:2; content:"|00 04 ff 08 00 01 55 00 00 00|"; distance:1; within:10; flowbits:set,ET.MSSQL; classtype:bad-unknown; sid:2013409; rev:3;)
Second rule:
alert tcp $HOME_NET any -> $EXTERNAL_NET 1433 (msg:"ET POLICY Outbound MSSQL Connection to Standard port (1433)"; flow:to_server,established; content:"|12 01 00|"; depth:3; content:"|00 00 00 00 00 00 15 00 06 01 00 1b 00 01 02 00 1c 00|"; distance:1; within:18; content:"|03 00|"; distance:1; within:2; content:"|00 04 ff 08 00 01 55 00 00 00|"; distance:1; within:10; flowbits:set,ET.MSSQL; classtype:bad-unknown; sid:2013410; rev:4;)
Third rule:
alert tcp $HOME_NET any -> $EXTERNAL_NET !1433 (msg:"ET TROJAN Bancos.DV MSSQL CnC Connection Outbound"; flow:to_server,established; flowbits:isset,ET.MSSQL; content:"|49 00 B4 00 4D 00 20 00 54 00 48 00 45 00 20 00 4D 00 41 00 53 00 54 00 45 00 52 00|"; classtype:trojan-activity; sid:2013411; rev:1;)
If you try to disable the first two rules without disabling the third rule (which has flowbits:isset,ET.MSSQL
) the third rule could never fire due to one of the first two rules needing to fire first. idstools helpfully resolves all of your flowbit dependencies, and in this case, is “re-enabling” that rule for you on the fly. Disabling all three of those rules by adding the following to disablesid.conf
has the obvious negative effect of disabling all three of the rules:
1:2013409
1:2013410
1:2013411
When you run sudo so-rule-update
, watch the “Setting Flowbit State…” section and you can see that if you disable all three (or however many rules share that flowbit) that the “Enabled XX flowbits” line is decremented and all three rules should then be disabled in your all.rules
.