Managing Alerts¶

Security Onion generates a lot of valuable information for you the second you plug it into a TAP or SPAN port. Between Zeek logs, alert data from Suricata, and full packet capture from Stenographer, you have enough information to begin identifying areas of interest and making positive changes to your security stance.

Note

Network Security Monitoring, as a practice, is not a solution you can plug into your network, make sure you see blinking lights and tell people you are “secure.” It requires active intervention from an analyst to qualify the quantity of information presented. One of those regular interventions is to ensure that you are tuning properly and proactively attempting to reach an acceptable level of signal to noise.

Alerting Engines & Severity¶

There are three alerting engines within Security Onion: Suricata, Wazuh and Playbook (Sigma). Though each engine uses its own severity level system, Security Onion converts that to a standardized alert severity:

event.severity: 4 ==> event.severity_label: critical

event.severity: 3 ==> event.severity_label: high

event.severity: 2 ==> event.severity_label: medium

event.severity: 1 ==> event.severity_label: low

All alerts are viewable in Alerts, Dashboards, Hunt, and Kibana.

Wazuh HIDS Alerts¶

If you want to tune Wazuh HIDS alerts, please see the Wazuh section.

NIDS Testing¶

The easiest way to test that our NIDS is working as expected might be to simply access http://testmynids.org/uid/index.html from a machine that is being monitored by Security Onion. You can do so via the command line using curl:

curl testmynids.org/uid/index.html

Alternatively, you could also test for additional hits with a utility called tmNIDS, running the tool in interactive mode:

curl -sSL https://raw.githubusercontent.com/0xtf/testmynids.org/master/tmNIDS -o /tmp/tmNIDS && chmod +x /tmp/tmNIDS && /tmp/tmNIDS

If everything is working correctly, you should see a corresponding alert (GPL ATTACK_RESPONSE id check returned root) in Alerts, Dashboards, Hunt, or Kibana. If you do not see this alert, try checking to see if the rule is enabled in /opt/so/rules/nids/all.rules:

grep 2100498 /opt/so/rules/nids/all.rules

You can also test using so-test.

Identifying rule categories¶

Rulesets come with a large number of rules enabled (over 20,000 by default). You should only run the rules necessary for your environment, so you may want to disable entire categories of rules that don’t apply to you. Run the following command to get a listing of categories and the number of rules in each:

cut -d\" -f2 /opt/so/rules/nids/all.rules | grep -v "^$" | grep -v "^#" | awk '{print $1, $2}'|sort |uniq -c |sort -nr

So what’s next?¶

In tuning your sensor, you must first understand whether or not taking corrective actions on this signature will lower your overall security stance. For some alerts, your understanding of your own network and the business being transacted across it will be the deciding factor. For example, if you don’t care that users are accessing Facebook, then you can silence the policy-based signatures for Facebook access.

Another consideration is whether or not the traffic is being generated by a misconfigured piece of equipment. If it is, then the most expedient measure may be to resolve the misconfiguration and then reinvestigate tuning.

There are multiple ways to handle overly productive signatures and we’ll try to cover as many as we can without producing a full novel on the subject.

Note

Check out our NIDS tuning video at https://youtu.be/1jEkFIEUCuI!

so-rule¶

so-rule allows you to disable, enable, or modify NIDS rules. Run so-rule without any options to see the help output:

so-rule
usage: so-rule [-h]  ...

optional arguments:
  -h, --help  show this help message and exit

commands:
  disabled            Manage and list disabled rules (add, remove, list)
  enabled             Manage and list enabled rules (add, remove, list)
  modify              Manage and list modified rules (add, remove, list)

Disable the SID¶

We can use so-rule to modify an existing NIDS rule. For example, suppose we want to disable SID 2100498. We can start by listing any currently disabled rules:

sudo so-rule disabled list
No rules disabled.

Next, let’s disable SID 2100498:

sudo so-rule disabled add 2100498
Configuration updated. Would you like to apply your changes now? (y/N) y
Applying idstools state...

Once that completes, we can then verify that 2100498 is now disabled with so-rule disabled list:

sudo so-rule disabled list
Disabled rules:
  - 2100498

Finally, we can check that 2100498 is commented out in /opt/so/rules/nids/all.rules:

grep 2100498 /opt/so/rules/nids/all.rules
# alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)

If you can’t run so-rule, then you can modify configuration manually. Security Onion uses idstools to download new signatures every night and process them against a set list of user generated configurations. To enable or disable SIDs for Suricata, the Salt idstools pillar can be used in the minion pillar file (/opt/so/saltstack/local/pillar/minions/<managername>_<role>.sls). In a distributed Security Onion environment, you only need to change the configuration in the manager pillar and then all other nodes will get the updated rules automatically.

If SID 4321 is noisy, you can disable it as follows:

idstools:
  sids:
    disabled:
      - 4321

From the manager, run the following to update the config:

sudo salt $SENSORNAME_$ROLE state.apply idstools

If you want to disable multiple rules at one time, you can use a regular expression, but make sure you enclose the full entry in single quotes like this:

idstools:
  sids:
    disabled:
      - 're:heartbleed'

Modify the SID¶

We can use so-rule to modify an existing NIDS rule. For example, suppose that we want to modify SID 2100498 and replace any instances of “returned root” with “returned root test”. We can start by listing any rules that are currently modified:

sudo so-rule modify list
No rules currently modified.

Let’s first check the syntax for the add option:

sudo so-rule modify add -h
usage: so-rule modify add [-h] [--apply] SID|REGEX SEARCH_TERM REPLACE_TERM

positional arguments:
  SID|REGEX     A valid SID (ex: "4321") or regular expression pattern (ex:
                "re:heartbleed|spectre")
  SEARCH_TERM   A quoted regex search term (ex: "\\\$EXTERNAL_NET")
  REPLACE_TERM  The text to replace the search term with

optional arguments:
  -h, --help    show this help message and exit
  --apply       After updating rule configuration, apply the idstools state.

Now that we understand the syntax, let’s add our modification:

sudo so-rule modify add 2100498 "returned root" "returned root test"
Configuration updated. Would you like to apply your changes now? (y/N) y
Applying idstools state...

Once the command completes, we can verify that our modification has been added:

sudo so-rule modify list
Modified rules + modifications:
  - 2100498 "returned root" "returned root test"

Finally, we can check the modified rule in /opt/so/rules/nids/all.rules:

grep 2100498 /opt/so/rules/nids/all.rules
alert ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root test"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2010_09_23;)

To include an escaped $ character in the regex pattern you’ll need to make sure it’s properly escaped. For example, if you want to modify SID 2009582 and change $EXTERNAL_NET to $HOME_NET:

sudo so-rule modify add 2009582 "\\\$EXTERNAL_NET" "\$HOME_NET"

The first string is a regex pattern, while the second is just a raw value. You’ll need to ensure the first of the two properly escapes any characters that would be interpreted by regex. The second only needs the $ character escaped to prevent bash from treating that as a variable.

If you can’t run so-rule, you can modify the configuration manually in the manager pillar file at /opt/so/saltstack/local/pillar/minions/<managername>_<role>.sls (where <role> is manager, managersearch, standalone, or eval depending on the manager type that was chosen during install). In this file, the idstools section has a modify sub-section where you can add your modifications. For example:

idstools:
  sids:
    modify:
      - '2019401 "seconds \d+" "seconds 3600"'

From the manager, run the following to update the config:

sudo salt $SENSORNAME_$ROLE state.apply idstools

If you need to modify a part of a rule that contains a special character, such as a $ in variable names, the special character needs to be escaped in the search part of the modify string. For example:

idstools:
  sids:
    modify:
      - '2826931 "\$EXTERNAL_NET" "!$HOME_NET"'

Rewrite the signature¶

In some cases, you may not want to use the modify option above, but instead create a local copy of the rule as shown in the Adding Local Rules section and then disable the original rule.

Edit the /opt/so/saltstack/local/salt/idstools/local.rules file using vi or your favorite text editor:
sudo vi /opt/so/saltstack/local/salt/idstools/local.rules
Paste the rule. You may want to bump the SID into the 90,000,000 range and set the revision to 1.
Now that we have a signature that will generate alerts a little more selectively, we need to disable the original signature. As shown above, we edit the minion pillar and add the SID to the idstools - sids - disabled section.
Finally, from the manager, update the config on the remote node:
```
salt $SENSORNAME_$ROLE state.highstate
```

Threshold¶

You can manage threshold entries for Suricata using Salt pillars. The format of the pillar file can be seen below, as well as in /opt/so/saltstack/default/pillar/thresholding/pillar.usage and /opt/so/saltstack/default/pillar/thresholding/pillar.example

Note

The signature id (SID) must be unique. If you have multiple entries for the same SID, it will cause an error in salt resulting in all of the nodes in your grid to error out when checking in.

Usage:

thresholding:
  sids:
    <signature id>:
      - threshold:
          gen_id: <generator id>
          type: <threshold | limit | both>
          track: <by_src | by_dst>
          count: <count>
          seconds: <seconds>
      - rate_filter:
          gen_id: <generator id>
          track: <by_src | by_dst | by_rule | by_both>
          count: <count>
          seconds: <seconds>
          new_action: <alert | pass>
          timeout: <seconds>
      - suppress:
          gen_id: <generator id>
          track: <by_src | by_dst | by_either>
          ip: <ip | subnet>

Example:

thresholding:
  sids:
    8675309:
      - threshold:
          gen_id: 1
          type: threshold
          track: by_src
          count: 10
          seconds: 10
      - threshold:
          gen_id: 1
          type: limit
          track: by_dst
          count: 100
          seconds: 30
      - rate_filter:
          gen_id: 1
          track: by_rule
          count: 50
          seconds: 30
          new_action: alert
          timeout: 30
      - suppress:
          gen_id: 1
          track: by_either
          ip: 10.10.3.7
    11223344:
      - threshold:
          gen_id: 1
          type: limit
          track: by_dst
          count: 10
          seconds: 10
      - rate_filter:
          gen_id: 1
          track: by_src
          count: 50
          seconds: 20
          new_action: pass
          timeout: 60
      - suppress:
          gen_id: 1
          track: by_src
          ip: 10.10.3.0/24

In order to apply the threshold to all nodes, place the pillar in /opt/so/saltstack/local/pillar/global.sls. If you want to apply the threshold to a single node, place the pillar in /opt/so/saltstack/local/pillar/minions/<MINION_ID>.sls

Warning

Salt sls files are in YAML format. When editing these files, please be very careful to respect YAML syntax, especially whitespace. For more information, please see:

https://docs.saltproject.io/en/latest/topics/troubleshooting/yaml_idiosyncrasies.html

Please note that Suricata 6 has a 64-character limitation on the IP field in a threshold. You can read more about this at https://redmine.openinfosecfoundation.org/issues/4377.

For example, the following threshold IP exceeds the 64-character limit:

thresholding:
  sids:
    2012454:
      - suppress:
          gen_id: 1
          track: by_dst
          ip: 1.1.1.1,2.2.2.2,3.3.3.3,4.4.4.4,5.5.5.5,6.6.6.6,7.7.7.7,8.8.8.8,9.9.9.9,10.10.10.10,11.11.11.11

This results in the following error in the Suricata log:

<Error> - [ERRCODE: SC_ERR_PCRE_COPY_SUBSTRING(325)] - pcre_copy_substring failed

The solution is to break the ip field into multiple entries like this:

thresholding:
  sids:
    2012454:
      - suppress:
          gen_id: 1
          track: by_dst
          ip: 1.1.1.1,2.2.2.2,3.3.3.3,4.4.4.4,5.5.5.5,6.6.6.6,7.7.7.7,8.8.8.8
      - suppress:
          gen_id: 1
          track: by_dst
          ip: 9.9.9.9,10.10.10.10,11.11.11.11

Suppressions¶

A suppression rule allows you to make some finer grained decisions about certain rules without the onus of rewriting them. With this functionality we can suppress rules based on their signature, the source or destination address and even the IP or full CIDR network block. This way, you still have the basic ruleset, but the situations in which they fire are altered. It’s important to note that with this functionality, care should be given to the suppressions being written to make sure they do not suppress legitimate alerts. See above for suppress examples.

Flowbits¶

idstools may seem like it is ignoring your disabled rules request if you try to disable a rule that has flowbits set.

Note

For a quick primer on flowbits, see https://blog.snort.org/2011/05/resolving-flowbit-dependancies.html.

For example, consider the following rules that reference the ET.MSSQL flowbit.

First rule:

alert tcp $HOME_NET any -> $EXTERNAL_NET !1433 (msg:"ET POLICY Outbound MSSQL Connection to Non-Standard Port - Likely Malware"; flow:to_server,established; content:"|12 01 00|"; depth:3; content:"|00 00 00 00 00 00 15 00 06 01 00 1b 00 01 02 00 1c 00|"; distance:1; within:18; content:"|03 00|"; distance:1; within:2; content:"|00 04 ff 08 00 01 55 00 00 00|"; distance:1; within:10; flowbits:set,ET.MSSQL; classtype:bad-unknown; sid:2013409; rev:3;)

Second rule:

alert tcp $HOME_NET any -> $EXTERNAL_NET 1433 (msg:"ET POLICY Outbound MSSQL Connection to Standard port (1433)"; flow:to_server,established; content:"|12 01 00|"; depth:3; content:"|00 00 00 00 00 00 15 00 06 01 00 1b 00 01 02 00 1c 00|"; distance:1; within:18; content:"|03 00|"; distance:1; within:2; content:"|00 04 ff 08 00 01 55 00 00 00|"; distance:1; within:10; flowbits:set,ET.MSSQL; classtype:bad-unknown; sid:2013410; rev:4;)

Third rule:

alert tcp $HOME_NET any -> $EXTERNAL_NET !1433 (msg:"ET TROJAN Bancos.DV MSSQL CnC Connection Outbound"; flow:to_server,established; flowbits:isset,ET.MSSQL; content:"|49 00 B4 00 4D 00 20 00 54 00 48 00 45 00 20 00 4D 00 41 00 53 00 54 00 45 00 52 00|"; classtype:trojan-activity; sid:2013411; rev:1;)

If you try to disable the first two rules without disabling the third rule (which has flowbits:isset,ET.MSSQL) the third rule could never fire due to one of the first two rules needing to fire first. idstools helpfully resolves all of your flowbit dependencies, and in this case, is “re-enabling” that rule for you on the fly. Disabling all three of those rules by adding the following to disablesid.conf has the obvious negative effect of disabling all three of the rules:

2013409
2013410
2013411

When you run sudo so-rule-update, watch the “Setting Flowbit State…” section and you can see that if you disable all three (or however many rules share that flowbit) that the “Enabled XX flowbits” line is decremented and all three rules should then be disabled in your all.rules.