Redis

From https://redis.io/:

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker. It supports data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs and geospatial indexes with radius queries.

During setup, you can choose to extend your manager node storage using separate search nodes. When you choose this option, Logstash on the manager node outputs to Redis. Search nodes then consume from Redis.

Queue

To see how many logs are in the Redis queue:

sudo so-redis-count

If the queue is backed up and doesn’t seem to be draining, try stopping Logstash on the manager node:

sudo so-logstash-stop

Then monitor the queue to see if it drains:

watch 'sudo so-redis-count'

If the Redis queue looks okay, but you are still having issues with logs getting indexed into Elasticsearch, you will want to check the Logstash statistics on the search node(s).

Tuning

We configure Redis to use 812MB of your total system memory. If you have sufficient RAM available, you may want to increase the redis_maxmemory setting in /opt/so/saltstack/local/pillar/global.sls. This value is in Megabytes so to set it to use 8 gigs of ram you would set the value to 8192.

Logstash on the manager node is configured to send to Redis. For best performance, you may want to ensure that batch is set to true and then tune the batch_events variable to find the sweet spot for your deployment.

See also

For more information about logstash’s output plugin for Redis, please see https://www.elastic.co/guide/en/logstash/current/plugins-outputs-redis.html.

Logstash on search nodes pulls from Redis. For best performance, you may want to tune batch_count and threads to find the sweet spot for your deployment.

See also

For more information about logstash’s input plugin for Redis, please see https://www.elastic.co/guide/en/logstash/current/plugins-inputs-redis.html.

Diagnostic Logging

Redis logs can be found at /opt/so/log/redis/.

More Information

See also

For more information about Redis, please see https://redis.io/.