In this tutorial, you will learn how to send alerts when no data is received on an index in ELK/Elastic stack using ElastAlert. According to ElastAlert documentation page, ElastAlert is a simple framework for alerting on anomalies, spikes, or other patterns of interest from data in Elasticsearch. If you have data being written into Elasticsearch in near real time and want to be alerted when that data matches certain patterns, ElastAlert is the tool for you.
ELK Send Alerts when no data is received on an index
Install ELK Stack
In order to be able to send alerts when no data is received on an index in ELK/Elastic stack, ensure that you have a running Elastic stack.
If you are using a Debian system, then the following link may be of help on install Elastic stack.
Install ELK/Elastic Stack on Debian 10
Install ElastAlert
In this setup, we are using Debian 10 and hence, all installation commands are as per the Debian 10 distro.
To install ElastAlert;
Install required packages/tools;
apt update
apt install python3.7 python3.7-dev python3-pip gcc git
Next, install ElastAlert;
pip3 install elastalert
Next, clone the ElastAlert Github repository directory to some directory;
git clone https://github.com/Yelp/elastalert.git /opt/elastalert
Configure ElastAlert to Send Alerts when no data is received on an index
Install ElastAlert Elasticsearch module (for ES version 5 and above).
pip3 install "elasticsearch>=5.0.0"
Configure ElastAlert by renaming the example configuration file, config.yml.example.
cp /opt/elastalert/config.yaml.example /opt/elastalert/config.yaml
Open the configuration file for editing and make the appropriate changes. In our setup, below is how our configuration file, with comment lines removed, looks like.
cat /opt/elastalert/config.yaml
rules_folder: example_rules
run_every:
minutes: 1
buffer_time:
minutes: 15
es_host: localhost
es_port: 9200
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
alert_time_limit:
days: 2Read more about the global config file, config.yaml, on documentation page.
Create ElastAlert Elasticsearch Index
The indices are required to save information and metadata about ElastAlert queries and alerts to Elasticsearch.
The indices can be created by running the command;
elastalert-create-index
Elastic Version: 7.10.2
Reading Elastic 6 index mappings:
Reading index mapping 'es_mappings/6/silence.json'
Reading index mapping 'es_mappings/6/elastalert_status.json'
Reading index mapping 'es_mappings/6/elastalert.json'
Reading index mapping 'es_mappings/6/past_elastalert.json'
Reading index mapping 'es_mappings/6/elastalert_error.json'
New index elastalert_status created
Done!Configure ElastAlert to Send Alerts when no data is received on an index
Alerts are based on conditions defined on the rules. Thus, you need to create a rule to define how you want to be notified when no data is received on an index.
There are different ElastAlert rule types. However, for the purposes of sending alerts when no data is received on an index, we will use Flatline rule.
Flatline rule matches when the total number of events is under a given threshold for a time period.
As defined in our ElastAlert global configuration file, rules file directory is defined as example_rules. So this is the location where we will create our alert rules.
Below is our rule configuration for sending alerts when no data is received on an index via email. The index being checked is called filebeat* in our case.
cat /opt/elastalert/example_rules/no-data.yaml
name: "Nodata flatline"
type: flatline
index: "filebeat*"
realert:
minutes: 10
threshold: 1
timeframe:
minutes: 1
filter:
- query:
match_all: {}
doc_type: _doc
alert:
- "email"
email:
- "[email protected]"
alert_subject: "No Data Flowing to Filebeat Index at {0}"
alert_subject_args:
- "@timestamp"
alert_text: |-
Hi Admin,
Kindly check on the hosts sending data to Filebeat Index. It appears there has not been any new data received.
Regards,
alert_text_type: alert_text_onlyBasically, the query checks if the index has not received more than 1 event data in the last 1 minute. We chose a short time frame and threshold for test purposes only. So, for example, if we chose a threshold like 10, with a time frame of 1 minute, then the rule would check if there less than 10 events received in last one minute and fire an alert.
Configure Mail Relay
In this setup, we use Postfix with gmail relay. You can check other tutorials on how to configure this.
Testing the rule
To test the rule, use the elastalert-test-rule utility as follows;
elastalert-test-rule --config /opt/elastalert/config.yaml /opt/elastalert/example_rules/no-data.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule No Index Data from 2021-05-12 12:51 EDT to 2021-05-12 12:52 EDT: 4 / 4 hits
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_status - {'rule_name': 'No Index Data', 'endtime': datetime.datetime(2021, 5, 12, 16, 52, 15, 989602, tzinfo=tzutc()), 'starttime': datetime.datetime(2021, 5, 12, 16, 51, 15, 389602, tzinfo=tzutc()), 'matches': 0, 'hits': 4, '@timestamp': datetime.datetime(2021, 5, 12, 16, 52, 16, 73516, tzinfo=tzutc()), 'time_taken': 0.016965866088867188}We have 4 hits, hence, if you run the rule, there wont be no alert.
Running ElastAlert Rule
You can run the rule using the elastalert command.
/usr/local/bin/elastalert --verbose --config /opt/elastalert/config.yaml --rule /opt/elastalert/example_rules/no-data.yaml
Example output of running the rule;
1 rules loaded
INFO:elastalert:Starting up
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999922 seconds
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:26 EDT to 2021-05-12 13:27 EDT: 0 / 0 hits
INFO:elastalert:Ran No Index Data from 2021-05-12 13:26 EDT to 2021-05-12 13:27 EDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2021-05-12 13:28 EDT
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-05-12 13:28 EDT
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.999553 seconds
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:27 EDT to 2021-05-12 13:28 EDT: 126 / 126 hits
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:28 EDT to 2021-05-12 13:28 EDT: 0 / 0 hits
INFO:elastalert:Ran No Index Data from 2021-05-12 13:27 EDT to 2021-05-12 13:28 EDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2021-05-12 13:29 EDT
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-05-12 13:29 EDT
...In our global configuration file, the rule is set to run every minute;
run_every:
minutes: 1But the alerts are set to alert after every 10 minutes for every match as per the specific rules file;
realert:
minutes: 10So, no alerts when the rule don’t match.
In order to trigger the rule, I will stop filebeat on the hosts sending data and check if the rule fires;
Once the filebeat is stopped and no data is streaming into the filebeat index, then such an output will show up;
...
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:32 EDT to 2021-05-12 13:33 EDT: 4 / 4 hits
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:33 EDT to 2021-05-12 13:33 EDT: 0 / 0 hits
INFO:elastalert:Ran No Index Data from 2021-05-12 13:32 EDT to 2021-05-12 13:33 EDT: 0 query hits (0 already seen), 0 matches, 0 alerts sent
INFO:elastalert:Background configuration change check run at 2021-05-12 13:34 EDT
INFO:elastalert:Background alerts thread 0 pending alerts sent at 2021-05-12 13:34 EDT
INFO:elastalert:Disabled rules are: []
INFO:elastalert:Sleeping for 59.99959 seconds
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:33 EDT to 2021-05-12 13:34 EDT: 0 / 0 hits
INFO:elastalert:Queried rule No Index Data from 2021-05-12 13:34 EDT to 2021-05-12 13:34 EDT: 0 / 0 hits
INFO:elastalert:Sent email to ['[email protected]']
INFO:elastalert:Ignoring match for silenced rule No Index Data.all
INFO:elastalert:Ran No Index Data from 2021-05-12 13:33 EDT to 2021-05-12 13:34 EDT: 0 query hits (0 already seen), 2 matches, 1 alerts sentcheck the line, INFO:elastalert:Ran No Index Data from 2021-05-12 12:55 EDT to 2021-05-12 13:10 EDT: 60 query hits (60 already seen), 1 matches, 1 alerts sent.
Now, check your mail.
And that is it.
Running ElastAlert Rule as a Service
You can configure ElastAlert to run your rule as a service as shown below;
Below is our sample config file;
cat /etc/systemd/system/elastalert-filebeat.service
[Unit]
Description=ElastAlert No Index data Service
After=elasticsearch.service
[Service]
Type=simple
WorkingDirectory=/opt/elastalert
ExecStart=/usr/local/bin/elastalert --verbose --config /opt/elastalert/config.yaml --rule /opt/elastalert/example_rules/no-data.yaml
[Install]
WantedBy=multi-user.targetEnsure you reload systemd every time you make changes to the service file;
systemctl daemon-reload
Starting the service;
systemctl start elastalert-filebeat
Checking the status;
systemctl status elastalert-filebeat
And there you go. That concludes our guide on how to configure ELK to Send Alerts when no data is received on an index using ElastAlert.
Read more about ElastAlert on ElastAlert Documentation page.
Other tutorials
Install and Setup Wazuh Server with ELK Stack on Ubuntu 20.04
