Welcome to our tutorial on how to install ELK/Elastic Stack on Debian 10. ELK is the acronym for three open source projects; Elasticsearch, Logstash, and Kibana.
Elasticsearchis a search and analytics engine.Logstashis a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a “stash” like Elasticsearch.Kibanalets users visualize data with charts and graphs in Elasticsearch.- And then the
Beatsgot dropped onto the stack to help in shipping and collection of logs from different endpoints and sends them to either Logstash or directly to Elasticsearch for further processing or indexing respectively.
Install ELK/Elastic Stack on Debian 10
So, let us see how to install ELK/Elastic stack on Debian 10.
- ELK stack components versions: While setting up ELK/Elastic Stack, you need to ensure that all the components of the stack are of the same version.
- ELK stack components installation order: Also, ensure that the installation of the ELK/Elastic stack components follow the following order:
- ELK/Elastic stack components installation method: There are different methods in which you can install ELK/Elastic stack on Debian 10;
- From the Elastic stack APT repository
- Manually using the stack components DEB binary packages
- Using docker containers
- Installing from ELK stack archives
However, in this method, we will go the use of the Elastic stack APT repository. This ensures a seamless upgrade in case you need to move to a later version.
- Installation Architecture: There are different deployment architectures for ELK stack. However, in this simple guide, we will be running a single node ELK cluster.
Install ELK/Elastic Stack APT Repository on Debian 10
None of the ELK stack components is provided by the default Debian 10 repositories. As such, you need to install the ELK/Elastic repository to enable you install the stack components.
Import the ELK stack repository PGP signing Key;
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
Install ELK APT repository on Debian 10.
sudo apt install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-7.x.list
Run system update to update the added repository;
apt update
Install and Configure Elasticsearch on Debian 10
Installing Elasticsearch
Once the repository is in place, you can now install Elasticsearch on Debian 10.
Note, since the installation is done from the ELK APT repositories, you will get the current latest and stable versions installed.
apt install elasticsearch
Configuring Elasticsearch
Once the installation is done, proceed to configure Elasticsearch. The default configuration file for Elasticsearch is /etc/elasticsearch/elasticsearch.yml.
Open the configuration file for editing using your preferred text editor;
vim /etc/elasticsearch/elasticsearch.yml
You can optionally set the name of the cluster or go with the default;
# ---------------------------------- Cluster -----------------------------------
...
cluster.name: debian-10Set the interface IP and port on which Elasticsearch can listen on. By default, it uses the loopback interface and port 9200/tcp. Setting the network host to an interface IP rather than the loopback address allows you to externally access Elasticsearch.
# ---------------------------------- Network -----------------------------------
...
network.host: 192.168.58.10
...
http.port: 9200By default, Elasticsearch tries to discovers other nodes to form a cluster when started. Since we are running a single node cluster, you need to specify the same by inserting the line, discovery.type: single-node, under the Discovery section.
# --------------------------------- Discovery ----------------------------------
...
discovery.type: single-nodeSave and exit the file.
Configure the JVM heap size and set it to about half the memory available on the system.
vim /etc/elasticsearch/jvm.options
...
# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space
-Xms512m
-Xmx512mSave and exit the file.
There are other important Elasticsearch configuration settings you should consider, especially if you are taking it to production. Check the Import Elasticsearch Configuration page for more tips.
Running Elasticsearch
Once you are done with Elasticsearch configurations, start and enable Elasticsearch to run on system boot.
systemctl enable --now elasticsearch
Checking the status;
systemctl status elasticsearch
systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-12-18 00:13:25 EST; 3s ago
Docs: https://www.elastic.co
Main PID: 1163 (java)
Tasks: 45 (limit: 2359)
Memory: 738.9M
CGroup: /system.slice/elasticsearch.service
├─1163 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -
└─1341 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
Dec 18 00:13:12 elk.itnixpro.com systemd[1]: Starting Elasticsearch...
Dec 18 00:13:25 elk.itnixpro.com systemd[1]: Started Elasticsearch.You can also use curl to check the same;
curl http://192.168.58.10:9200
{
"name" : "elk.itnixpro.com",
"cluster_name" : "debian-10",
"cluster_uuid" : "2QCjK3l0QdyOX0ts1Q136w",
"version" : {
"number" : "7.10.1",
"build_flavor" : "default",
"build_type" : "deb",
"build_hash" : "1c34507e66d7db1211f66f3513706fdf548736aa",
"build_date" : "2020-12-05T01:00:33.671820Z",
"build_snapshot" : false,
"lucene_version" : "8.7.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}Similarly, verify the port;
ss -altnp | grep :9200
LISTEN 0 128 [::ffff:192.168.58.10]:9200 *:* users:(("java",pid=1163,fd=255))Install and Configure Kibana on Debian 10
The next step in installing ELK stack on Debian is to install Kibana. Since you already have ELK stack APT repository installed, simply execute the command below to install it.
apt install kibana
Once the installation is done, you can configure Kibana. The default configuration file for Kibana is /etc/kibana/kibana.yml.
Open the file editing.
vim /etc/kibana/kibana.yml
The default host and port settings configure Kibana to run on localhost:5601. We need to change, the host especially, to enable us to reach Kibana externally.
# Kibana is served by a back end server. This setting specifies the port to use.
# server.port: 5601
server.port: 5601
...
# To allow connections from remote users, set this parameter to a non-loopback address.
#server.host: "localhost"
server.host: "192.168.58.10"Next, you need to configure how Kibana will connect to Elasticsearch;
# The URLs of the Elasticsearch instances to use for all your queries.
#elasticsearch.hosts: ["http://localhost:9200"]
elasticsearch.hosts: ["http://192.168.58.10:9200"]Replace the IP addresses appropriately.
In it basic setup, that is just enough. Save and exit the file.
Running Kibana
Now start and enable Kibana to run on system boot;
systemctl enable --now kibana
Check the status;
systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-12-18 02:21:27 EST; 26s ago
Main PID: 1964 (node)
Tasks: 11 (limit: 2359)
Memory: 335.3M
CGroup: /system.slice/kibana.service
└─1964 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist
Dec 18 02:21:40 elk.itnixpro.com kibana[1964]: {"type":"log","@timestamp":"2020-12-18T07:21:40Z","tags":["info","savedobjects-service"]
...Accessing Kibana Interface
You can now access Kibana interface using the URL, http://elk-server-IP-or-hostname:5601.
Note that, if there any firewall running, be sure to open Kibana port 5601.
Upon accessing Kibana for the first time, you are welcomed by such an interface.
For our case, we do not have any data yet, hence can just click Explore on my own to proceed to Kibana Home dashboard.
To visualize and explore data in Kibana, you must create an index pattern to retrieve data from Elasticsearch. We will look into this in our next guide.
Install and Configure Logstash on Debian 10
Follow the link below to learn how to install and configure Logstash on Debian 10.
install and configure Logstash on Debian 10
Just to note, Logstash is optional unless you need to process your logs further to extract fields of your interset, you can install beats on the end point systems and push the data directly to Elasticsearch.
Install and Configure Filebeat on Debian 10
Install and Configure Filebeat on Debian 10
Stay tuned for our next guide on installing Beats on Debian 10.
