Setting Up ELK Stack (Elasticsearch, Logstash, Kibana) for Centralized Log Management

If your applications and servers are scattered across multiple machines, checking logs one server at a time doesn't scale. The ELK Stack solves this by pulling every log — from your web servers, applications, and network devices — into one searchable place. Instead of SSHing into ten different servers to find one error message, you search it in seconds from a single dashboard.

Quick answer: ELK Stack is three open-source tools working together: Elasticsearch (stores and searches your log data), Logstash (collects and cleans up the logs before storing them), and Kibana (the dashboard you actually look at). This guide walks through installing all three on a Linux server, connecting them, and getting your first logs searchable in Kibana.

What Is the ELK Stack, Exactly?

Each letter in "ELK" is a separate tool, and it helps to understand what each one actually does before installing anything:

• Elasticsearch — a search and storage engine. Think of it as a database built specifically to search through huge volumes of text quickly. This is where your log data actually lives.

• Logstash — a data pipeline. It reads raw logs (which are often messy, inconsistent text), reshapes them into a clean, structured format, and sends them into Elasticsearch.

• Kibana — the visual dashboard. It connects to Elasticsearch and lets you search logs, build charts, and set up dashboards, all through a web browser.

A fourth tool, Filebeat, is usually added alongside these three. It's a lightweight agent installed on each server you want to monitor — its only job is to read log files and forward them to Logstash or Elasticsearch. Filebeat is why people sometimes call this the "Elastic Stack" rather than just "ELK."

How This Is Different from Prometheus/Grafana or Wazuh

If you've already set up monitoring tools, it's worth being clear on what problem ELK actually solves, since these tools often get confused:

• Prometheus and Grafana are built for metrics — numbers over time, like CPU usage, memory, or request counts. They answer "how much" and "how fast."

• Wazuh is built for security monitoring — detecting intrusions, file changes, and suspicious behaviour across your systems. It answers "is something wrong or malicious happening."

• ELK Stack is built for log analytics — full, searchable text logs. It answers "what exactly happened, and when, and show me the full message." If an application throws an error and you need to read the actual error text, stack trace, or request details, that's a log analytics job, not a metrics or security job.

Many production setups actually run all three side by side: Prometheus/Grafana for performance metrics, Wazuh for security alerts, and ELK for digging into the detailed logs once something's flagged by the other two.

Prerequisites and Hardware Requirements

ELK is genuinely resource-hungry, especially Elasticsearch, which keeps a lot of data in memory to search it quickly. Undersizing this is the most common reason people get frustrated with ELK early on.

Storage speed matters more than most people expect here. Elasticsearch is constantly writing new log data to disk and reading it back for searches, so slow storage becomes the bottleneck long before CPU or RAM does. This is exactly the kind of workload NVMe storage (a much faster type of SSD) is built for — if you're setting this up for a real production environment rather than a quick test, a bare-metal server with NVMe storage will feel noticeably more responsive than budget SATA SSD storage once your log volume grows. For teams expecting to retain large volumes of historical logs, it's also worth looking at options built specifically for high-capacity storage.

For this tutorial, we'll set up a single-server ELK install — enough to get real log data flowing and searchable. Larger deployments typically split Elasticsearch across multiple nodes (a "cluster"), but that's a scaling step you can take later once this base setup is working.

Step 1: Update Your Server and Install Prerequisites

Connect via SSH and make sure your package list is current.

sudo apt update && sudo apt upgrade -y
sudo apt install -y curl gnupg apt-transport-https
                            

You don't need to install Java separately — both Elasticsearch and Logstash come bundled with a compatible Java runtime, so this step is handled automatically during their installation.

Step 2: Add the Elastic Package Repository

All three tools (Elasticsearch, Logstash, Kibana) come from the same vendor and share one repository, so you only need to set this up once.

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elastic.gpg
echo "deb [signed-by=/usr/share/keyrings/elastic.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update
                            

Step 3: Install and Configure Elasticsearch

sudo apt install -y elasticsearch
                            

Important: as of Elasticsearch 8.x, security is turned on automatically during installation — you don't need to enable it manually like in older versions. When the install finishes, it will print out:

• A password for the built-in elastic superuser account

• A TLS certificate fingerprint (used to verify secure connections)

• An enrollment token (used later to connect Kibana securely)

Copy all of this down before you close the terminal — you'll need it in the next steps, and regenerating it later takes extra commands.

For a single-server setup, open the configuration file and confirm these lines:

sudo nano /etc/elasticsearch/elasticsearch.yml
                            

cluster.name: my-log-cluster
node.name: node-1
network.host: 0.0.0.0
discovery.type: single-node
                            

discovery.type: single-node tells Elasticsearch not to expect other servers to form a cluster with — necessary for a one-server setup, but removed later if you scale to multiple nodes.

Start the service and set it to launch on boot:

sudo systemctl enable --now elasticsearch
                            

Confirm it's running (replace <password> with the elastic password from the install output):

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:<password> https://localhost:9200
                            

A working setup returns a JSON response with your cluster name and version details.

Step 4: Install and Configure Kibana

sudo apt install -y kibana
                            

Edit the config file to allow access from your own machine, not just localhost:

sudo nano /etc/kibana/kibana.yml
                            

server.host: "0.0.0.0"
server.port: 5601
                            

Start Kibana:

sudo systemctl enable --now kibana
                            

Open http://your-server-ip:5601 in a browser. On first launch, it will ask for an enrollment token to link Kibana to Elasticsearch securely — this is the token you saved back in Step 3. If you didn't save it, generate a new one from the Elasticsearch server:

sudo /usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s kibana
                            

Paste the token into Kibana's setup screen, then log in using the elastic username and the password from Step 3.

Step 5: Install and Configure Logstash

sudo apt install -y logstash
                            

Logstash works off "pipeline" configuration files — you define an input (where logs come from), a filter (how to clean/structure them), and an output (where they go). Create a basic pipeline for receiving logs from Filebeat:

sudo nano /etc/logstash/conf.d/logstash.conf
                            

input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
}

output {
  elasticsearch {
    hosts => ["https://localhost:9200"]
    user => "elastic"
    password => ""
    cacert => "/etc/elasticsearch/certs/http_ca.crt"
    index => "logs-%{+YYYY.MM.dd}"
  }
}
                            

The grok filter here is set up for standard web server access logs (Apache/Nginx-style format) — it's a pattern-matching tool that pulls structured fields (IP address, timestamp, status code, etc.) out of a single line of raw text. You'll adjust this pattern depending on what kind of logs you're actually shipping.

Start Logstash:

sudo systemctl enable --now logstash
                            

Step 6: Install Filebeat on the Servers You Want to Monitor

Filebeat is the lightweight piece installed on every server whose logs you want to collect — your web servers, application servers, and so on — not just the ELK server itself.

sudo apt install -y filebeat
                            

Point it at Logstash:

sudo nano /etc/filebeat/filebeat.yml
                            

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/nginx/access.log
      - /var/log/nginx/error.log

output.logstash:
  hosts: ["your-elk-server-ip:5044"]
                            

Start it up:

sudo systemctl enable --now filebeat
                            

Within a minute or two, logs from this server should start flowing through Logstash and into Elasticsearch.

Step 7: View Your Logs in Kibana

Back in the Kibana browser dashboard:

1. Go to Stack Management → Data Views (called "index patterns" in older versions).

2. Create a new data view matching logs-* — this tells Kibana which stored data to search through.

3. Go to Discover in the left menu. You should now see your actual log entries, searchable and filterable by time, field, or free text.

4. From here, you can build a dashboard: go to Dashboard → Create, then add visualizations like "requests over time" or "top error codes," built from the fields Logstash extracted.

This is the payoff moment — instead of grepping through log files by hand, you're now searching structured, indexed data across every server sending logs into the stack.

Keeping the Stack Running Smoothly

A few practical habits that matter once this is running for real:

• Set up Index Lifecycle Management (ILM). Logs pile up fast. ILM automatically moves old log data to cheaper storage or deletes it after a set period, so your disk doesn't fill up silently.

• Watch Elasticsearch's memory usage. It's common to give Elasticsearch too little RAM and see search performance degrade as log volume grows. If dashboards start feeling sluggish, RAM is usually the first thing to check.

• Don't skip the security setup. Since Elasticsearch 8.x turns on authentication by default, resist the temptation to disable it "just for testing" — an exposed, unauthenticated Elasticsearch instance on the open internet is a common and entirely avoidable security mistake.

• Back up your Elasticsearch data if these logs matter for compliance or auditing — by default, this is operational data, not automatically backed up like a database might be.

Conclusion

With logs flowing and searchable in Kibana, the next real decision is capacity planning — how much storage and RAM you'll need as log volume grows. If you're moving this setup into production, a bare-metal server with dedicated NVMe storage will handle Elasticsearch's disk I/O far better than shared or virtualized hosting, and gives you full control to scale RAM and storage as your log volume grows.