How to Set Up Grafana Loki for Centralized Log Aggregation on Ubuntu Server

ByAkensai

Learning how to set up Grafana Loki for centralized log aggregation on Ubuntu Server is one of the best decisions you can make for your infrastructure. Managing logs across multiple services is painful without the right tools. Loki collects, indexes, and stores logs efficiently. It integrates directly with Grafana dashboards, giving you a powerful visual interface for searching and analyzing log data. In this tutorial, you will install Loki and Promtail on Ubuntu Server, configure them to ship logs to a central location, and connect everything to Grafana. By the end, you will have a working log aggregation stack running on your server.

Prerequisites and Requirements for Centralized Log Aggregation

Before you start, make sure your environment meets these requirements.

System requirements:

  • Ubuntu Server 20.04 or 22.04 (64-bit)
  • At least 2GB of RAM and 2 CPU cores
  • 20GB of free disk space for log storage
  • A non-root user with sudo privileges
  • Grafana already installed and running (version 9.x or higher recommended)

Assumed knowledge:

  • Basic Linux command line usage
  • Familiarity with systemd services
  • Basic understanding of log files and their locations

Estimated time: 30 to 45 minutes.

You will need an active internet connection to download binaries. Make sure port 3100 is open on your firewall if you plan to ship logs from remote servers. You can check your UFW rules with sudo ufw status. If Grafana is not yet installed, refer to the official Grafana installation documentation before continuing.

How to Set Up Grafana Loki on Ubuntu Server

Related tutorial: How to Install and Configure Fail2ban on Ubuntu to Protect Against Brute-force Attacks

Follow these steps carefully. Each step builds on the last.

Step 1: Download the Loki binary

Grab the latest Loki release directly from GitHub. At the time of writing, version 2.9.x is stable.

cd /tmp
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.2/loki-linux-amd64.zip"
unzip loki-linux-amd64.zip
sudo mv loki-linux-amd64 /usr/local/bin/loki
sudo chmod +x /usr/local/bin/loki

Verify the installation worked with loki --version. You should see version output in your terminal.

Step 2: Create a Loki configuration file

Loki needs a config file to know where to store data and how to handle ingestion.

sudo mkdir -p /etc/loki
sudo nano /etc/loki/loki-config.yaml

Paste the following configuration into the file:

auth_enabled: false

server:
  http_listen_port: 3100

ingester:
  lifecycler:
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
  chunk_idle_period: 5m
  chunk_retain_period: 30s

schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb_shipper:
    active_index_directory: /var/loki/index
    cache_location: /var/loki/cache
    shared_store: filesystem
  filesystem:
    directory: /var/loki/chunks

limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h

chunk_store_config:
  max_look_back_period: 0s

table_manager:
  retention_deletes_enabled: false
  retention_period: 0s

Save and close the file with Ctrl+X, then Y, then Enter.

Step 3: Create the storage directories

sudo mkdir -p /var/loki/index /var/loki/cache /var/loki/chunks
sudo useradd --system --no-create-home loki
sudo chown -R loki:loki /var/loki

Step 4: Create a systemd service for Loki

Running Loki as a systemd service keeps it alive after reboots.

sudo nano /etc/systemd/system/loki.service

Add this 

[Unit]
Description=Loki Log Aggregation Service
After=network.target

[Service]
User=loki
ExecStart=/usr/local/bin/loki -config.file=/etc/loki/loki-config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target

Now enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable loki
sudo systemctl start loki

Check the status with sudo systemctl status loki. You should see it listed as active and running.

Step 5: Install and configure Promtail

Promtail is the log shipping agent. It reads log files and sends them to Loki.

cd /tmp
curl -O -L "https://github.com/grafana/loki/releases/download/v2.9.2/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
sudo mv promtail-linux-amd64 /usr/local/bin/promtail
sudo chmod +x /usr/local/bin/promtail

Create a Promtail config file:

sudo mkdir -p /etc/promtail
sudo nano /etc/promtail/promtail-config.yaml

Paste this configuration:

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /var/log/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets:
          - localhost
        labels:
          job: varlogs
          __path__: /var/log/log

Step 6: Create a systemd service for Promtail

sudo nano /etc/systemd/system/promtail.service
[Unit]
Description=Promtail Log Shipping Agent
After=network.target

[Service]
ExecStart=/usr/local/bin/promtail -config.file=/etc/promtail/promtail-config.yaml
Restart=on-failure

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable promtail
sudo systemctl start promtail

Step 7: Connect Loki to Grafana

Open your Grafana interface in a browser. Go to Configuration → Data Sources → Add data source. Select Loki from the list. In the URL field, enter http://localhost:3100. Click Save & Test. Grafana will confirm the connection is working. You can now use the Explore tab to query your logs using LogQL. For more on writing queries, check the official LogQL documentation.

Troubleshooting Common Loki and Promtail Issues

Loki service won’t start:
Check the journal logs with sudo journalctl -u loki -f. Missing directories or wrong file permissions are the most common causes. Re-run the chown command from Step 3.

Promtail isn’t shipping logs:
Confirm Loki is running and reachable on port 3100. Run curl http://localhost:3100/ready. You should get a response of ready. If you don’t, Loki hasn’t fully started yet.

No logs appearing in Grafana:
Check that your log file paths in the Promtail config actually exist. Run ls /var/log/log to confirm. Also check that the Promtail service is active with sudo systemctl status promtail.

Port 3100 is blocked:
Open the port with sudo ufw allow 3100/tcp. This is only needed if you are shipping logs from a remote server.

Schema errors in Loki logs:
If you see schema version errors, double-check your loki-config.yaml file. A copy-paste formatting issue can break YAML parsing. Use loki -config.file=/etc/loki/loki-config.yaml -verify-config to

Similar Posts