How to Configure Prometheus Alertmanager with Alert Rules and Email Notifications on Ubuntu Server

ByAkensai

Learning how to configure Prometheus Alertmanager with alert rules and email notifications on Ubuntu Server is one of the most practical skills you can add to your monitoring toolkit. Prometheus is a powerful open-source monitoring system, but it only becomes truly useful when you pair it with Alertmanager. Alertmanager handles alerts fired by Prometheus and routes them to the right people. In this tutorial, you’ll install and configure Alertmanager, write alert rules, and set up email notifications. By the end, your Ubuntu server will automatically send you an email when something goes wrong , whether that’s high CPU usage, a service going down, or disk space running low.

Prerequisites for Configuring Prometheus Alertmanager on Ubuntu

Before you start, make sure you have the following in place.

Required software and access:

  • Ubuntu Server 20.04 or 22.04
  • Prometheus already installed and running
  • A valid SMTP server or Gmail account for email delivery
  • Root or sudo access to the server
  • Basic familiarity with the Linux command line

If you don’t have Prometheus installed yet, check the official Prometheus installation documentation before continuing. This tutorial assumes Prometheus is already running on your server and accessible at http://localhost:9090.

Estimated time to complete: 30–45 minutes.

You’ll work with YAML configuration files throughout this guide. YAML is whitespace-sensitive. Use spaces, not tabs. A single indentation error will break your configuration.

How to Configure Prometheus Alertmanager: Installation and Setup

See also: VPN on Linode using Debian (PPTP)

Follow these steps to download, install, and configure Alertmanager on your Ubuntu server.

Step 1: Download and install Alertmanager

Visit the Alertmanager releases page to find the latest version. Then run the following commands on your server.

cd /tmp
wget https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz
tar xvf alertmanager-0.27.0.linux-amd64.tar.gz
sudo mv alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/

Step 2: Create a dedicated system user

Running Alertmanager as root is a bad idea. Create a dedicated user instead.

sudo useradd --no-create-home --shell /bin/false alertmanager
sudo mkdir /etc/alertmanager
sudo mkdir /var/lib/alertmanager
sudo chown alertmanager:alertmanager /etc/alertmanager
sudo chown alertmanager:alertmanager /var/lib/alertmanager

Step 3: Create the Alertmanager configuration file

This is the most important step. The config file controls where alerts get sent. Create it now.

sudo nano /etc/alertmanager/alertmanager.yml

Paste the following configuration. Replace the placeholder values with your actual SMTP details.

global:
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'your-app-password'
  smtp_require_tls: true

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 1h
  receiver: 'email-notifications'

receivers:
  - name: 'email-notifications'
    email_configs:
      - to: '[email protected]'
        send_resolved: true

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

If you’re using Gmail, generate an App Password from your Google account settings. Don’t use your regular Gmail password here.

Step 4: Create a systemd service for Alertmanager

You need Alertmanager to start automatically on boot. Create a systemd unit file.

sudo nano /etc/systemd/system/alertmanager.service
[Unit]
Description=Alertmanager
Wants=network-online.target
After=network-online.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager 
  --config.file=/etc/alertmanager/alertmanager.yml 
  --storage.path=/var/lib/alertmanager

[Install]
WantedBy=multi-user.target

Now enable and start the service.

sudo systemctl daemon-reload
sudo systemctl enable alertmanager
sudo systemctl start alertmanager
sudo systemctl status alertmanager

You should see active (running) in the output.

Step 5: Write Prometheus alert rules

Alert rules tell Prometheus when to fire an alert. Create a rules file.

sudo nano /etc/prometheus/alert_rules.yml
groups:
  - name: system_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[2m]))  100) > 85
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 85% for more than 2 minutes."

      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} is down"
          description: "Prometheus target has been unreachable for more than 1 minute."

      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes)  100 < 15
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Available disk space is below 15%."

Step 6: Link Alertmanager to Prometheus

Open your Prometheus configuration file and add the Alertmanager address and rules file path.

sudo nano /etc/prometheus/prometheus.yml

Add or update these sections.

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

rule_files:
  - "/etc/prometheus/alert_rules.yml"

Restart Prometheus to apply the changes.

sudo systemctl restart prometheus

Step 7: Verify everything is working

Open your browser and go to http://your-server-ip:9093. You should see the Alertmanager web interface. Go to http://your-server-ip:9090/rules in Prometheus to confirm your alert rules loaded correctly. If the rules show a green checkmark, they’re active.

Troubleshooting Common Alertmanager Configuration Problems

Even with careful setup, things can go wrong. Here are the most common issues.

Alertmanager won’t start: Run sudo journalctl -u alertmanager -f to see live logs. YAML syntax errors are the most common cause. Check your indentation carefully.

Emails aren’t being delivered: Confirm your SMTP credentials are correct. If you’re using Gmail, make sure you’ve enabled 2-Factor Authentication and generated an App Password. Regular passwords won’t work with Gmail’s SMTP.

Alert rules aren’t loading in Prometheus: Run promtool check rules /etc/prometheus/alert_rules.yml to validate your rules file. Fix any errors it reports, then restart Prometheus.

Prometheus can’t reach Alertmanager: Check that port 9093 is open. Run sudo ufw allow 9093/tcp if you’re using UFW. Also confirm Alertmanager is actually running with sudo systemctl status alertmanager.

Test alerts aren’t firing: Use amtool alert add alertname=TestAlert severity=warning to manually push a test alert into Alertmanager. This helps you confirm email delivery is working before relying on real alerts.

Conclusion

You now know how to configure Prometheus Alertmanager with alert rules and email notifications on Ubuntu Server. You installed Alertmanager, wrote a working configuration with SMTP email delivery, created alert rules for CPU, disk space, and instance availability, and connected everything back to Prometheus. Your server will now email you automatically when something needs attention.

From here, you can expand your setup by adding Slack notifications, PagerDuty routing, or more advanced alert grouping rules. You can also explore silences and inhibitions in Alertmanager

Similar Posts