How to Use Prometheus Recording Rules and Grafana Dashboard Provisioning as Code on Ubuntu Server

ByAkensai

Learning how to use Prometheus recording rules and Grafana dashboard provisioning as code on Ubuntu Server gives your monitoring stack a serious upgrade. Instead of clicking through dashboards manually or running expensive queries on every page load, you define everything in config files. This approach makes your setup repeatable, version-controllable, and much faster to recover after a failure.

In this tutorial, you will set up Prometheus recording rules to pre-compute heavy PromQL queries. You will also configure Grafana to automatically load dashboards from JSON files on disk. By the end, your entire monitoring configuration lives in code. That means you can store it in Git, deploy it across multiple servers, and never lose a dashboard again.

Prerequisites for Prometheus Recording Rules and Grafana Dashboard Provisioning

Before you start, make sure you have the following in place:

– Ubuntu Server 22.04 or 24.04 (fresh or existing install)
– Prometheus installed and running (version 2.40 or later)
– Grafana installed and running (version 9.0 or later)
– A non-root user with sudo privileges
– Basic familiarity with YAML and JSON file formats
– A terminal or SSH connection to your server

You should already have Prometheus scraping at least one target. A Node Exporter setup works perfectly for this tutorial. Estimated time to complete: 45–60 minutes.

If you haven’t installed Prometheus yet, refer to the official Prometheus getting started guide before continuing.

Step-by-Step Guide to Recording Rules and Dashboard Provisioning

For a related walkthrough, see: How to Configure Redis Object Cache for Wordpress Performance Optimization

Step 1: Create a Prometheus recording rules file

Recording rules tell Prometheus to evaluate a PromQL expression on a schedule. The result gets stored as a new time series. This saves CPU time on dashboards that run the same heavy query repeatedly.

Create a rules directory and a new rules file:

sudo mkdir -p /etc/prometheus/rules
sudo nano /etc/prometheus/rules/node_recording.yml

Add the following 

groups:
  - name: node_recording_rules
    interval: 1m
    rules:
      - record: job:node_cpu_usage:avg
        expr: avg without(cpu, mode) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))
      - record: job:node_memory_used_bytes:avg
        expr: avg(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes)
      - record: job:node_filesystem_used_ratio:avg
        expr: avg(1 - node_filesystem_free_bytes / node_filesystem_size_bytes)

Save and close the file.

Step 2: Reference the rules file in your Prometheus config

Open your main Prometheus configuration file:

sudo nano /etc/prometheus/prometheus.yml

Add or update the rule_files section:

rule_files:
  - "/etc/prometheus/rules/.yml"

Make sure this block sits at the top level of the YAML file. It should not be nested under scrape_configs.

Step 3: Validate your Prometheus configuration

Always check your config before reloading. A bad YAML file will crash Prometheus on restart.

promtool check config /etc/prometheus/prometheus.yml

You should see: SUCCESS: /etc/prometheus/prometheus.yml is valid prometheus config file syntax

Also validate the rules file directly:

promtool check rules /etc/prometheus/rules/node_recording.yml

Step 4: Reload Prometheus to apply the rules

Send a reload signal instead of restarting. This avoids downtime:

sudo systemctl reload prometheus

If your Prometheus binary doesn’t support reload via systemd, use:

curl -X POST http://localhost:9090/-/reload

Open the Prometheus UI at http://your-server-ip:9090/rules. You should see your new recording rules listed and healthy.

Step 5: Create a Grafana provisioning directory for dashboards

Grafana can watch a folder on disk and load dashboards automatically. This is the core of dashboard provisioning as code.

Create the required directories:

sudo mkdir -p /etc/grafana/provisioning/dashboards
sudo mkdir -p /var/lib/grafana/dashboards

Step 6: Write the Grafana dashboard provisioning config

Create a provisioning config file:

sudo nano /etc/grafana/provisioning/dashboards/node_dashboards.yml

Add this 

apiVersion: 1

providers:
  - name: "Node Exporter Dashboards"
    orgId: 1
    folder: "Infrastructure"
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    options:
      path: /var/lib/grafana/dashboards

This tells Grafana to scan /var/lib/grafana/dashboards every 30 seconds. Any JSON file it finds gets loaded automatically.

Step 7: Add a dashboard JSON file

You can export any existing Grafana dashboard as JSON. For this tutorial, create a minimal dashboard that uses your new recording rules:

sudo nano /var/lib/grafana/dashboards/node_overview.json

Paste in a valid Grafana dashboard JSON. You can grab a community dashboard from the Grafana dashboard library and save it as a file here. Make sure the JSON references your recorded metric names like job:node_cpu_usage:avg in its panel queries.

Step 8: Set correct file permissions

Grafana runs as the grafana user. Make sure it can read your files:

sudo chown -R grafana:grafana /var/lib/grafana/dashboards
sudo chmod -R 755 /var/lib/grafana/dashboards

Step 9: Restart Grafana and verify

sudo systemctl restart grafana-server
sudo systemctl status grafana-server

Open Grafana at http://your-server-ip:3000. Navigate to Dashboards. You should see a new folder called “Infrastructure” with your dashboard loaded inside it.

Troubleshooting Common Issues with Recording Rules and Dashboard Provisioning

Prometheus rules not loading

Run promtool check rules on your file. YAML indentation errors are the most common cause. Make sure you use spaces, not tabs.

Recording rules show as “inactive” in the UI

Check that Prometheus is actually scraping the metrics your rule depends on. Go to http://localhost:9090/targets and confirm your targets are UP.

Grafana dashboard doesn’t appear after restart

Check the Grafana logs for provisioning errors:

sudo journalctl -u grafana-server -n 50 --no-pager

Look for lines containing “provisioning”. A malformed JSON file will silently fail. Validate your JSON with python3 -m json.tool your_dashboard.json before placing it in the dashboards folder.

Grafana shows “Dashboard not found” after provisioning

This usually means the uid field in your JSON conflicts with an existing dashboard. Open the JSON file and change the uid value to something unique.

Prometheus reload returns a 405 error

Enable the lifecycle API in your Prometheus startup flags:

--web.enable-lifecycle

Add this flag to your systemd service file under ExecStart.

Conclusion: Managing Your Monitoring Stack as Code

You now know how to use Prometheus recording rules and Grafana dashboard provisioning as code on Ubuntu Server. Your recording rules pre-compute expensive queries and store the results as new metrics. Your Grafana dashboards load automatically from JSON files on disk.

This setup changes how you manage monitoring. Everything is auditable. You can track changes in Git. You can rebuild your entire monitoring stack from files alone. That’s a huge win for any production environment.

From here, consider storing your rules files and dashboard JSON in a Git repository. You can also extend this pattern to Grafana datasource provisioning and alerting rules. Both follow the same file-based approach you used today.

Similar Posts