How to Expose Custom Application Metrics to Prometheus Using the Node Exporter Textfile Collector on Ubuntu Server

ByAkensai

How to Expose Custom Application Metrics to Prometheus Using the Node Exporter Textfile Collector on Ubuntu Server is a skill every serious Linux administrator should have. If you run custom applications on Ubuntu, Prometheus won’t automatically know how to scrape their internal data. The Textfile Collector solves this problem. It lets you write metric data to plain text files in a specific format. Node Exporter then picks up those files and exposes the data to Prometheus automatically.

This tutorial walks you through the entire setup. You’ll configure Node Exporter to read from a textfile directory, write a custom metrics script, and verify that Prometheus is scraping your data correctly. By the end, you’ll have a working pipeline for any custom metric your application can output.

Prerequisites and Requirements for Exposing Custom Metrics

Before you start, make sure you have the following in place:

– A Ubuntu Server 20.04 or 22.04 instance with sudo access
– Prometheus installed and running on the same server or accessible remotely
– Node Exporter already installed (version 1.0 or higher recommended)
– Basic familiarity with the Linux command line and systemd services
– A text editor like nano or vim

Estimated time to complete: 30–45 minutes.

You don’t need to be a Prometheus expert. If you’ve set up Node Exporter before and know how to edit systemd unit files, you’re ready. This guide assumes Node Exporter is installed at /usr/local/bin/node_exporter and managed by systemd.

How to Expose Custom Application Metrics to Prometheus Using the Textfile Collector

For a related walkthrough, see: How to Harden Linux Server Security by Securing Filesystem Mount Options for /tmp, /dev/shm, and /proc on Ubuntu Server

Follow these steps carefully. Each one builds on the last.

Step 1: Create the Textfile Collector Directory

Node Exporter reads .prom files from a directory you specify. Create that directory now.

sudo mkdir -p /var/lib/node_exporter/textfile_collector
sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector

If you don’t have a dedicated node_exporter system user, create one first:

sudo useradd --no-create-home --shell /bin/false node_exporter

Step 2: Update the Node Exporter Systemd Service

You need to pass the --collector.textfile.directory flag to Node Exporter. Edit its systemd unit file:

sudo nano /etc/systemd/system/node_exporter.service

Find the ExecStart line and update it to include the flag:

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter 
  --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

Save the file and reload systemd:

sudo systemctl daemon-reload
sudo systemctl restart node_exporter
sudo systemctl status node_exporter

Confirm the service is active before moving on.

Step 3: Write Your First Custom Metric File

Prometheus uses the OpenMetrics exposition format for text-based metrics. Each .prom file must follow this format exactly.

Create a test metric file manually:

sudo nano /var/lib/node_exporter/textfile_collector/app_metrics.prom

Add the following 

# HELP app_active_users Number of currently active users in the application
# TYPE app_active_users gauge
app_active_users 42

# HELP app_jobs_processed_total Total number of background jobs processed
# TYPE app_jobs_processed_total counter
app_jobs_processed_total 1587

Save and close the file. Make sure the file is readable by the node_exporter user:

sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector/app_metrics.prom

Step 4: Verify Node Exporter Is Serving the Metrics

Check that Node Exporter is exposing your custom metrics on its default port:

curl http://localhost:9100/metrics | grep app_

You should see output like this:

# HELP app_active_users Number of currently active users in the application
# TYPE app_active_users gauge
app_active_users 42

If you see your metrics here, the Textfile Collector is working correctly.

Step 5: Automate Metric Collection with a Shell Script

Manually editing .prom files doesn’t scale. Write a script that collects data from your application and outputs it in the correct format. Use atomic writes to prevent Prometheus from reading a half-written file.

sudo nano /usr/local/bin/collect_app_metrics.sh

Add this script:

#!/bin/bash

TEXTFILE_DIR="/var/lib/node_exporter/textfile_collector"
OUTPUT_FILE="$TEXTFILE_DIR/app_metrics.prom"
TEMP_FILE="$OUTPUT_FILE.$$"

# Replace these commands with your actual application queries
ACTIVE_USERS=$(who | wc -l)
DISK_USED=$(df / | awk 'NR==2 {print $3}')

cat > "$TEMP_FILE" <<EOF
# HELP app_active_users Number of active sessions on the server
# TYPE app_active_users gauge
app_active_users $ACTIVE_USERS

# HELP app_root_disk_used_kb Root filesystem disk usage in kilobytes
# TYPE app_root_disk_used_kb gauge
app_root_disk_used_kb $DISK_USED
EOF

mv "$TEMP_FILE" "$OUTPUT_FILE"
chown node_exporter:node_exporter "$OUTPUT_FILE"

Make the script executable:

sudo chmod +x /usr/local/bin/collect_app_metrics.sh

Step 6: Schedule the Script with Cron

Run the script every minute using cron:

sudo crontab -e

Add this line:

     /usr/local/bin/collect_app_metrics.sh

Wait one minute, then check your metrics endpoint again to confirm the values are updating.

Step 7: Confirm Prometheus Is Scraping the Data

Open your Prometheus web UI (usually at http://your-server-ip:9090). Go to the Graph tab and search for app_active_users. If Prometheus is already scraping Node Exporter, your custom metrics appear immediately. No extra Prometheus configuration is needed.

Troubleshooting Common Textfile Collector Problems

Metrics don’t appear in the Node Exporter output

Check that the --collector.textfile.directory flag is set correctly in your service file. Run sudo systemctl status node_exporter and look at the ExecStart line.

Also confirm your .prom file has no syntax errors. Node Exporter silently skips malformed files. Run this to check:

promtool check metrics < /var/lib/node_exporter/textfile_collector/app_metrics.prom

Permission denied errors

The node_exporter user must own or have read access to all files in the textfile directory. Double-check ownership:

ls -la /var/lib/node_exporter/textfile_collector/

Stale metrics appearing

If your cron job fails, old data stays in the .prom file. Add a timestamp metric to detect this:

# HELP app_metrics_last_updated Unix timestamp of last successful metric collection
# TYPE app_metrics_last_updated gauge
app_metrics_last_updated $(date +%s)

You can then alert in Prometheus when this value is too old.

Node Exporter not restarting after service file changes

Always run sudo systemctl daemon-reload before restarting the service. Skipping this step means systemd uses the old configuration.

Conclusion: Exposing Custom Metrics to Prometheus on Ubuntu Server

You now know how to expose custom application metrics to Prometheus using the Node Exporter Textfile Collector on Ubuntu Server. You’ve set up the directory, updated the systemd service, written a metric collection script, and scheduled it with cron. This approach works for any application that can’t be directly instrumented with a Prometheus client library.

From here, you can build Grafana dashboards from your custom metrics. You can also create Prometheus alerting rules based on the values you’re now collecting. The Textfile Collector is one of the simplest and most flexible tools in the Prometheus ecosystem. Use it whenever you need quick, scriptable metric exposure without modifying application source code.

Similar Posts