How to Set Up Celery Distributed Task Queues with Redis and Flower Monitoring on Ubuntu Server

ByAkensai

Learning how to set up Celery distributed task queues with Redis and Flower monitoring on Ubuntu Server is one of the most valuable skills for any backend developer or server administrator. Celery lets your application handle time-consuming tasks , like sending emails, processing images, or running reports , in the background. This keeps your web app fast and responsive. Redis acts as the message broker, passing tasks between your app and Celery workers. Flower gives you a real-time web dashboard to monitor those workers. In this tutorial, you’ll install and configure all three components on a fresh Ubuntu server. You’ll also learn how to run Celery as a systemd service so it starts automatically on reboot. By the end, you’ll have a fully working task queue system ready for production use.

Prerequisites for Setting Up Celery with Redis and Flower

Before you start, make sure your environment meets these requirements.

You’ll need:

  • Ubuntu 20.04 or 22.04 server (fresh install recommended)
  • A non-root user with sudo privileges
  • Python 3.8 or higher installed
  • pip and virtualenv available
  • A basic Python web application (Flask or Django works fine)
  • SSH access to your server

Assumed knowledge: You should be comfortable with the Linux command line. You don’t need to be a Python expert, but knowing basic Python syntax helps.

Estimated time: 30–45 minutes from start to finish.

If you don’t have Python installed yet, run sudo apt install python3 python3-pip python3-venv -y before continuing. You should also update your package list first with sudo apt update && sudo apt upgrade -y.

How to Set Up Celery Distributed Task Queues with Redis on Ubuntu

Related tutorial: How to Configure Ssl/tls Certificates with Let’s Encrypt on Nginx

Follow these steps carefully. Each step builds on the last.

Step 1: Install Redis on Ubuntu

Redis is the message broker. It holds tasks in a queue until a Celery worker picks them up.

sudo apt install redis-server -y
sudo systemctl enable redis-server
sudo systemctl start redis-server

Verify Redis is running:

sudo systemctl status redis-server

You should see active (running) in the output. Test the connection with redis-cli ping. Redis should respond with PONG.

Step 2: Create a Python Virtual Environment

Always use a virtual environment. It keeps your project dependencies isolated.

mkdir ~/myproject && cd ~/myproject
python3 -m venv venv
source venv/bin/activate

Your terminal prompt will change to show (venv). This confirms the environment is active.

Step 3: Install Celery, Redis Client, and Flower

Now install the required Python packages inside your virtual environment.

pip install celery redis flower

Check the installed versions with pip show celery. You want version 5.x or higher. The official Celery documentation covers all available configuration options in detail.

Step 4: Create Your Celery Application

Create a file called celery_app.py in your project directory.

nano ~/myproject/celery_app.py

Add the following 

from celery import Celery

app = Celery(
    'myproject',
    broker='redis://localhost:6379/0',
    backend='redis://localhost:6379/0'
)

app.conf.update(
    task_serializer='json',
    result_serializer='json',
    accept_content=['json'],
    timezone='UTC',
    enable_utc=True,
)

@app.task
def add(x, y):
    return x + y

Save the file with Ctrl+O, then exit with Ctrl+X.

Step 5: Test Celery Worker Manually

Start a Celery worker from your project directory to make sure everything works.

cd ~/myproject
source venv/bin/activate
celery -A celery_app worker --loglevel=info

You should see the worker start and display registered tasks. Open a second terminal and test the task:

cd ~/myproject
source venv/bin/activate
python3 -c "from celery_app import add; result = add.delay(4, 6); print(result.get())"

The output should print 10. If it does, your task queue is working correctly.

Step 6: Launch Flower for Monitoring

Flower is a web-based monitoring tool. It shows active workers, task history, and queue stats.

celery -A celery_app flower --port=5555

Open your browser and go to http://your-server-ip:5555. You’ll see the Flower dashboard with your active worker listed.

Step 7: Run Celery as a systemd Service

Running Celery manually isn’t practical for production. Set it up as a systemd service instead.

Create the service file:

sudo nano /etc/systemd/system/celery.service

Add this configuration:

[Unit]
Description=Celery Worker Service
After=network.target redis.service

[Service]
Type=forking
User=your-username
Group=your-username
WorkingDirectory=/home/your-username/myproject
ExecStart=/home/your-username/myproject/venv/bin/celery 
    -A celery_app worker 
    --loglevel=info 
    --logfile=/var/log/celery/celery.log 
    --pidfile=/var/run/celery/celery.pid
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID
Restart=always

[Install]
WantedBy=multi-user.target

Replace your-username with your actual Ubuntu username. Create the required directories:

sudo mkdir -p /var/log/celery /var/run/celery
sudo chown your-username:your-username /var/log/celery /var/run/celery

Enable and start the service:

sudo systemctl daemon-reload
sudo systemctl enable celery
sudo systemctl start celery
sudo systemctl status celery

Troubleshooting Common Celery and Redis Issues

Even with careful setup, things can go wrong. Here are the most common problems and how to fix them.

Problem: Celery can’t connect to Redis

Check that Redis is running: sudo systemctl status redis-server. Also confirm your broker URL in celery_app.py matches the Redis port. The default port is 6379.

Problem: Tasks stay in PENDING state forever

This usually means no worker is running. Check your systemd service status with sudo systemctl status celery. Review logs at /var/log/celery/celery.log for specific errors.

Problem: Flower dashboard shows no workers

Make sure you’re running Flower with the same -A celery_app flag that matches your application module name. Also check that your firewall allows port 5555:

sudo ufw allow 5555/tcp

Problem: Permission denied on log or pid directories

Run the chown commands again and make sure the user in your service file matches the actual file owner.

Security tip: Don’t expose Flower to the public internet without authentication. Add basic auth with --basic-auth=username:password when starting Flower. The Ubuntu Server documentation has good guidance on firewall and security best practices.

Conclusion

You now know how to set up Celery distributed task queues with Redis and Flower monitoring on Ubuntu Server from scratch. You installed Redis as a message broker, configured a Celery application, ran background tasks, and set up Flower for real-time monitoring. You also created a systemd service to keep Celery running after reboots. This setup handles background jobs reliably and scales well as your application grows. Your next steps could include adding multiple Celery workers, configuring task priorities, or setting up dedicated queues for different task types. You can also explore Celery Beat for scheduling periodic tasks on a timer. With this foundation in place, your application can handle heavy workloads without slowing down user-facing requests.

Similar Posts