How to Build a Background Job Queue in Node.js with Bullmq and Redis

ByAkensai

Learning how to build a background job queue in Node.js with BullMQ and Redis gives your application the power to handle time-consuming tasks without blocking your main process. Think about sending emails, resizing images, or processing payments. These tasks shouldn’t make a user wait. A background job queue offloads that work so your app stays fast and responsive. In this tutorial, you’ll set up Redis on a Linux server, install BullMQ, create a queue, add jobs, and process them with a worker. By the end, you’ll have a working job queue system you can build on for real production applications.

Prerequisites and Requirements for Building a Node.js Job Queue

Before you start, make sure you have the following in place.

Required software and access:

  • A Linux server running Ubuntu 20.04 or 22.04 (local or VPS)
  • Node.js version 16 or higher installed
  • npm version 7 or higher
  • Redis installed and running locally or via a remote instance
  • Basic terminal access and sudo privileges

Assumed knowledge:

  • Comfortable with the Linux command line
  • Familiar with JavaScript and Node.js basics
  • Understands what async/await means in JavaScript

Estimated time: 30–45 minutes

If you don’t have Redis installed yet, you’ll set it up in Step 1. BullMQ is a TypeScript-first job queue library built on top of Redis. It handles retries, delays, priorities, and concurrency out of the box. You can read more about it in the official BullMQ documentation.

Step-by-Step Guide to Building a Background Job Queue in Node.js with BullMQ and Redis

See also: How to Migrate Wordpress Classic Meta Boxes to Modern Block Editor Sidebar for Wordpress 7.0

Follow these steps carefully. Each one builds on the last.

Step 1: Install Redis on Ubuntu

Redis is the backbone of your queue. BullMQ uses it to store and manage jobs.

Run the following commands to install Redis:

sudo apt update
sudo apt install redis-server -y
sudo systemctl enable redis-server
sudo systemctl start redis-server

Verify Redis is running with:

redis-cli ping

You should see PONG in the terminal. If you don’t, check the service status with sudo systemctl status redis-server.

Step 2: Create Your Node.js Project

Set up a fresh project directory and initialize it with npm.

mkdir bullmq-demo
cd bullmq-demo
npm init -y

This creates a package.json file. You’ll use this project folder for all your queue files.

Step 3: Install BullMQ

Install the BullMQ package using npm:

npm install bullmq

BullMQ requires no separate Redis client installation. It manages the Redis connection internally. This keeps your setup clean and simple.

Step 4: Create the Queue

Create a file called queue.js. This file defines your queue and adds jobs to it.

// queue.js
const { Queue } = require('bullmq');

const emailQueue = new Queue('emailQueue', {
  connection: {
    host: '127.0.0.1',
    port: 6379,
  },
});

async function addEmailJob(to, subject, body) {
  await emailQueue.add('sendEmail', {
    to,
    subject,
    body,
  });
  console.log(`Job added: send email to ${to}`);
}

addEmailJob('[email protected]', 'Welcome!', 'Thanks for signing up.');

The Queue constructor takes a name and a connection config. The job name 'sendEmail' helps you identify jobs in the queue. The second argument is the job data payload.

Step 5: Create the Worker

The worker picks up jobs and processes them. Create a file called worker.js:

// worker.js
const { Worker } = require('bullmq');

const worker = new Worker('emailQueue', async (job) => {
  console.log(`Processing job ${job.id}`);
  console.log(`Sending email to: ${job.data.to}`);
  console.log(`Subject: ${job.data.subject}`);

  // Simulate sending email
  await new Promise((resolve) => setTimeout(resolve, 1000));

  console.log('Email sent successfully!');
}, {
  connection: {
    host: '127.0.0.1',
    port: 6379,
  },
});

worker.on('completed', (job) => {
  console.log(`Job ${job.id} completed`);
});

worker.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed: ${err.message}`);
});

The worker listens on the same queue name: 'emailQueue'. It processes each job using the async callback. The completed and failed events let you track job status.

Step 6: Run the Queue and Worker

Open two terminal windows. In the first, start the worker:

node worker.js

In the second terminal, add a job to the queue:

node queue.js

You’ll see the worker pick up the job and log the output. This confirms your queue is working end to end.

Step 7: Add Job Options for Retries and Delays

Real-world queues need retry logic. Update your addEmailJob function in queue.js:

await emailQueue.add('sendEmail', {
  to,
  subject,
  body,
}, {
  attempts: 3,
  backoff: {
    type: 'exponential',
    delay: 2000,
  },
  delay: 5000,
});

The attempts option retries failed jobs up to 3 times. The backoff setting increases the wait time between retries. The delay option waits 5 seconds before the job runs for the first time. These options make your queue production-ready.

Troubleshooting Common Issues with BullMQ and Redis

Even with careful setup, you may hit a few snags. Here are the most common problems and how to fix them.

Problem: “Connection refused” error when starting the worker

This means Redis isn’t running. Fix it with:

sudo systemctl start redis-server

Double-check the host and port in your connection config match your Redis setup.

Problem: Jobs are added but never processed

Make sure your worker is running before or right after you add jobs. The worker must be active and listening on the correct queue name. Queue names are case-sensitive. 'emailQueue' and 'EmailQueue' are different queues.

Problem: BullMQ throws “ERR unknown command” in older Redis versions

BullMQ requires Redis version 5 or higher. Check your version with:

redis-server --version

If you’re running an older version, upgrade Redis using the official Redis installation guide for Linux.

Problem: Worker crashes on unhandled errors

Wrap your job logic in a try/catch block inside the worker callback. BullMQ will still trigger the failed event, but catching errors yourself gives you more control over logging.

Tip: Use the BullMQ Board or Bull Board UI to monitor your queues visually. It shows active, waiting, completed, and failed jobs in a web dashboard. This is especially useful during development.

Conclusion

You now know how to build a background job queue in Node.js with BullMQ and Redis from scratch. You installed Redis, created a queue, added jobs with retry logic, and processed them with a worker. This pattern works for email delivery, report generation, webhook processing, and much more. From here, you can explore adding multiple workers for concurrency, using job priorities, or connecting BullMQ to a PostgreSQL database for persistent job records. Understanding how to build a background job queue in Node.js with BullMQ and Redis is a skill that scales with your application as it grows. Start small, test your setup thoroughly, and expand from there.

Similar Posts