How to Set Up Apache Kafka with Kraft Mode on Ubuntu Server to Produce, Consume, and Partition Messages Without Zookeeper

Learning how to set up Apache Kafka with KRaft mode on Ubuntu Server to produce, consume, and partition messages without Zookeeper is one of the most practical skills for modern backend infrastructure. Kafka has long been a go-to tool for high-throughput message streaming. For years, it required Zookeeper to manage cluster metadata. KRaft mode changes that. It removes the Zookeeper dependency entirely. This makes Kafka simpler to deploy and easier to maintain. In this tutorial, you’ll install Kafka on Ubuntu, configure it to run in KRaft mode, create a topic with partitions, and test message production and consumption from the command line. By the end, you’ll have a working single-node Kafka setup ready for development or testing.

Prerequisites and Requirements for Running Kafka in KRaft Mode

Before you begin this guide on how to set up Apache Kafka with KRaft mode on Ubuntu Server to produce, consume, and partition messages without Zookeeper, make sure you have the following in place.

You will need:

  • Ubuntu Server 22.04 LTS or 20.04 LTS
  • A non-root user with sudo privileges
  • Java 11 or Java 17 installed (Kafka requires the JVM)
  • At least 2GB of RAM and 10GB of free disk space
  • Basic familiarity with the Linux command line
  • SSH access to your server

Estimated time to complete: 30–45 minutes.

You don’t need Zookeeper installed. That’s the whole point of KRaft mode. If you have Zookeeper running on the same machine, it won’t interfere, but you won’t use it here.

Check your Java version first:

java -version

If Java isn’t installed, run:

sudo apt update
sudo apt install openjdk-17-jdk -y

Confirm the installation with java -version before moving on.

Step-by-Step Guide to Set Up Apache Kafka with KRaft Mode on Ubuntu

You might also find this useful: How to Configure Redis Persistence with Rdb Snapshots and Aof Logging for Data Durability and Backup on Ubuntu Server

Step 1: Download Apache Kafka

Head to the official Apache Kafka downloads page to find the latest stable release. At the time of writing, Kafka 3.7.x is the current stable version. Download it directly to your server:

cd /opt
sudo wget https://downloads.apache.org/kafka/3.7.0/kafka_2.13-3.7.0.tgz
sudo tar -xzf kafka_2.13-3.7.0.tgz
sudo mv kafka_2.13-3.7.0 kafka

This extracts Kafka into /opt/kafka. That’s a clean, standard location for server software.

Step 2: Create a Dedicated Kafka User

Running Kafka as root is a bad idea. Create a dedicated system user instead:

sudo useradd -r -s /bin/false kafka
sudo chown -R kafka:kafka /opt/kafka

This locks down file permissions and keeps things secure.

Step 3: Generate a Cluster UUID for KRaft

KRaft mode requires a unique cluster ID. Kafka includes a built-in tool to generate one:

cd /opt/kafka
sudo -u kafka bin/kafka-storage.sh random-uuid

Copy the UUID output. You’ll use it in the next step. It looks something like 5L6g3nShT-eMCtK--X86sw.

Step 4: Configure KRaft Mode

The KRaft configuration file is already included with Kafka. Open it for editing:

sudo nano /opt/kafka/config/kraft/server.properties

Check or update these key values:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@localhost:9093
listeners=PLAINTEXT://:9092,CONTROLLER://:9093
log.dirs=/opt/kafka/kraft-combined-logs

Save and close the file. These settings tell Kafka to act as both broker and controller. That’s the combined mode, which is perfect for a single-node setup.

Step 5: Format the Storage Directory

Now format the log directory using the UUID you generated earlier. Replace YOUR-UUID-HERE with your actual UUID:

sudo -u kafka bin/kafka-storage.sh format 
  -t YOUR-UUID-HERE 
  -c config/kraft/server.properties

You should see output confirming the storage is formatted. This step initializes the metadata storage that replaces Zookeeper’s role.

Step 6: Start the Kafka Broker

Start Kafka using the KRaft config:

sudo -u kafka bin/kafka-server-start.sh config/kraft/server.properties &

Wait about 10 seconds. Then confirm it’s running:

sudo -u kafka bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

No errors means Kafka is up and listening.

Step 7: Create a Topic with Partitions

Create a test topic called akensai-events with 3 partitions and a replication factor of 1:

sudo -u kafka bin/kafka-topics.sh 
  --bootstrap-server localhost:9092 
  --create 
  --topic akensai-events 
  --partitions 3 
  --replication-factor 1

Partitions allow Kafka to distribute messages across consumers. Even on a single node, partitioning is good practice for testing realistic workloads.

Step 8: Produce Messages

Open a terminal and start the console producer:

sudo -u kafka bin/kafka-console-producer.sh 
  --bootstrap-server localhost:9092 
  --topic akensai-events

Type a few messages and press Enter after each one. For example:

Hello from Akensai
Kafka KRaft mode works
No Zookeeper needed

Press Ctrl+C to stop the producer.

Step 9: Consume Messages

Open a second terminal and run the console consumer:

sudo -u kafka bin/kafka-console-consumer.sh 
  --bootstrap-server localhost:9092 
  --topic akensai-events 
  --from-beginning

You’ll see the messages you produced appear in the terminal. This confirms your Kafka setup is producing and consuming messages correctly.

Troubleshooting Common Kafka KRaft Setup Issues

Kafka won’t start after formatting:
This usually means the log directory path in server.properties doesn’t match what you formatted. Double-check the log.dirs value and re-run the format command.

Port 9092 already in use:
Run sudo ss -tulnp | grep 9092 to find the conflicting process. Kill it or change Kafka’s listener port in server.properties.

UUID errors during format:
Make sure you’re copying the UUID exactly as generated. No extra spaces or newline characters. Regenerate it if needed.

Consumer shows no messages:
Always use --from-beginning when consuming from a fresh terminal. Without it, the consumer only reads new messages. Also check that the topic name matches exactly, including case.

Java version mismatch:
Kafka 3.x requires Java 11 or higher. Run java -version to confirm. If you see Java 8, upgrade with sudo apt install openjdk-17-jdk -y.

For deeper reference, check the official Apache Kafka documentation. It covers advanced KRaft configurations and multi-node cluster setups in detail.

Conclusion

You now know how to set up Apache Kafka with KRaft mode on Ubuntu Server to produce, consume, and partition messages without Zookeeper. You downloaded Kafka, configured it for KRaft, formatted storage with a cluster UUID, started the broker, and tested a full produce-consume cycle with a partitioned topic. That’s a complete working setup.

From here, you can explore running Kafka as a systemd service for automatic restarts. You might also look into setting up multiple partitions with consumer groups, or connecting Kafka to a real application using a client library like kafka-python or the Java SDK. KRaft mode is the future of Kafka deployments. Getting comfortable with it now puts you ahead of the curve.

Similar Posts