Lesson 13. KRaft, and What Replaced ZooKeeper

Mission link: Stage 8 opens the cluster's own coordination layer, the part every prior lesson quietly assumed worked: something has to decide which broker leads which partition, and track that decision durably enough to survive a failure. Older Kafka clusters answered this with an external system, ZooKeeper; current Kafka answers it with KRaft, a consensus protocol Kafka runs itself.
Primary source: Docs: "KRaft", Apache Kafka
Prerequisites: Lesson 12, Partition

Warm-up

1. ▢ Why does max.poll.interval.ms catch a failure session.timeout.ms alone would miss?

2. ▢ A producer's batch.size is reached well before linger.ms elapses. Does the producer wait out the rest of linger.ms anyway?

Know this

What ZooKeeper used to hold for a Kafka cluster

Older Kafka clusters stored their cluster metadata, broker registration, topic and partition configuration, ACLs, and which broker was the controller, in ZooKeeper, a separate distributed coordination service Kafka depended on but did not implement itself. Controller election worked through an ephemeral znode: whichever broker successfully created it became the controller, and if that broker died, the znode disappeared and a new election ran. This worked, but it meant operating two distributed systems, each with its own quorum, its own security configuration, and its own failure modes, to run one Kafka cluster.

The specific cost that motivated replacing it: a full state re-fetch on every failover

The expensive part wasn't ZooKeeper's normal operation; it was what happened on controller failover. A newly elected controller had to re-read the cluster's entire metadata state from ZooKeeper before it could do anything, then push that full state back out to every broker, rather than picking up from wherever the previous controller left off. At a small partition count this was fast enough not to matter; at the partition counts a large cluster actually accumulates, exactly what lesson 7 warned a topic layout not to grow carelessly, this full re-fetch-and-rebroadcast became slow enough to noticeably lengthen how long a cluster stayed partially unavailable during failover.

KRaft: Kafka's own metadata quorum, replicated via Raft, with no full re-fetch

KRaft (Kafka Raft) removes the external dependency by having Kafka run its own metadata consensus, using the Raft protocol, among a small set of nodes called the controller quorum. Metadata changes are appended to a replicated metadata log, and every controller in the quorum, not just the active one, continuously replays that log as it arrives. One controller is active (the QuorumController, a single-threaded event loop that processes every metadata change atomically, topic creation, partition reassignment, leader election, broker heartbeats, in the order it received them); the rest are standby, already caught up because they've been replaying the same log the whole time. Failover, then, is nothing more than a standby that was already current becoming active, not a fresh read of the entire cluster's state from an external store.

process.roles: a node is a broker, a controller, or both

A KRaft node's process.roles setting decides its job: broker (serves produce and fetch requests), controller (participates in the metadata quorum), or broker,controller together, combined mode, common for a small cluster or local development where running dedicated controller nodes isn't worth the overhead. A larger production cluster typically runs a small, dedicated controller quorum (often 3 nodes, an odd number so a majority is always well-defined) separate from the brokers actually serving traffic, so metadata-quorum load never competes with data-plane load on the same process. ZooKeeper-specific configuration has been removed from current Kafka entirely: this isn't an optional mode alongside ZooKeeper anymore, it's the only way a current Kafka cluster manages its own metadata.

Practice

1. ▢ A cluster running ZooKeeper mode loses its controller broker. What has to happen before the new controller can start processing metadata changes, and why did this get slower as the cluster's partition count grew?

2. ▢ In KRaft mode, why doesn't a newly active controller need to do that same full re-fetch?

3. ▢ A small team runs a single-node development cluster and sets process.roles=broker,controller on it. What does this combined mode actually mean, and why would a large production cluster typically avoid it?

4. ▢ Why does the controller quorum typically use an odd number of nodes, such as 3 or 5?

5. ▢ Which claim correctly describes the difference between ZooKeeper mode and KRaft mode?

   • a) Both store cluster metadata in an external system; KRaft simply uses a faster external database than ZooKeeper
   • b) ZooKeeper mode requires a new controller to re-fetch and rebroadcast the cluster's full metadata state on failover; KRaft mode's standby controllers continuously replay a replicated metadata log, so failover needs no such re-fetch
   • c) KRaft is an optional mode that current Kafka still lets a cluster disable in favor of ZooKeeper
   • d) process.roles=broker,controller is invalid configuration and only single-role nodes are supported

Real-world reps

• [ ] For a Kafka cluster you have access to, check whether its controller nodes run in combined mode or as a dedicated controller quorum, and how many controller nodes the quorum has.
• [ ] If you've operated (or read a postmortem of) an older ZooKeeper-mode cluster, look for whether a controller failover was ever the visible cause of a longer-than-expected availability gap.
• [ ] Tomorrow: read the primary source's KRaft documentation in full, and note what kafka-metadata-quorum.sh describe --status actually reports about a running quorum's health.

Going further

• Docs: "KRaft", Apache Kafka
• Docs: "ZooKeeper to KRaft Migration", Apache Kafka
• Resources

Not landing? Reread the primary source at the top, since this lesson compresses it and compression is where understanding leaks. Check the glossary for any term that felt slippery.

If the lesson itself is unclear rather than the material, that is a defect: open an issue.