Phillip Groves

As a distributed system, Kafka does not only require streaming data to function; it also needs to know information about the distributed system itself, known as metadata. Which node is the controller? What is the topic configuration? Who is assigned to access control lists (ACLs)? Metadata answers all of these questions. Metadata management with Kafka has evolved through its history, and recently underwent its most drastic change: a migration from ZooKeeper to KRaft. In this session, we will discuss what metadata is and why it is important in an event streaming system. We will also explore the difference between ZooKeeper and KRaft, learn the reasons why a migration is necessary, and understand the general operations that may be followed for migrating with the Confluent Platform. ZooKeeper has been paramount to the Kafka ecosystem since its inception. Being a general system for managing distributed metadata, ZooKeeper requires separate nodes within a cluster. For this reason, and more that we will discuss, a quorum-based approach was developed from the ground-up. KRaft, based on the Raft consensus algorithm, allows each Kafka broker to dually act as a metadata manager, reducing…