David Jacot

The Apache Kafka Clients have come a long way since 4.0, with exciting improvements landing across many areas of the Consumer and Producer. The new rebalance protocol is GA and the Classic one is on its way out; the producer can allocate memory on demand; rebalances are faster and more stable; and a steady stream of smaller KIPs has extended client functionality, sharpened observability, and made applications more resilient. The catch: many of these wins only pay off if you know they exist, turn them on, and migrate off the old defaults safely. Others need some configuration tuning to best fit your application. This session is the map: what's improved, why it matters for your applications, and how to adopt it without breaking things and get the most out of it. We'll tour what's landed across the 4.x releases and what's coming, with detail on how each change works, the benefits they bring, and how to adopt it today. On the consumer side, we'll cover the Classic protocol's deprecation timeline (KIP-1274) and how to migrate safely, assignment epochs and offload (KIP-1251, KIP-1263), the resilience improvements around DNS resolution (KIP-909), new regex subscriptions, and more. On the…

Rebalances are the part of Kafka everyone fears and nobody can see. The new consumer group protocol (KIP-848) changed how they work — server-driven and incremental instead of the classic stop-the-world dance — but both protocols are still a black box. KAOS opens them up: a live, open-source simulator built to teach both protocols. Watch a rebalance unfold under each one in your browser — spin up consumers, kill them, see partitions reconcile — and finally see why the new one is smoother. I'll demo it end to end and show how it turns two notoriously opaque protocols into something you can actually watch, teach, and debug.

Apache Kafka 4.0 makes the new consumer group rebalance protocol (KIP‑848) generally available, shifting assignment computation from clients to the broker-side coordinator and eliminating the classic leader-driven, stop-the-world rebalances. This change simplifies consumers and improves stability and time-to-recover during membership or metadata churn, but it also removes support for client-side partition assignors on standard consumers in favor of pluggable server-side assignors. For teams that previously relied on custom assignment strategies, the good news is that you can still customize behavior by implementing a broker-side assignor via the new server assignor SPI and enabling it based on group-level and client configurations. This talk demystifies how assignment works in the KIP‑848 era and provides a practical, end-to-end guide to writing, testing, and safely rolling out a custom broker-side assignor. We’ll cover the coordinator’s target-assignment model and incremental reconciliation, as well as the constraints and contracts your assignor must respect (stickiness, determinism, payload limits). We’ll share a working example: real code and configs to write and use our very…

[KIP-848](https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol) introduces a new consumer rebalancing protocol to Kafka that differs significantly from its existing one. This session guides attendees through a detailed comparison of the existing and new consumer rebalance protocols and a thorough examination of the migration mechanisms involved in transitioning between them. We offer a comprehensive overview of the fundamentals underlying the classic rebalancing protocols with different assignment strategies, as well as the newly introduced incremental rebalancing protocol. The overview includes the intricacies of group coordination, partition assignment strategies of each protocol. Following the overview, we delve deeply into the mechanisms that drive the migration process between the old and new rebalance protocols. This includes exploration of stop-the-world offline migration methodologies and the more sophisticated online migration techniques that support non-empty group conversions. Detailed case studies illustrate the steps of upgrading and downgrading consumer groups, including handling the intermediate states…

Explore the past achievements, present realities, and future possibilities of Kafka/Kafka Streams with a mix of community leaders, contributors, and Apache Kafka committers. This is a panel so audience interaction is central and encouraged! Bring us your questions, challenge assumptions, and help shape the future of Kafka/Kafka Streams with this session

A new major version of the KafkaConsumer is out, bringing in fundamental changes and improvements, as it’s the first version to fully implement the next generation of the Consumer Group Rebalance Protocol, introduced with KIP-848. It’s all a brand new production-ready feature now! Want to hear about how these major changes materialize in the KafkaConsumer? What’s in? What’s out? What’s different? This talk is for you then! We will cover the core of the new rebalance protocol, its implementation on the Java client, and how it significantly improves and simplifies the whole group consumption experience, addressing its main pain points. We will also share about the revamped KafkaConsumer threading model, shipped alongside the new rebalance protocol client implementation. It all sounds promising, but we do know that upgrades might be scary, right? Whether you’re a Kafka developer, operator, or architect, this talk will equip you with everything you need to confidently adopt KafkaConsumer 4.0 in your client applications. From how the live upgrade and protocol interoperability works, to detailed client changes: configuration changes, API deprecations and additions, improved API…