Lianet Magrans

The Apache Kafka Clients have come a long way since 4.0, with exciting improvements landing across many areas of the Consumer and Producer. The new rebalance protocol is GA and the Classic one is on its way out; the producer can allocate memory on demand; rebalances are faster and more stable; and a steady stream of smaller KIPs has extended client functionality, sharpened observability, and made applications more resilient. The catch: many of these wins only pay off if you know they exist, turn them on, and migrate off the old defaults safely. Others need some configuration tuning to best fit your application. This session is the map: what's improved, why it matters for your applications, and how to adopt it without breaking things and get the most out of it. We'll tour what's landed across the 4.x releases and what's coming, with detail on how each change works, the benefits they bring, and how to adopt it today. On the consumer side, we'll cover the Classic protocol's deprecation timeline (KIP-1274) and how to migrate safely, assignment epochs and offload (KIP-1251, KIP-1263), the resilience improvements around DNS resolution (KIP-909), new regex subscriptions, and more. On the…

Apache Kafka 4.0 makes the new consumer group rebalance protocol (KIP‑848) generally available, shifting assignment computation from clients to the broker-side coordinator and eliminating the classic leader-driven, stop-the-world rebalances. This change simplifies consumers and improves stability and time-to-recover during membership or metadata churn, but it also removes support for client-side partition assignors on standard consumers in favor of pluggable server-side assignors. For teams that previously relied on custom assignment strategies, the good news is that you can still customize behavior by implementing a broker-side assignor via the new server assignor SPI and enabling it based on group-level and client configurations. This talk demystifies how assignment works in the KIP‑848 era and provides a practical, end-to-end guide to writing, testing, and safely rolling out a custom broker-side assignor. We’ll cover the coordinator’s target-assignment model and incremental reconciliation, as well as the constraints and contracts your assignor must respect (stickiness, determinism, payload limits). We’ll share a working example: real code and configs to write and use our very…

A new major version of the KafkaConsumer is out, bringing in fundamental changes and improvements, as it’s the first version to fully implement the next generation of the Consumer Group Rebalance Protocol, introduced with KIP-848. It’s all a brand new production-ready feature now! Want to hear about how these major changes materialize in the KafkaConsumer? What’s in? What’s out? What’s different? This talk is for you then! We will cover the core of the new rebalance protocol, its implementation on the Java client, and how it significantly improves and simplifies the whole group consumption experience, addressing its main pain points. We will also share about the revamped KafkaConsumer threading model, shipped alongside the new rebalance protocol client implementation. It all sounds promising, but we do know that upgrades might be scary, right? Whether you’re a Kafka developer, operator, or architect, this talk will equip you with everything you need to confidently adopt KafkaConsumer 4.0 in your client applications. From how the live upgrade and protocol interoperability works, to detailed client changes: configuration changes, API deprecations and additions, improved API…