Lucas Brutschy

Apache Kafka’s scalability and resilience depend not just on its brokers, but crucially on its client ecosystem—most notably, Kafka Streams. Maintaining stability forces Streams applications to redistribute partitioned workloads during infrastructure shifts. These rebalances are notorious for causing operational friction, performance degradation, and unexpected downtime. KIP-1071 (Streams groups) introduces a major evolutionary step towards resolving this friction. Built on the foundation of next-generation consumer groups (KIP-848), Streams groups move task assignment to the broker side and enable fully incremental rebalancing. The result is faster, more stable, and predictable coordination that fundamentally changes how Kafka Streams handles scaling in, scaling out, and failing over. In this talk, we will dive into the core assignment logic, specifically focusing on the performance of the new protocol, its sticky task assignor, and a direct comparison to classic rebalancing. We will also explore a completely new incarnation of the high-availability assignor, which is designed explicitly to allow smooth scaling out for state-heavy applications. You will see how this updated…

Kafka Streams, the native stream processing technology of Kafka, stands out for its unique ability to assign work and propagate metadata among clients without the need for separate coordination infrastructure — requiring only a Kafka broker. Traditionally, this has been accomplished by embedding Streams-specific payloads within generic consumer group RPCs of the Kafka protocol and plugging custom assignment logic into the standard Kafka Consumer. As the Kafka consumer transitions to the next-generation consumer protocol, Streams applications are starting to reveal their true identity when communicating with the broker. In the accepted Kafka Improvement Proposal 1071, new Streams-specific RPCs are being added to the Kafka protocol. These will be used by applications to form groups, exchange metadata, and distribute workloads. A new type of group, the Streams group, will coexist alongside the consumer group, elevating Kafka Streams to a first-class citizen on the Kafka broker. Assignments, member metadata, and group configurations will have a single source of truth, and the assignment logic will run iteratively and centrally as part of the broker’s group coordinator. This deeply…