Bruno Cadonna

Kafka Streams, the native stream processing technology of Kafka, stands out for its unique ability to assign work and propagate metadata among clients without the need for separate coordination infrastructure — requiring only a Kafka broker. Traditionally, this has been accomplished by embedding Streams-specific payloads within generic consumer group RPCs of the Kafka protocol and plugging custom assignment logic into the standard Kafka Consumer. As the Kafka consumer transitions to the next-generation consumer protocol, Streams applications are starting to reveal their true identity when communicating with the broker. In the accepted Kafka Improvement Proposal 1071, new Streams-specific RPCs are being added to the Kafka protocol. These will be used by applications to form groups, exchange metadata, and distribute workloads. A new type of group, the Streams group, will coexist alongside the consumer group, elevating Kafka Streams to a first-class citizen on the Kafka broker. Assignments, member metadata, and group configurations will have a single source of truth, and the assignment logic will run iteratively and centrally as part of the broker’s group coordinator. This deeply…