Andrew Schofield

When Apache Kafka introduced share groups, the first question from every architect was: "What's the overhead?" This talk answers that question with data. The talk presents a comprehensive performance analysis of share groups across throughput, latency, and resource utilization under diverse workload profiles. We benchmark share groups against consumer groups in head-to-head comparisons — from high-throughput streaming workloads to fan-out patterns, bursty task queues, and heterogeneous worker pools with variable processing times. Share groups add record-level state tracking, acquisition locks, and per-message acknowledgment — but how much does that actually cost, and when does it pay for itself? We show how performance shifts as you scale consumers beyond partition count, introduce out-of-order acknowledgements, vary message sizes, and mix fast and slow consumers on the same topic. The results reveal where share groups match consumer group performance, where they pull ahead, and the effect of application design and configuration. You'll leave with a mental model of share group performance characteristics and practical information for designing your applications and tuning your…

Confluent's non-Java Kafka clients handle over 350 billion connections a month to Confluent Cloud. They're built on librdkafka, a C library that has served the community well for over a decade. But every new KIP follows the same painful path: original implementation in Java, wait months for someone to reimplement it in C, then wrap it for Python, .NET, JavaScript and Go. Some features never arrive at all. Over 40% of Kafka users are stuck in this permanent feature gap. And users encounter subtle behavioral and configuration differences between the Java and non-Java clients which just causes confusion. What if we stopped reimplementing and started translating? What if AI-assisted productivity could enable us to attempt something we would have never attempted previously? What if we could leverage AI tools to take the Java reference implementation and produce a working Rust client, with bindings for all five languages, within weeks of a Java release? That's what we built. This talk is a deep dive into using AI to write production-grade systems code -- not prototypes, not demos -- but a Kafka client that has to handle the same scale and correctness bar as the library it replaces. -…

The introduction of Kafka Share Groups fundamentally re-architects Kafka to decouple consumer scaling from partition count, enabling queue-like consumption over standard topics. This talk focuses on the essential protocol enhancements introduced in Apache Kafka 4.2 that bridge the gap to queue semantics. Specifically, will detail, KIP-1206: Record Limit and Batch Optimized Behaviour, which improves the records delivery mechanism to enable work-queue-like workloads; KIP-1222: Records Lease Renew, which provides a mechanism for applications to process records which takes longer processing time; KIP-1226: Share Lag Computation, which enables the auto scaler to monitor the share lag for the group to manage horizontal scaling. Attendees will gain a clear, actionable understanding of the resiliency associated with long-running tasks, workload optimization and operational scaling associated with the share groups.

Queues for Kafka introduces a new paradigm for consuming data from Apache Kafka, along with the new Share Consumer API. If you’ve ever struggled with building message-queuing applications on top of Kafka, struggle no more. The Share Consumer API in Apache Kafka 4.2 makes this easy. The Share Consumer API still consumes data from Kafka topics, but if you think about how it works, it behaves much more like a message queue. Learn all about how partitions are shared by the consumers, taking precise control over record fetching, the ways of acknowledging record delivery properly, error handling strategies, and even how best to deal with records which take a very long time to process. If you’ve wondered how to read records from Kafka and process them with a group of workers using generative AI without worrying about head-of-line blocking, this is the talk for you.

Event streaming is great but sometimes it’s easier to use a queue, especially when parallel consumption is more important than ordering. If you're already using Apache Kafka for streaming, wouldn't it be great if you could also use it as a message queue when that's more convenient?

Let’s be honest: who wants to have more than one client to connect to a data system? Now consider Apache Kafka. It ships with four different Java clients: producer, consumer, admin, and streams. Want to create a topic in a producer application? Use the admin client and the producer client. Want to produce and consume? Either use the producer and the consumer, or use Kafka Streams. So how did we get here? And more importantly: how can we simplify it? Are incremental improvements enough? In this talk, we’ll propose a radical approach: a single unified Java client built from scratch for producing, consuming, processing, and administration tasks. We take you on a brainstorming session about what we can and cannot do, and what we want to achieve. How can we make simple things easy and difficult things possible? What does a modern Java API look like, using the standard library, a reasonable threading model, lambdas, and futures for async calls? We think it's high time that we take another look at the Java clients and build a client ready for the next decade. Come and join the conversation about the future of Kafka clients.

Event streaming is great but sometimes it’s easier to use a queue, especially when parallel consumption is more important than ordering. Wouldn't it be great if you had the option of consuming your data in Apache Kafka just like a message queue? For workloads where each message is an independent work item, you’d really like to be able to run as many consumers as you need, cooperating to handle the load, and to acknowledge messages one at a time as the work is completed. You might even want to be able to retry specific messages. This is much easier to achieve using a queue rather than a topic with a consumer group. KIP-932 brings queuing semantics to Apache Kafka. It introduces the concept of share groups. Share groups let your applications consume data off regular Kafka topics with per-message acknowledgement and without worrying about balancing the number of partitions and consumers. With this KIP, you can bring your queuing workloads to Apache Kafka. Come and hear about this innovative new feature starting with Early Access in Apache Kafka 4.0, and then Preview in Apache Kafka 4.1.

Let’s be honest: who wants to have more than one client to connect to a data system? Now consider Apache Kafka. It ships with four different Java clients: producer, consumer, admin, and streams. Want to create a topic in a producer application? Use the admin client and the producer client. Want to produce and consume? Either use the producer and the consumer, or use Kafka Streams. So how did we get here? And more importantly: how can we simplify it? Are incremental improvements enough? In this talk, we’ll propose a radical approach: a single unified Java client built from scratch for producing, consuming, processing, and administration tasks. We take you on a brainstorming session about what we can and cannot do, and what we want to achieve. How can we make simple things easy and difficult things possible? What does a modern Java API look like, using the standard library, a reasonable threading model, lambdas, and futures for async calls? We think it's high time that we take another look at the Java clients and build a client ready for the next decade. Come and join the conversation about the future of Kafka clients.

Event streaming is great but sometimes it’s easier to use a queue, especially when parallel consumption is more important than ordering. Wouldn't it be great if you had the option of consuming your data in Apache Kafka just like a message queue? For workloads where each message is an independent work item, you’d really like to be able to run as many consumers as you need, cooperating to handle the load, and to acknowledge messages one at a time as the work is completed. You might even want to be able to retry specific messages. This is much easier to achieve using a queue rather than a topic with a consumer group. KIP-932 brings queuing semantics to Apache Kafka. It introduces the concept of share groups. Share groups let your applications consume data off regular Kafka topics with per-message acknowledgement and without worrying about balancing the number of partitions and consumers. With this KIP, you can bring your queuing workloads to Apache Kafka. Come and hear about this innovative new feature being added to Apache Kafka 4.0.