Sam Barker

Every platform team has dreamt of splitting an Apache Kafka cluster, but the reality is far messier. Application teams think in topics and clusters, but Kafka forces them to care about broker topology. We can't fix that, but this talk is about giving platform teams the tools required to make that dream come true. Most proxies operate on Kafka at Layer 4 — they forward connections, not messages. Kroxylicious works at Layer 7: it understands individual Kafka protocol messages and controls the broker topology the client sees. Multiple physical clusters can appear as one, or one cluster can appear as many — the client's view is whatever the platform team decides it should be. The foundation for this is Kroxylicious's new routing API — a plugin interface that lets you build custom routing logic as first-class extensions. Router authors decide where messages go; the proxy handles making the broker topology look coherent to the client. This enables a family of patterns: Union Clusters — topics from multiple physical clusters presented as one; Union Topics — a single topic sharded across clusters; and Cluster Aliasing — presenting one cluster under the identity of another. It also lets…

As Apache Kafka usage continues to grow, it gets deployed in increasingly sensitive and regulated environments. At the same time, data engineering teams have more and more requirements to satisfy the needs of businesses to support AIs and provide real time business intelligence. Unfortunately, for historical or design reasons, Apache Kafka is not able to provide all the features everybody needs. One solution gaining traction over the last couple of years is to proxy Apache Kafka. This session introduces Kroxylicious, an open source Kafka protocol aware transparent proxy (part of the Commonhaus foundation). Kroxylicious offers developers a standardised Filter API to allow them to customize the messages passing through the proxy as well as a plug-in based extension mechanism to allow them to interact with remote resources, such as Key Management Systems or Schema registries. All this is completely invisible to clients and clusters and does not require updating them. Out of the box Kroxylicious provides a customizable record encryption to ensure data at rest is safe even if you use a cloud provider. It also integrates with a schema registry so you can ensure that records sent to…