Sébastien Viale

Kafka Streams automatically creates internal topics —such as changelog and repartition topics— based on the structure of a topology. While convenient, this mechanism can be fragile : small topology changes like reordering processors or inserting a new step may alter internal topic names, breaking state reuse and causing unexpected reprocessing in production. KIP‑1111 introduces a solution : it allows Kafka Streams applications to enforce explicit naming for internal resources and fail fast at startup when auto-generated names are still present. In a short, demo-driven session 🎬, I will: - 🔍 Show how topology-derived internal topic naming can unintentionally break stateful applications - ⚙️ Demonstrate how to enable and use KIP‑1111 in a real Kafka Streams application - 📈 Explain how explicit internal topic names simplify upgrades and monitoring Attendees will leave with a clear understanding of why explicit internal naming matters, how KIP‑1111 catches problems early, and when to adopt this approach in production Kafka Streams applications.

In this session, we will delve into the practical boundaries of the Kafka Streams DSL and showcase why the Processor API stands out as the ultimate tool for addressing complex streaming scenarios. Using Michelin’s tire delivery process as a guiding example, we will illustrate how the different join types can be implemented with the DSL and where its limitations begin to emerge. The challenges of joining events from multiple topics, whether driven by event-based or time-based logic, and achieving fine-grained control over state stores led us to embrace the Processor API. While the DSL is convenient and expressive for many use cases, the Processor API consistently proves to be the most powerful solution for real-world applications requiring precision and flexibility. Whether you’re an architect, developer, or Kafka enthusiast, this session will equip you with actionable insights into designing custom state stores, optimizing for low latency, and implementing adaptable join logic to meet evolving business needs. Rather than advocating for abandoning the DSL entirely, the session highlights the importance of recognizing its limitations and understanding why the Processor API is often…

A major concern when starting with Kafka Streams is how to handle (un)expected errors. Generally, you want to track these errors, identify the records that caused the failures, and possibly reprocess them. To achieve this, you often need to implement a custom try-catch mechanism and send these errors to a dedicated topic. Does this challenge sound familiar? Welcome aboard! At Michelin, we face it too. For our own needs, we embedded this kind of error-handling mechanism in a home-made solution, but this solution has its limitations. Thus, we proposed two Kafka Improvement Proposals to enhance the Kafka Streams exception handling experience. KIP-1033 introduces a new processing exception handler, complementing existing deserialization and production exception handlers. Now, any exceptions that occur during processing are caught and transmitted to the handler, allowing you to define your error-handling logic. Complementary to this, KIP-1034 adds native support for routing failed records to a dead-letter queue topic of your choice. By the end of this talk, you will walk away with the latest updates these KIPs bring, helping you build more robust Kafka Streams applications against…