Adam Souquieres

Most Kafka Streams tutorials end where production begins. This interactive, detailed-design, whiteboarding session bridges that gap by designing a real-world payment authorization system: collaboratively, honestly, and without hiding the messy parts. We will start optimistically with Kafka Streams DSL approaches: elegant joins, straightforward windowing: and systematically discover their breaking points. As Apache Kafka Streams DSL solutions fail under real-world constraints, we'll architect production-grade alternatives using the Processor API, custom state store designs, and advanced patterns for handling late data, implementing timeouts, managing state lifecycle, and ensuring exactly-once semantics. We'll visually map out state management strategies, “watermarking” approaches, and the trade-offs between different join patterns when dealing with temporal uncertainty. Here’s the challenge (Inspired from one of my past engagements) ! Two asynchronous input streams: shiny new payment authorization requests arriving in real-time account state updates that arrive delayed, because that is how the legacy integration actually works. The business requirement is straightforward: decide…

In this session, we will delve into the practical boundaries of the Kafka Streams DSL and showcase why the Processor API stands out as the ultimate tool for addressing complex streaming scenarios. Using Michelin’s tire delivery process as a guiding example, we will illustrate how the different join types can be implemented with the DSL and where its limitations begin to emerge. The challenges of joining events from multiple topics, whether driven by event-based or time-based logic, and achieving fine-grained control over state stores led us to embrace the Processor API. While the DSL is convenient and expressive for many use cases, the Processor API consistently proves to be the most powerful solution for real-world applications requiring precision and flexibility. Whether you’re an architect, developer, or Kafka enthusiast, this session will equip you with actionable insights into designing custom state stores, optimizing for low latency, and implementing adaptable join logic to meet evolving business needs. Rather than advocating for abandoning the DSL entirely, the session highlights the importance of recognizing its limitations and understanding why the Processor API is often…