Sophie Blee-Goldman

Since the very dawn of Kafka Streams it has been haunted by the possibility of high latencies in a topology. Custom logic with heavy processing, RPCs, and remote state stores were more or less considered out of the question, and those who bravely tried anyways ran into endless problems. At the same time, even simple applications were getting in trouble as companies grew and so did their workloads, yet apps could only scale up to the partition count. Are slow and/or heavy applications just not a good fit for Kafka Streams? The answer of course is no! With Kafka Streams anything is possible. In this talk we'll introduce Responsive's Async Processor, a lightweight wrapper that turns your slowest applications into medal winners. With just a few lines of code you can easily convert any Kafka Streams app to async. By injecting an async thread pool to hand over the actual processing work, records can be executed in parallel, even within a partition all without sacrificing any of the usual correctness guarantees like same-key ordering or exactly-one semantics. This feature is production-ready and available now through the Responsive SDK, but we'll end by discussing our vision for native…