Paolo Patierno

AI and Machine Learning systems only create value through the data that feeds them. As environments change continuously, static datasets and offline pipelines are no longer sufficient. Modern AI/ML requires real-time data streams to remain accurate, responsive, and trustworthy. This session explores how Apache Kafka acts as the central event backbone for AI/ML, connecting data producers, feature pipelines, training processes, inference services, and feedback loops. Kafka enables event-driven AI/ML systems that evolve alongside the data they observe. The talk introduces a set of AI/ML streaming patterns that show how Machine Learning workloads can be built around continuous ingestion rather than periodic batch jobs. It covers patterns for streaming data ingestion and preprocessing, live features extraction, and training models directly from event streams, including near-real-time retraining and online learning approaches. It also explores real-time inference and feedback-loop patterns, where predictions and outcomes are streamed back into Kafka for monitoring and model drift detection, as well as LLM-oriented patterns where Kafka provides continuously refreshed context for…

From version 4.0 of Kafka, clusters will not support ZooKeeper anymore. This is a change that has been in the works for a while, and Kafka now has its own consensus protocol named KRaft. If you want to create a new Kafka cluster, using KRaft is pretty straightforward, but what if you already have existing Kafka clusters? Since Kafka 3.6, you can migrate ZooKeeper based clusters to KRaft. The process has a few intricacies about it and even more so if you want to automate it, rather than doing it manually for each cluster. In this session we will take you along for the ride of migrating a ZooKeeper based Kafka cluster, to a KRaft one. We’ll share the pitfalls we discovered while implementing an automated process in the open source Kafka operator Strimzi to handle this operation. It was certainly an adventure, so come along to see how you can navigate this migration as smoothly as possible, by avoiding a lot of error-prone manual steps on your production clusters.

Is your organization scaling Apache Kafka on Kubernetes with the need for efficient resource management and automated balancing? Strimzi, an open-source solution for running Kafka on Kubernetes, integrates with Cruise Control to offer advanced workload rebalancing, monitoring, and optimization....