Nandini Singhal

Operating Kafka in production is hard. Operating thousands of Kafka clusters globally—without customer-visible incidents—is an entirely different problem. At Datadog, Kafka is the backbone of our real-time data ingestion and streaming platform, processing petabytes of data every day across thousands of clusters and tens of thousands of brokers. At this scale, failures are rarely loud or localized. Instead, they surface as subtle latency shifts, uneven consumer lag, stalled rebalances, or slow partitions that quietly degrade customer experience if not caught early. The hardest part is not detecting symptoms—it’s identifying root causes fast enough to prevent impact. Standard Kafka monitoring (JMX metrics, broker health, consumer lag) breaks down when incidents span multiple clusters, teams, and regions. This talk explores how Datadog runs a massive Kafka fleet in production while minimizing incidents and customer impact, and the observability practices that make this possible. Through real production scenarios, we’ll show how we correlate signals across brokers, consumers, storage layers, and infrastructure to understand why something is wrong—not just that it is. We’ll dive into…