Bipul Karnani

In this session, Dream11 engineering team will share the secret sauce and the innovation around Apache Kafka consumers, processing tens of millions of events using a re-engineered Kafka consumer library. Dream11 is one of the largest fantasy sports platforms in the world, handling peak user concurrency of over 15 million during IPL 2024, with edge RPM surpassing 300 million. The business operates under highly time-sensitive conditions, experiencing hockey-stick traffic surges just before the start of matches. To ensure real-time updates for game users, the Dream11 platform heavily relies on Apache Kafka in the critical pipelines of end user services. As the scale grew, the legacy Kafka consumer (simple, high-level) began facing challenges such as delays and data loss, severely impacting user trust. To address these issues, the Dream11 engineering team innovated and developed a low-level Kafka consumer. In this consumer, polling is decoupled from processing and executing both in parallel which fixed our frequent rebalancing problem. For processing the messages we created dedicated worker pool which improved our speed significantly. We disabled the auto-commit and commits were done…