Vivek Chandela

In August 2023, WarpStream introduced itself as a Kafka-compatible, S3-native streaming solution offering powerful features such as a BYOC-native approach, decoupling of storage and compute as well as data and metadata, offset-preserving replication, and direct-to-S3 writes. It shines in a specific niche—logging, observability, and data lake feeding—where a slight increase in latency is a fair trade-off for substantial cloud cost savings and simplified operations. In this session, we'll take a look into ShareChat's journey of migrating our logging systems from managed Kafka-compatible solutions to WarpStream. At ShareChat, logging suffered from 2 issues: highly unpredictable workloads and high inter-zone fees for data replication across brokers. Logging volume could spike up to 5 times the normal rate for brief periods before returning to baseline. We had to over-provision our Kafka clusters to prevent costly rebalancing and scaling issues, resulting in unnecessary expenses. WarpStream offers a solution with its stateless, autoscaling agents—eliminating the need to manage local disks or rebalance brokers. Moreover, by leveraging S3 for replication, WarpStream allows us to…