Shivam Yadav

Windowed aggregation on streams is a powerful concept for analytical data processing, especially in cases where waiting hours or even minutes for data to be available is inconceivable. While most people think of aggregations as an analytical requirement, they also help trim down data to size and can be critical to scaling systems without leading to ballooning costs. We had a similar use case in ShareChat where we had hundreds of thousands of counter increments (updates)/sec for everything from the number of views on a post to revenue numbers. Our databases could not keep up with the write volume, and there were frequent hot-spotting issues. Furthermore, the data was often inconsistent due to multiple writers, and taking locks further added to our misery. To solve this issue, we used Kafka and Kafka Streams to build an aggregation framework that can handle hundreds of thousands of counter increments per second. Streaming aggregation helped us batch updates and helped reduce the write throughput in our DBs. Further, it helped solve our hot-spotting issues and eliminated the need to take locks. This talk discusses the challenges of building the platform, managing an in-house Kafka…