Songting Wang

Enterprises ingest and analyze massive volumes of streaming data in Flink to analyze and derive real-time insights. For instance, financial institutions process credit card transactions to monitor risk and detect fraud, while observability platforms ingest telemetry data to monitor application performance. While traditional Flink analytics pipelines have served us well so far, the rising scale and complexity of data are causing an untenable increase in cloud costs as well as increased latency that prohibits real-time decision-making. Thus, there is a need to rethink the design of aggregate analytics pipelines. Sketching algorithms provide an effective alternative to traditional aggregation by leveraging compact, probabilistic data structures to provide highly accurate and low-cost analytics. These algorithms are designed to estimate various aggregates like distinct counts, frequency, and quantiles, and are amenable to massively parallel processing. Sketches are backed by extensive research and estimate aggregates, with mathematically bounded errors. Unfortunately, implementations of these algorithms have not made it into the Flink ecosystem, preventing the Flink community from…