Neha Pawar

Real-time analytics systems face the dual challenge of serving high-concurrency queries at millisecond latency while keeping up with ever-growing, dynamic datasets. Apache Pinot excels as a low-latency serving layer, but low-latency serving directly from a data lake is a completely different challenge with its own unique complexities. In this talk, we present native Iceberg support in Apache Pinot, enabling the industry’s first low-latency serving directly on data lakes. We cover the evolution of Pinot’s architecture—from tightly coupled storage and compute to tiered cloud storage, and now from Pinot segment format to natively supporting Parquet format. We will walk through key innovations such as the Parquet forward indexes, the ability to use all of Pinot’s indexes directly on top of Parquet files, index pinning, Parquet page-level reads, and chunk prefetching. These optimizations go far beyond metadata-based pruning typically offered by Parquet, enabling highly precise, high-concurrency query execution. Attendees will gain insight into how these design choices unlock interactive, real-time analytics on large-scale lakehouse datasets, bridging the gap between serving and modern…

For real-time analytics, you need systems that can provide ultra low latency (milliseconds) and extremely high throughput (hundreds of thousands of queries per second).

Among GitHub users with at least ten contributions, a mere 6% were women. This is way less than the ratio of women in tech that various research shows at 26%. Given the amount of investment going into and the growth / success of companies based on open source as well as the enormous demand for devel