Chinmay Soman

Modern applications generate an endless stream of events, but users expect to analyze both what happened seconds ago and what happened years ago—through a single, low-latency interface.

Apache Iceberg has become the open standard for modern data platforms, yet most adoptions approach migration as a forklift. Governance improves, storage is standardized, and BI workloads run reliably; but the most demanding analytics, the ones closest to revenue and customer experience, are rarely considered in scope for Iceberg.

Real-time analytics systems face the dual challenge of serving high-concurrency queries at millisecond latency while keeping up with ever-growing, dynamic datasets. Apache Pinot excels as a low-latency serving layer, but low-latency serving directly from a data lake is a completely different challenge with its own unique complexities. In this talk, we present native Iceberg support in Apache Pinot, enabling the industry’s first low-latency serving directly on data lakes. We cover the evolution of Pinot’s architecture—from tightly coupled storage and compute to tiered cloud storage, and now from Pinot segment format to natively supporting Parquet format. We will walk through key innovations such as the Parquet forward indexes, the ability to use all of Pinot’s indexes directly on top of Parquet files, index pinning, Parquet page-level reads, and chunk prefetching. These optimizations go far beyond metadata-based pruning typically offered by Parquet, enabling highly precise, high-concurrency query execution. Attendees will gain insight into how these design choices unlock interactive, real-time analytics on large-scale lakehouse datasets, bridging the gap between serving and modern…

In this talk, we present how Apache Flink powers the world of OnLine Analytical Processing (OLAP) at Uber. OLAP systems such as Apache Pinot and AresDB are good at consuming structured data and making it available for slicing and dicing.