Dipankar Mazumdar

A common challenge with large-scale Apache Iceberg tables is keeping performance and costs under control as use and volumes grow. Optimizing tables is the solution, but it’s a tough problem to solve.

Apache Iceberg manifests store essential statistics such as min/max values and null counts, but as tables scale, query engines increasingly require richer metadata to make efficient optimization decisions. Puffin was introduced in Iceberg to address this gap by providing a file format for storing advanced statistics, sketches, and secondary indexes that do not naturally fit within manifests. In this session, we will explore the Puffin file format under the hood, including its structure, blob-based design, and snapshot-aware metadata model. We'll examine the two Puffin blob types supported today - Theta Sketches for approximate distinct counts and Deletion Vectors for row-level deletes introduced in Iceberg v3, and discuss how they enable smarter query planning, more efficient delete handling, and metadata-driven optimizations without rewriting data files. We will also look at how Puffin is already being adopted in production, with systems such as AWS Glue and Impala producing and consuming Puffin metadata to power smarter optimizers and table operations. Finally, we will discuss why Puffin is becoming an increasingly important building block in Iceberg's evolution and how it lays…

There is a lot of excitement around the lakehouse architecture today, which unifies two mainstream data architectures - data warehouse & data lakes - promising to do more with less.

Apache Iceberg works great at scale & provides strong transactional guarantees, but production reality still has sharp edges when tables operate under real concurrency, high file counts & aggressive maintenance schedules. This lightning talk distills four recurring Iceberg failure modes observed in real production systems: •Commit conflicts under concurrent writers  •Metadata bloat driven by hig

Apache Iceberg, Hudi & Delta Lake are leading open-source projects providing decoupled storage with advanced transaction & metadata layers, known as table formats, in cloud storage. When data is stored in a distributed file system, these formats offer similar features: a table abstraction over files that includes schema, commit history, partitions & column statistics.

Apache Hudi, Iceberg, and Delta Lake have emerged as leading open-source projects, providing decoupled storage with a powerful set of primitives that offer transaction and metadata layers (commonly referred to as table formats) in cloud storage.

Data Lakes have allowed organizations to store huge amounts of data with cheaper costs to run robust analytical workloads such as BI and Machine Learning. However, data lakes have no data management capabilities (quality, governance, etc.) & can give analysts/data scientists a hard time in making se