Daniel Becker

Storing extensive metadata is one of the key features of the Apache Iceberg table format, helping query engines plan and execute queries efficiently. As Iceberg provides an API to query this metadata, it can be presented in query engines as a set of virtual tables, which can be queried with SQL, including filtering, aggregation and joins with other metadata or even regular tables.

Reading file formats efficiently is a crucial part of big data systems - in selective scans data is often only big before hitting the first filter and becomes manageable during the rest of the processing. The talk describes this early stage of query execution in Apache Impala, from reading the bytes of Parquet files on the filesystem to applying predicates and runtime filters on individual rows.