Anton Okolnychyi

Spark’s DataSource V2 integration has taken a major step forward, beginning with the addition of a procedure catalog and row identifier support to enable richer table management and row-level operations across modern data sources. Building on this foundation, recent updates improve MERGE INTO with safer schema evolution and enhance partition filtering for more efficient query planning and execution. DML summaries now provide clearer visibility into write behavior, while key cache fixes resolve long-standing correctness issues in Spark execution. DataSource V2 has also been extended with first-class SQL features such as table constraints, complex default values, and generated columns, laying the groundwork for more advanced table semantics. We’ll also look ahead to deeper Change Data Feed (CDF) integration in Spark to support robust incremental processing, highlighting what’s available today and what’s coming next as Spark continues to close the gap with traditional warehouse systems.

Iceberg’s Spark integration on DataSource V2 has taken a major step forward, beginning with the addition of a procedure catalog and row identifier support to enable more complete table management and row-level operations. Building on this foundation, ongoing updates improve MERGE INTO with safer schema evolution and enhance partition filtering for more efficient query planning and execution.

DSv2, Spark's next-generation Catalog API, is gaining traction among data source developers. It shifts complexity to Apache Spark™, improves connector reliability and unlocks new functionality such as catalog federation, MERGE operations, storage-partitioned joins, aggregate pushdown, stored procedures and more. This session covers the design of DSv2, current strengths and gaps and its evolving roadmap. It's intended for Spark users and developers working with data sources, whether custom-built or off-the-shelf.

A critical aspect of any table format is the rapid identification of files relevant for a query irrespective of the underlying data volume. The focus of this presentation is on the job planning process in Apache Iceberg, highlighting its efficiency and ability to scale to tens of millions of files.