Scott Sandre

With the introduction of Catalog-Managed Tables, Delta Lake has unlocked unified governance, enforceable constraints, and a path towards multi-table updates.As a result, Delta connectors must adapt to properly load, read, and write Delta Tables, as the Catalog is now the authoritative broker of commits, not the filesystem.In this talk, learn how we designed Catalog Managed Tables support in Delta Kernel—the one right Delta abstraction to rule them all. We’ve built a unified catalog-agnostic API against which connectors can develop, enabled deep and powerful engine integrations for DuckDB, Delta-Spark, and Delta-Flink, all while keeping the complexity of catalog support out of Kernel.

Open table formats had a breakthrough year—and now the focus shifts to how to use them effectively. In this session, we’ll break down what’s new across Delta Lake and Apache Iceberg, including an early look at Delta 5.0 and the Iceberg v4 Adaptive Metadata Tree, which brings the two formats closer together at the metadata layer for high-performance, multi-engine workloads. More importantly, we’ll share best practices for working across formats today. Learn how teams are using the Unified Delta Kernel (now GA, written in Rust) across Spark, Flink, and DuckDB, and how Catalog-Managed Commits enable safer multi-engine writes, stronger governance, and multi-table transactions. We’ll also cover Auto CDF for building efficient incremental pipelines. Finally, we’ll highlight innovations for AI workloads, including the FILE data type and VARIANT shredding. Walk away with practical guidance to build, optimize, and future-proof your open data architecture.

Delta Lake is redesigning its Spark connector through the combination of three key technologies: First, we're updating our Spark APIs to DSv2 to achieve deeper catalog integration and improved integration with the Spark optimizer. Second, we're fully integrating on top of Delta Kernel to take advantage of its simplified abstraction of Delta protocol complexities, accelerating feature adoption and improving maintainability. Third, we are transforming Delta to become a catalog-aware lakehouse format with Catalog Commits, enabling more efficient metadata management, governance and query performance. Join us to explore how we're advancing Delta Lake's architecture, pushing the boundaries of metadata management and creating a more intelligent, performant data lakehouse platform.

Apache Flink and Delta Lake together allow you to build the foundation for your data lakehouses by ensuring the reliability of your concurrent streams from processing to the underlying cloud object-store.