Tyler Croy

Managing multimodal inference is one of the defining challenges facing AI teams. LLMs now consider documents, images, and videos, producing higher quality results and reducing time wasted with data prep. The lakehouse architecture solves part of the problem but many teams are struggling to incorporate an increasing variety of data into their workflow. Rather than ingesting, re-encoding, or modifying massive troves of existing data, multimodal teams are looking to work with original artifacts. This talk introduces Virtual Delta Tables and the associated open source code designed for multimodal inference. Virtual Delta Tables incorporate original data from different sources into a singular logical structure for ease of use by data scientists and ML engineers. By offering a Delta Lake interface the entire ecosystem from Databricks to DuckDB can be used in multimodal work. We'll highlight just how far you can push the lakehouse architecture in the rapidly changing AI data landscape!

Delta Lake is a fantastic technology for quickly querying massive data sets, but first you need those massive data sets! In this session we will dive into the cloud-native architecture Scribd has adopted to ingest data from AWS Aurora, SQS, Kinesis Data Firehose and more. By using off-the-shelf open source tools like kafka-delta-ingest, oxbow and Airbyte, Scribd has redefined its ingestion architecture to be more event-driven, reliable, and most importantly: cheaper. No jobs needed! Attendees will learn how to use third-party tools in concert with a Databricks and Unity Catalog environment to provide a highly efficient and available data platform. This architecture will be presented in the context of AWS but can be adapted for Azure, Google Cloud Platform or even on-premise environments.

Join us for an in-depth Ask Me Anything (AMA) on how Rust is revolutionizing Lakehouse formats like Delta Lake and Apache Iceberg through projects like delta-rs and iceberg-rs! Discover how Rust’s memory safety, zero-cost abstractions and fearless concurrency unlock faster development and higher-performance data operations. Whether you’re a data engineer, Rustacean or Lakehouse enthusiast, bring your questions on how Rust is shaping the future of open table formats!