Roland Fäustlin

Serverless Compute for Notebooks, Jobs, and, Pipelines has reached a turning point. What started as a convenience for ad hoc workloads now runs petabyte-scale production and mission-critical SLAs at some of the largest Databricks customers. This session covers the most recent launches across runtime, performance, and cost for Notebooks, Jobs, and Pipelines, as well as upcoming. Then we share how Deloitte Global moved its global data platform, serving 60,000+ auditors with 20,000+ daily jobs across three geos, from classic compute to serverless: the migration path, the architecture changes, and the results. If you have been waiting for serverless to be ready for your hardest and largest workloads, this is the session to recalibrate.

This is an overview of migrating from Apache Airflow to Lakeflow Jobs for modern data orchestration. It covers key differences, best practices and practical examples of transitioning from traditional Airflow DAGs orchestrating legacy systems to declarative, incremental ETL pipelines with Lakeflow. Attendees will gain actionable tips on how to improve efficiency, scalability and maintainability in their workflows.

Discover how Databricks serverless compute revolutionizes data workflows by eliminating infrastructure management, enabling rapid scaling and optimizing costs for Notebooks, Jobs and Lakeflow Declarative Pipelines. This session will delve into the serverless architecture, highlighting its ability to dynamically allocate resources, reduce idle costs and simplify development cycles. Learn about recent advancements, including cost savings and practical strategies for migration and optimization. Tailored for Data Engineers and Architects, this talk will also explore use cases, features, limitations and future roadmap, empowering you to make informed infrastructure decisions while unlocking the full potential of Databricks’ serverless capabilities.