Saad Ansari

Modern data platforms need more than a scheduler. They need orchestration that understands data, integrates with the platform, and scales with both engineering and AI workloads. Join this session to see how Lakeflow Jobs helps teams orchestrate pipelines, notebooks, dbt projects, SQL, and external components, natively on the Databricks Platform. We will show how Lakeflow Jobs goes beyond basic scheduling with data-aware orchestration that understands the assets and systems it is coordinating. You will see how teams can trigger workflows based on data changes, build modular workflows in Python, orchestrate external systems alongside Databricks workloads, and use Declarative Automation Bundles (DABs) for a more reliable CI/CD path. We will also cover how Serverless Jobs and observability features such as lineage and system tables help teams reduce operational overhead and better understand what is happening across their workflows. Finally, we will preview how agentic authoring and diagnostics can speed up building, troubleshooting, and improving production jobs, and how you can embed agents directly into your workflows. Whether you are modernizing from legacy schedulers or building…

Still running orchestration outside Databricks? That often means more tools to manage, less context about your data, and more overhead connecting pipelines, schedules, and dependencies across systems. Join this session to learn how to migrate to Lakeflow Jobs, Databricks’ native orchestrator for data and AI workloads. We will cover practical migration patterns, the tooling available to accelerate the move, and the tips, techniques, and best practices that help teams reduce risk and avoid common pitfalls. You will learn: - Why teams are consolidating orchestration onto Databricks - How common migration patterns from legacy orchestrators map to Lakeflow Jobs - What tooling, tips, and techniques can speed up migration and simplify rollout - The best practices that help teams move faster with less operational overhead - What real customer migrations look like in practice, including key challenges, tradeoffs, and outcomes

This session is repeated.Managing data and AI workloads in Databricks can be complex. Databricks Asset Bundles (DABs) simplify this by enabling declarative, Git-driven deployment workflows for notebooks, jobs, Lakeflow Declarative Pipelines, dashboards, ML models and more.Join the DABs Team for a Deep Dive and learn about:The Basics: Understanding Databricks asset bundlesDeclare, define and deploy assets, follow best practices, use templates and manage dependenciesCI/CD & Governance: Automate deployments with GitHub Actions/Azure DevOps, manage Dev vs. Prod differences, and ensure reproducibilityWhat’s new and what's coming up! AI/BI Dashboard support, Databricks Apps support, a Pythonic interface and workspace-based deploymentIf you're a data engineer, ML practitioner or platform architect, this talk will provide practical insights to improve reliability, efficiency and compliance in your Databricks workflows.

Monitoring data pipelines is key to reliability at scale. In this session, we’ll dive into the observability experience in Lakeflow, Databricks’ unified DE solution — from intuitive UI monitoring to advanced event analysis, cost observability and custom dashboards. We’ll walk through the revamped UX for Lakeflow observability, showing how to: - Monitor runs and task states, dependencies and retry behavior in the UI - Set up alerts for job and pipeline outcomes + failures - Use pipeline and job system tables for historical insights - Explore run events and event logs for root cause analysis - Analyze metadata to understand and optimize pipeline spend - How to build custom dashboards using system tables to track performance data quality, freshness, SLAs and failure trends, and drive automated alerting based on real-time signals This session will help you unlock full visibility into your data workflows.

This session is repeated. Curious about orchestrating data pipelines on Databricks? Join us for an introduction to Lakeflow Jobs (formerly Databricks Workflows) — an easy-to-use orchestration service built into the Databricks Data Intelligence Platform. Lakeflow Jobs simplifies automating your data and AI workflows, from ETL pipelines to machine learning model training. In this beginner-friendly session, you'll learn how to: - Build and manage pipelines using a visual approach - Monitor workflows and rerun failures with repair runs - Automate tasks like publishing dashboards or ingesting data using Lakeflow Connect - Add smart triggers that respond to new files or table updates - Use built-in loops and conditions to reduce manual work and make workflows more dynamic We’ll walk through common use cases, share demos and offer tips to help you get started quickly. If you're new to orchestration or just getting started with Databricks, this session is for you.