Theresa Hammer

It is 3 AM. A dashboard is stale, an SLA slipped, and everyone is guessing which job, pipeline, or dependency broke first. That is when surface-level monitoring stops being useful. You need observability that shows the health of your workflows and helps you find the cause fast. In this session, we will show how Lakeflow Observability helps data engineering teams move from reactive monitoring to real operational health. You will see how to monitor runs, tasks, dependencies, alerts, and health metrics across jobs and pipelines, and how Databricks system tables provide the historical depth needed for root-cause analysis, performance analysis, cost tracking, and reliability reporting at scale. We will also cover practical patterns for tracking streaming backlog, freshness, SLA risk, and failure trends, and show how Genie Code can help teams troubleshoot faster using natural language. Your 3 AM self will thank you.

Monitoring data pipelines is key to reliability at scale. In this session, we’ll dive into the observability experience in Lakeflow, Databricks’ unified DE solution — from intuitive UI monitoring to advanced event analysis, cost observability and custom dashboards. We’ll walk through the revamped UX for Lakeflow observability, showing how to: - Monitor runs and task states, dependencies and retry behavior in the UI - Set up alerts for job and pipeline outcomes + failures - Use pipeline and job system tables for historical insights - Explore run events and event logs for root cause analysis - Analyze metadata to understand and optimize pipeline spend - How to build custom dashboards using system tables to track performance data quality, freshness, SLAs and failure trends, and drive automated alerting based on real-time signals This session will help you unlock full visibility into your data workflows.