Lennart Kats

Genie Code is an autonomous AI partner for data work, with depth across the data engineering lifecycle. We show how builders use it to iterate on Spark Declarative Pipelines and Lakeflow Jobs, accelerating authoring. It knows your data, grounded in Unity Catalog metadata, semantics, and governance policies. This grounding helps Genie Code take accurate and context-aware actions. Beyond authoring, Genie Code keeps workflows healthy, proactively monitoring and maintaining data and AI in production. We discuss how it autonomously detects, diagnoses, and proposes fixes the moment failures happen, with every agent running sandboxed, so production stays untouched.

Modern data engineering teams need more than deployment automation. They need projects that are source-controlled, governed, and reviewable, so both humans and AI agents can safely build, deploy, and operate them at scale. Declarative Automation Bundles (DABs) provide that foundation. By expressing your entire Databricks project as source code, DABs not only enable scalable processes for human developers, but provide the exact structured interface AI agents need to interact with and manage your workflows. In this session, we will show how to use DABs in practice, including best practices and the latest authoring workflows across Genie Code, local IDEs, and the Workspace UI. We will also introduce Deployment History, a new service to track exactly who deployed what, when, and with what result. Building on this foundation, we will share a first look at our Genie Code powered product that monitors jobs, pipelines, tables, and other data assets for failures and optimization opportunities. It explains the root cause, proposes a fix, and validates it in a sandboxed environment. Today, data engineers spend 50%+ of their time maintaining data assets. As agentic tools make code…

Building robust, production-grade data pipelines goes beyond writing transformation logic — it requires rigorous testing, version control, automated CI/CD workflows and a clear separation between development and production. In this talk, we’ll demonstrate how Lakeflow, paired with Databricks Asset Bundles (DABs), enables Git-based workflows, automated deployments and comprehensive testing for data engineering projects. We’ll share best practices for unit testing, CI/CD automation, data quality monitoring and environment-specific configurations. Additionally, we’ll explore observability techniques and performance tuning to ensure your pipelines are scalable, maintainable and production-ready.