Justin Breese

Spark Connect is more than a new client interface for Apache Spark™ — it is the foundation for a modern, AI-ready, and production-friendly Spark experience. As AI assistants generate more Spark code, the defaults they learn will shape how the next generation of data applications is built — that makes Spark Connect a strategic shift, not just a technical one. In this session, we will share our vision for making Spark Connect the default way developers build with Spark. We will discuss how DataFrame-first guidance, better AI-generated code compatibility, and machine-readable API discovery can make Connect the easiest path for developers. We will also cover the work needed to build production trust, including closing migration gaps, improving observability, and increasing ecosystem compatibility. Attendees will leave with a clear view of why Spark Connect matters now and what it will take to make it the standard interface for modern Spark applications.

Databricks is bringing first‑class Docker support everywhere. This session shows how we’re making containers a core part of the Databricks developer experience, delivering faster startup, tighter governance, and clearer observability. We’ll walk through the product architecture, the UX investments that reduce time‑to‑first‑result, and how we approach compatibility and safe rollout for critical workloads at scale.

On classic compute, upgrading a DBR version and stitching together cluster libraries, init scripts and Spark configs consume a meaningful share of a platform team’s time, and every upgrade risks breaking production pipelines. Serverless eliminates both with versionless runtime and Databricks’ Environment model. This session covers the two pieces that make versionless trustworthy at enterprise scale. Lakehouse Replay continuously replays a representative sample of real production workloads to validate Databricks runtime changes before they ship, catching regressions with zero customer effort. Environments deliver a unified, portable dependency strategy with faster startup, tighter governance and clearer observability for notebooks, jobs, and pipelines. Attendees leave with a clear picture of how versionless and environments fit together and how to plan their move off manual DBR upgrades.

This session explains how we've made our Apache Spark™ versionless for end users by introducing a stable client API, environment versioning and automatic remediation. These capabilities have enabled auto-upgrade of hundreds of millions of workloads with minimal disruption for Serverless Notebooks and Jobs. We'll also introduce a new approach to dependency management using environments. Admins will learn how to speed up package installation with Default Base Environments, and users will see how to manage custom environments for their own workloads.