Scott Haines

Apache Spark's pipeline story has evolved dramatically— from DStreams to Spark Structured Streaming and now Lakeflow Spark Declarative Pipelines— each generation simplified how we reason about data. But with the rise of agentic AI and vibe coding, how do we ensure our most critical data pipelines remain deterministic, testable, and production-ready? In this session, Scott Haines (Staff Developer Advocate) and Allison Wang (Staff Software Engineer, Databricks and Spark Contributor) will dive into Spark's new Declarative Pipelines API, exploring how to harness the productivity gains of agentic workflows without sacrificing the reliability your data demands. You'll leave with practical patterns for building pipelines that are both developer-friendly and production-hardened and last but not least— open source.

As agents query, transform and feed models, the lineage they leave looks less like a chain and more like a graph. Scott Haines (Databricks) and Jaz Ku (PuppyGraph) show an open pipeline — a Spark plugin emitting OpenLineage over ConnectRPC into Delta/Iceberg, then queried as a graph with Cypher.

What if classic video games could teach you new tricks — and what if community gameplay data could power automation? Join Andrew Sitz (Databricks MVP) and Scott Haines (Staff Developer Advocate) as they build a live gaming environment on Databricks. We'll load a Game from Unity Catalog Volumes, host interactive gameplay through Databricks Apps, and capture every input and state change as streaming data. Using Lakebase and Delta Lake, we persist player sessions, track progress, and serve real-time leaderboards with low latency and full ACID guarantees. From there, we explore how aggregate play data reveals community patterns — surfacing strategies and insights players might never find alone. Finally, we show how recorded gameplay can train simple automation strategies, closing the loop from observation to action. Come for the nostalgia, stay for the data engineering.

Apache Iceberg standardized open table metadata and enabled multi-engine access, but consistent governance across engines remains unsolved. This session explores how Iceberg catalog abstractions and Scan/Plan APIs can act as the control plane for discovery, authorization, and fine-grained policy enforcement in an open lakehouse.

The days of treating data quality as an afterthought are over. While streaming-first architectures have made real-time data more accessible than ever, most organizations still push data validation to downstream systems, creating a cascade of quality issues, broken pipelines, and emergency fixes. What if your streaming data could validate itself? What if you could guarantee semantic correctness at ingestion, eliminating quality problems before they propagate throughout your entire data ecosystem? In this session, Scott Haines will show you how to build intelligent streaming data pipelines that validate data meaning and structure in real-time, freeing your data engineering teams from the endless cycle of reactive fixes. You'll discover: * Why Protobuf is becoming essential for modern data engineering and how it enables semantic validation at scale * How to implement real-time data quality checks using the open-source Protovalidate library * Proven architectural patterns for embedding validation guardrails directly into Kafka streams * Advanced techniques for semantic validation in zero-ETL Unity Catalog environments with Managed Iceberg Walk away with a practical toolkit for…

As data engineering continues to evolve the shift from batch-oriented to streaming-first has become standard across the enterprise. The reality is these changes have been taking shape for the past decade — we just now also happen to be standing on the precipice of true disruption through automation, the likes of which we could only dream about before. Yes, AI Agents and LLMs are already a large part of our daily lives, but we (as data engineers) are ultimately on the frontlines ensuring that the future of AI is powered by consistent, just-in-time data — and Delta Lake is critical to help us get there. This session will provide you with best practices learned the hard way by one of the authors of The Delta Lake Definitive Guide including: - Guide to writing generic applications as components - Workflow automation tips and tricks - Tips and tricks for Delta clustering (liquid, z-order, and classic) - Future facing: Leveraging metadata for agentic pipelines and workflow automation