Allison Wang

Debugging distributed Spark jobs is notoriously difficult, often requiring hours of sifting through massive logs to find root causes. We introduce AI_DIAGNOSE, a new Spark SQL procedure in Databricks Runtime that automates this process using large language models (LLMs). This session explores the architecture of AI_DIAGNOSE, detailing how we implemented a ReAct-style agent within the Spark engine. We will demonstrate how the agent autonomously gathers context, retrieves distributed logs, and analyzes job failures. We'll cover technical challenges like managing LLM context windows with large logs, ensuring structured actionable outputs, and leveraging Databricks Model Serving for secure inference. Join us to learn how AI_DIAGNOSE works under the hood to drastically reduce Mean Time To Resolution (MTTR) for data pipelines.

Apache Spark's pipeline story has evolved dramatically— from DStreams to Spark Structured Streaming and now Lakeflow Spark Declarative Pipelines— each generation simplified how we reason about data. But with the rise of agentic AI and vibe coding, how do we ensure our most critical data pipelines remain deterministic, testable, and production-ready? In this session, Scott Haines (Staff Developer Advocate) and Allison Wang (Staff Software Engineer, Databricks and Spark Contributor) will dive into Spark's new Declarative Pipelines API, exploring how to harness the productivity gains of agentic workflows without sacrificing the reliability your data demands. You'll leave with practical patterns for building pipelines that are both developer-friendly and production-hardened and last but not least— open source.

PySpark’s Arrow-based Python UDFs open the door to dramatically faster data processing by avoiding expensive serialization overhead. At the same time, Polars, a high-performance DataFrame library built on Rust, offers zero-copy interoperability with Apache Arrow.

Join us for an interactive Ask Me Anything (AMA) session on the latest advancements in Apache Spark 4, including Spark Connect — the new client-server architecture enabling seamless integration with IDEs, notebooks and custom applications. Learn about performance improvements, enhanced APIs and best practices for leveraging Spark’s next-generation features. Whether you're a data engineer, Spark developer or big data enthusiast, bring your questions on architecture, real-world use cases and how these innovations can optimize your workflows. Don’t miss this chance to dive deep into the future of distributed computing with Spark!

PySpark has long been a cornerstone of big data processing, excelling in data preparation, analytics and machine learning tasks within traditional data lakes. However, the rise of multimodal AI and vector search introduces challenges beyond its capabilities. Spark’s new Python data source API enables integration with emerging AI data lakes built on the multi-modal Lance format. Lance delivers unparalleled value with its zero-copy schema evolution capability and robust support for large record-size data (e.g., images, tensors, embeddings, etc), simplifying multimodal data storage. Its advanced indexing for semantic and full-text search, combined with rapid random access, enables high-performance AI data analytics to the level of SQL. By unifying PySpark's robust processing capabilities with Lance's AI-optimized storage, data engineers and scientists can efficiently manage and analyze the diverse data types required for cutting-edge AI applications within a familiar big data framework.