Jerry Peng

Apache Spark Structured Streaming is a distributed stream processing engine built on Spark SQL and is widely used for ETL-style workloads, where seconds- to minutes-level latency is acceptable. But millisecond-latency use cases have historically been out of reach.

Apache Spark Structured Streaming has quickly become the foundational open-source technology for powering mission-critical streaming ingestion and ETL pipelines worldwide. The Spark 4.1 release introduces powerful updates, most notably a new Real-Time Mode that enables operational pipelines to process data with latencies in milliseconds, all within the Spark ecosystem. Stateful transformations have been significantly bolstered through TransformWithState and stream-stream join enhancements, offering broader coverage and higher performance. Furthermore, robust improvements—such as revamped locking and integrity verification—provide improved stability for stateful queries. Looking ahead to Spark 4.2, the roadmap includes state repartitioning, query evolution, hybrid sources, and other enhancements, further establishing Spark as the gold standard for all streaming workloads. Join us to explore these recent advancements and the exciting future of open-source Structured Streaming!

Feature freshness is critical for fraud-detection, ad tech, real-time recommendations, dynamic pricing, etc. Historically, ML engineers have had to maintain two separate systems for feature engineering, leveraging Spark and Spark Structure Streaming for batch/near real-time features, while building bespoke pipelines when they needed sub-second feature freshness. In this session, we'll learn about Real-Time Mode in Apache Spark Structured Streaming, the engine behind the Databricks Feature Store, and how companies like DraftKings are using it to power real-time features with millisecond-level freshness. You'll learn how to pair Real-Time Mode with Lakebase to build a state-of-the-art online feature store and leave with a practical blueprint for a unified feature engineering platform that eliminates logic spread across different systems and shifts the focus from infrastructure orchestration to semantic feature modeling.

As enterprise streaming adoption accelerates, more teams are turning to real-time processing to support operational workloads that require sub-second response times. To address this need, Databricks introduced Project Lightspeed in 2022, which recently delivered Real-Time Mode in Apache Spark™ Structured Streaming. This new mode achieves consistent p99 latencies under 300ms for a wide range of stateless and stateful streaming queries. In this session, we’ll define what constitutes an operational use case, outline typical latency requirements and walk through how to meet those SLAs using Real-Time Mode in Structured Streaming.

Real-time mode is a new low-latency execution mode for Apache Spark™ Structured Streaming. It can consistently provide p99 latencies less than 300 milliseconds for a broad set of stateless and stateful streaming queries. Our talk focuses on the technical aspects of making this possible in Spark. We’ll dive into the core architecture that enables these dramatic latency improvements, including a concurrent stage scheduler and a non-blocking shuffle. We’ll explore how we maintained Spark’s fault-tolerance guarantees, and we’ll also share specific optimizations we made to our streaming SQL operators. These architectural improvements have already enabled Databricks customers to build workloads with latencies up to 10x lower than before. Early adopters in our Private Preview have successfully implemented real-time enrichment pipelines and feature engineering for machine learning — use cases that were previously impossible at these latencies.