Siying Dong

Databricks’ Photon engine has revolutionized Spark execution, delivering state-of-the-art query performance and cost efficiency. While many Structured Streaming workloads already benefit from this vectorized execution engine, a subset of streaming-specific operators has historically relied on the standard JVM execution model, leaving potential optimization on the table. Until now. In this session, we reveal our recent engineering initiatives to expand Photon coverage to these critical streaming operators. We will dive into the technical journey of “Photonizing” these components, detailing the complex engineering challenges we overcame to bring vectorized execution to stateful and streaming-specific logic. Attendees will see side-by-side performance benchmarks demonstrating how these enhancements translate into lower latency, higher throughput, and reduced Total Cost of Ownership (TCO) for streaming pipelines

Real-time mode is a new low-latency execution mode for Apache Spark™ Structured Streaming. It can consistently provide p99 latencies less than 300 milliseconds for a broad set of stateless and stateful streaming queries. Our talk focuses on the technical aspects of making this possible in Spark. We’ll dive into the core architecture that enables these dramatic latency improvements, including a concurrent stage scheduler and a non-blocking shuffle. We’ll explore how we maintained Spark’s fault-tolerance guarantees, and we’ll also share specific optimizations we made to our streaming SQL operators. These architectural improvements have already enabled Databricks customers to build workloads with latencies up to 10x lower than before. Early adopters in our Private Preview have successfully implemented real-time enrichment pipelines and feature engineering for machine learning — use cases that were previously impossible at these latencies.