Daniel Tenedorio

Apache Spark™ is optimized for distributed large-scale processing, but this architecture introduces significant overhead for small, local datasets — often making sub-100 MB queries take several seconds. In this talk, we present a suite of optimizations that dramatically improve Spark’s local-mode performance across three fronts: smarter query compilation and task scheduling that eliminates unnecessary shuffles, an Arrow-based df.cache implementation that cuts cache load and query times in half and a novel shuffle-free execution mode using Java virtual threads for lightweight in-process data transfer. We will walk through the design, benchmarks and engineering trade-offs of each approach, and show how together they position Apache Spark as a practical gateway engine from laptop-scale exploration to petabyte-scale production.

Explore the exciting new features of Apache Spark 4.1 with major new capabilities for both data engineering and analytics workflows. We introduce Spark Declarative Pipelines (SDP) to let users define what their data pipelines should accomplish while Spark orchestrates execution, dependencies, parallelism and retries, an official Real-Time Mode in Structured Streaming for sub-second, continuous low-latency processing, and extensive improvements to the PySpark ecosystem with Arrow-native UDFs/UDTFs, Python Data Source filter pushdown, and better Python worker logging. The release also brings Spark Connect enhancements including GA support for Spark ML on the Python client and increased stability for large workloads, and expands SQL support with GA SQL Scripting, the VARIANT type with shredding optimizations, support for recursive CTEs, and new approximate data sketch functions. Spark 4.1 prioritizes higher-level abstractions, real-time performance, richer Python support, and more.

Join this session for a concise tour of Apache Spark™ 4.0’s most notable enhancements: - SQL features: ANSI by default, scripting, SQL pipe syntax, SQL UDF, session variable, view schema evolution, etc. - Data type: VARIANT type, string collation - Python features: Python data source, plotting API, etc. - Streaming improvements: State store data source, state store checkpoint v2, arbitrary state v2, etc. - Spark Connect improvements: More API coverage, thin client, unified Scala interface, etc. - Infrastructure: Better error message, structured logging, new Java/Scala version support, etc. Whether you’re a seasoned Spark user or new to the ecosystem, this talk will prepare you to leverage Spark 4.0’s latest innovations for modern data and AI pipelines.