Carlos Rodrigues

Spark Structured Streaming offers powerful built-in operators for stream-stream joins, but what happens when you need to join two streams based on temporal proximity rather than exact key matches? As-of temporal joins—where each event from one stream matches the most recent corresponding record from another—are essential for scenarios such as currency conversion at transaction time, enriching trades with the latest quotes, or correlating sensor readings with their nearest calibration values. Apache Spark 4.0 introduces transformWithState, a next-generation stateful processing operator that replaces the limitations of flatMapGroupsWithState. This talk demonstrates how to leverage transformWithState to build production-grade as-of temporal joins between two event streams. We'll explore the core building blocks: using MapState to maintain versioned lookup data keyed by time, processing incoming events to find the nearest temporal match, and managing state lifecycle with TTL-based eviction to prevent unbounded growth. You'll see how the new object-oriented StatefulProcessor model separates concerns cleanly—handling input events in handleInputRows and expired state in…

With ever-growing data volumes and real-time requirements, tuning Apache Iceberg for performance while controlling costs can be challenging. This builders’ session dives deep into how Iceberg works, its configuration, advanced performance optimization options, and how to tune Iceberg for different workload patterns.

With ever-growing data volumes and real-time requirements, configuring Apache Hudi for performance while controlling costs can be challenging. This builders’ session dives deep into how Hudi works, its configuration, options for advanced performance optimization, and how to tune Hudi for different workload patterns.