Craig Lukasik

This presentation will review the new change feed and snapshot capabilities in Apache Spark™ Structured Streaming’s State Reader API. The State Reader API enables users to access and analyze Structured Streaming's internal state data. Readers will learn how to leverage the new features to debug, troubleshoot and analyze state changes efficiently, making streaming workloads easier to manage at scale.

Data engineering teams are frequently tasked with building bespoke ingest and/or egress solutions for myriad custom, proprietary, or industry-specific data sources or sinks. Many teams find this work cumbersome and time-consuming. Recognizing these challenges, Databricks interviewed numerous companies across different industries to better understand their diverse data integration needs. This comprehensive feedback led us to develop the Python Data Source API for Apache Spark™.

As event-driven architectures powered by Apache Kafka™ continue to redefine real-time data processing, the demand for flexible, scalable, and efficient stateful streaming solutions has never been higher. Enter transformWithState, Apache Spark™’s groundbreaking new operator for Structured Streaming, designed to tackle the complexities of stateful processing head-on. In this session, we’ll dive into how transformWithState empowers developers to build sophisticated, low-latency streaming applications with Kafka as the backbone. From flexible state management and timer-driven logic to seamless schema evolution and integration with Kafka, we’ll explore real-world use cases—like real-time fraud detection and session-based analytics—that showcase its power. Attendees will leave with a clear understanding of how to leverage transformWithState to supercharge their Kafka-powered Spark pipelines, complete with practical examples, performance insights, and best practices for production deployment. Whether you’re optimizing stateful aggregations or chaining complex event-driven workflows, this talk will equip you to push the boundaries of what’s possible with Kafka and Spark.