Andreas Neumann

Learn how to construct clean, correct, cost-conscious CDC pipelines that conquer chaotic change, column creep and complicated SCD calculations. Today, implementing CDC typically means writing complex, expensive merge logic. Handling straightforward cases is manageable — but complexity escalates quickly when you introduce out-of-order data, duplicates, schema evolution and slowly changing dimensions. What works in development often breaks under real production conditions. Spark Declarative Pipelines (SDP) addresses this with Auto CDC, a declarative API that abstracts away merge orchestration and correctness guarantees. Instead of implementing imperative patterns, you declare intent — such as applying SCD Type 2 — and SDP executes it with built-in consistency, incremental processing and resilience. In this session, we’ll walk through the Auto CDC API, examine how it automates complex merge operations and show how declarative CDC improves reliability across batch and streaming workloads.

Learn how to build batch and streaming pipelines faster while improving correctness and reducing operational complexity with Apache Spark™ Declarative Pipelines. Production Spark pipelines often require extensive orchestration code for dependency management, checkpointing, retries, and execution ordering—surrounding a relatively small amount of transformation logic. As pipelines scale, this scaffolding becomes increasingly difficult to maintain and evolve. Introduced in Spark 4.1, Spark Declarative Pipelines (SDP) shifts this model by allowing developers to declare datasets and transformations while Spark constructs and manages the execution plan. By separating what a pipeline does from how it runs, SDP reduces boilerplate and accelerates time to production. We’ll examine the architectural foundations of declarative development in Spark and how SDP handles dependency resolution, parallelization, checkpoint coordination, and failure recovery. We’ll also cover incremental processing and emerging testing patterns for declarative pipelines.

This presentation will show tips and tricks how to design dynamic and relatively complex forms in QGIS desktop - with the help of the drag and drop form designer, widget configurations, dynamic expressions, data-defined widget visibility, default values, constraints, embedded for…