Partha Sarathi Sahoo

Discover how National Australia Bank (NAB) scales production streaming with Spark Declarative Pipelines (SDP), using Streaming Tables and Materialized Views to enable rewind and replay — restoring pipelines to a consistent state, applying fixes safely and reprocessing only affected data. Explore how this approach reduces brittle manual interventions and avoids full refreshes by leaning on declarative semantics, governed data products, and robust checkpoint/versioning patterns and helps proactively prevent bad data from reaching downstream systems and ensures that data demographics can be safely restored to their previous usable state in the Lakehouse. Walk away with actionable designs, checklists and success criteria for resilient, cost‑efficient streaming at enterprise scale in regulated environments. Outcomes include faster recovery, lower operational effort and cost, improved reliability and correctness and sustained performance as concurrency grows.

Building data processing architectures for multi-tenancy while ensuring low latency, governed access, data integrity, and data observability substantially increases the challenges and complexities of the architecture.

As the scale of business-critical pipelines grows, data operations and data integration teams may find it increasingly difficult to detect problems that impact data pipeline health, to trace root cause across systems, and to predict future issues.