Ganesh Mahadevan

As data volumes scale, the friction between external Apache Spark clusters and cloud warehouses often creates latency and governance challenges. BNC (Banque Nationale du Canada / National Bank of Canada) addressed this by migrating PySpark pipelines directly into Snowflake’s unified compute platform using Snowpark Connect. By eliminating external infrastructure management while maintaining Python-based flexibility, BNC streamlined its operations and simplified data oversight. This session explores the technical architecture of the migration, strategies for handling complex dependency management, and benchmark results comparing their legacy Spark environment to the new, optimized Snowflake implementation.

This session explores how RGA transformed its large-scale risk model pipeline by migrating from a complex Spark and Docker architecture to a unified, Snowflake-native solution. By leveraging Snowpark and user-defined table functions, RGA eliminated data movement and the security risks of external processing. Attendees will learn how consolidating onto a single platform simplified infrastructure while improving performance, security and operational manageability. We will highlight how this streamlined architecture provides a highly scalable framework built to handle massive volume growth, ensuring the pipeline remains resilient as data demands increase.

Unlock optimized performance and cost efficiency! Explore Penske’s successful journey converting Databricks workloads to process 50% faster in Snowpark. This real-world case study will equip you with key technical paradigms and keen insights you’ll need to achieve a frictionless migration for your organization. Deliver a more streamlined and performant data engineering environment that also translates into tangible financial savings through reduced infrastructure costs and optimized compute.