Jiale Tan

Apache Spark, a pivotal component in large-scale data processing, has experienced a surge in adoption as organizations embark on the utilization of cloud environments for their analytics and machine learning applications. While Spark provides native support for Kubernetes, constructing a production-ready batch data platform utilizing Spark necessitates substantial effort.

Apache Spark, a cornerstone in large-scale data processing, faces challenges in fault tolerance when running on Kubernetes, especially in environment like shared cluster or with Spot virtual machines, where shuffle data are vulnerable to underlying infrastructure instabilities. To make Spark super reliable and also leverage Spot Instance/VM to save cost, we built a new Shuffle Manager to store shu