Lu Niu

Pinterest ML uses Flink for streaming and Spark for batch, but we’re moving toward unifying batch and streaming for PB-scale ML workloads on Flink. For large offline joins that build feature/label and training tables, Spark relies on Storage Partitioned Joins (SPJ): when Iceberg tables are compatibly bucketed, both sides are co-located by key, avoiding expensive shuffles and dramatically reducing

AI/ML is central to Pinterest’s mission of delivering personalized and inspiring experiences to our users. Accelerating new feature exploration, speeding up model training iterations, and shortening the path from development to production are critical challenges for our engineering teams.

At Pinterest, we streaming data processing team built flink as a service to onboard and operate mission critical use cases, directly affect Pinterest revenue and user engagement.

At Pinterest, stream processing is essential for discovering data insights in real-time for many use cases such as ads reporting, ML signals, and real-time experimentation.