Jun He

Incremental processing substantially reduces compute costs and execution time, leading to fewer failures and less manual intervention. However, enabling it on large-scale data workflows presents significant challenges around scalability, ease of adoption, and user experience.

Incremental processing, an approach that processes only new or updated data in workflows, substantially reduces compute resource costs and execution time, leading to fewer potential failures and less need for manual intervention.

As big data and ML became more prevalent and impactful, the scalability, reliability, and usability of the orchestrating ecosystem have become increasingly important for Netflix.