Tom Stepp

This session explores recent innovations that enhance pipeline scalability, reliability, and availability. We will cover key updates in autoscaling, high availability, and reliability, alongside progress in streaming ML and IO excellence. Attendees will discover how these enhancements facilitate...

The presentation at this year’s Beam Summit will highlight the latest advancements in Dataflow Streaming. Join us to learn about fast job updates, KafkaIO improvements, Streaming AI/ML enhancements, performance and observability enhancements.

Level up your streaming pipelines by learning about the latest advancements in Dataflow Streaming. This includes new features such as At-Least-Once Mode, Active Load Balancing, Autoscaling Hints, In-flight Autoscaling Updates, best practices for Kafka source pipelines and ML pipelines.

During this talk we will show what we have been working on for the last year and what we are doing to make Dataflow better. This includes improvements to autoscaling, load balancing, autosharding, and Pub/Sub integration.