Pablo Estrada

Jump on a stage and tell us about your latest big data efforts with Beam or otherwise. The first session will be a call to action to build a Generative AI prompt repository for Beam. Then, the rest of sessions will come from volunteers. Sign up with the organizers or sheets on the walls, or...

Beam's mission has been to meet developers where they are: Language of choice, and runner of choice.The Beam Typescript SDK started as a hackathon project to test out how much functionality we could add to a Beam SDK in a week.

The team that builds the Dataflow runner likes to say it “just works”. But how does it really work? How does it decide to autoscale? What happens on failures? Where is state data stored? Can I have infinite state data? What goes into Dataflow’s latency? What’s the checkpoint logic? These and...

Big data systems have implemented the ability to scale up from the cluster perspective: Add more workers, and parallelize further. An issue with this is that many transforms still work with a fixed number of shards that can’t scale up or down with the cluster. We recently added this capability...

Brittany and Austin will provide an update of Apache Beam from the perspective of the community, its growth and initiatives. Pablo will provide an update on the current state of Apache Beam, where and how it is being used.

In this session, we will talk about DebeziumIO, which is a new transform that allows us to read change streams from various databases into Beam by relying on Debezium. The great thing about DebeziumIO is that we can run the Debezium connector within our Beam Pipeline, without having to depend on...

A team of Apache Beam committers & developers will be hosting office hours where they will answer any questions you may have regarding the different components of Apache Beam. 18:00-19:00 Beam ease of use (Alex Amato & Daniel Millls) 18:00-19:00 Python SDK (Pablo Estrada) 19:00-20:00 Beam model...

Learning a whole new SDK and style to write massively parallel data pipelines is certainly a daunting task for any engineer, data scientist, or developer. We have tried to build an end-to-end example of an Apache Beam pipeline that utilizes various valuable Beam features, and tries to establish...

In this talk, we will talk about our experience building pipelines to process data coming from Change-data-capture (CDC) systems. We will review different features, including dynamically adding new tables, managing schema evolution, support for user-provided map functions, dead-letter queue...

Apache Beam was open sourced by the big data team at Google in 2016, and has become an active community with participants from all over. Beam is a framework to define data processing workflows and run them on various runners (Flink included).