Israel Herraiz

The “Beam Model” is incredibly powerful, but its complexity—balancing windowing, triggers, and stateful processing—often creates a steep learning curve. In the era of agentic development, we are moving beyond simple AI code completion toward a world of Agent Skills: modular, grounded...

With some design, you can leverage Beam ML to make a LLM transform your questions into dynamic transformations that you can apply to your data. In this talk, we show how to use Gemma to accept natural language questions about your data, and offer an answer in real time applied to the main stream...

How to get data processed in order even when the queue deliver messages unordered? One of the main problems when working with data in streaming is out of order. Data will come unordered and late. How come we apply a logic to that stream if it requires data to be ordered? Is there any solution to...

Businesses are looking to harness the power of real-time data with streaming analytics and more often or not that involves computing stateful transformations out of the raw data stream. When the business logic to be applied in streaming is complex and does not use time-based aggregations, it is...

Imagine you have an two unlimited stream of events, one contains IDs and their hashed counterparts for lookups, and one the full information about every object with its hashed id. Your job is to output the full information for each object with it’s clear text id. You could query a single source...

Beam does not have a dedicated complex event processing module, but this can be overcome using state & timers. In this workshop, we will show an example of how to reconstruct sessions from a stream of data, processing the data in order and identifying sessions not based on temporal properties...

In this workshop we will develop a streaming pipeline, showing how to get data in JSON format and parse it (using Beam schemas), how to aggregate it and write it in batches to Avro files. We will apply complex analytics to the stream, calculating properties of a session for different users,...

Apache Beam is an open source SDK for data pipelines, with an unified model for batch and streaming pipelines. The SDK is multi-runner and portable: you may run your pipeline on Apache Spark, Apache Flink, Cloud Dataflow and other runners.

Based in the inspiring talk about the Apollo XI “Light years ahead”, in this talk we will cover 6 design principles for data pipelines, with a focus on streaming pipelines. Submitting a job to a runner is not that different (well, a bit) from “submitting” a spacecraft into space. You lose direct...

We will review the concept of Splittable DoFns and we will write two I/O connectors using this kind of DoFns: one in batch (for reading large files in a given format), and one for streaming (for reading from Kafka topics).

Get your ticket for this workshop Tensorflow Extended (TFX) can run machine learning pipelines on Airflow, but all the steps are run by default in the same workers where the Airflow DAG is running.

In this session, we will explore different windowing strategies, and the impact on dropping (or not) late data. We will use a unit test with a synthetic stream, and then we will apply different windows and will make a grouping calculation (aggregation) to check how different windows group data...