Lorenzo Nicora

Watermarks are arguably the most misunderstood concept in Apache Flink. While the internet offers excellent foundational guides, live production environments and complex Flink SQL statements find creative ways for pipelines to go wrong. You write a straightforward query, and suddenly windows refuse to fire, data drops into the void, or processing halts entirely. You try to write a slightly more complex query, only to find yourself wondering: “Hey! Why can’t I put a window aggregation after a simple join?” Turning to an LLM for quick assistance often proves futile; given the sparse documentation and intricate execution paths, AI tends to hallucinate its way through these subtleties of stream processing. This session skips the introductory definitions and dives right into the engine mechanics of watermark generation and propagation. We will dissect how watermarks travel through execution graphs and how some SQL operators manipulate time progression behind your back. The discussion breaks down the tense relationship between partition idleness and watermark alignment, detailing how these two features cooperate or conflict during traffic fluctuations and skewed data distributions.…

Prometheus has become the go-to solution for monitoring and alerting, ingesting metrics from applications and infrastructure. The ability to efficiently store high volumes of dimensional time series also makes Prometheus a perfect fit for broader operational analytics use cases. Examples include observing fleets of IoT devices, connected vehicles, media streaming devices, and any distributed resources. However, the high cardinality and frequency of events generated by these sources can be challenging. Apache Flink can preprocess observability events in real-time before writing to Prometheus. Reducing cardinality or frequency can improve the efficiency of your observability platform. Adding contextual information and calculating derived metrics enables deeper operational analysis in real time. Observing Flink with Prometheus is a solved problem, using Flink Prometheus Exporters. The new Flink-Prometheus connector, a recent addition to the Apache Flink connector family, addresses a different challenge. It enables using Flink to preprocess large volumes of observability data from various sources and write directly to Prometheus at scale. Kafka completes this architecture by providing…

Prometheus is a de facto standard for monitoring and alerting, ingesting metrics from applications and infrastructure. The ability to efficiently store high volumes of dimensional time series also makes Prometheus a perfect fit for other operational analytics use cases, like observing fleets of IoT devices.

In today’s fast-paced business landscape, timely insights from data are crucial for organizations to remain competitive and responsive to dynamic market and customer demands. In this chalk talk, explore the competitive advantages of transitioning from traditional batch processing to stream processing.