Gyula Fora

Managing complex applications such as data processing systems on Kubernetes is a formidable challenge even for the most seasoned engineers. Whether you want to build applications that operate themselves or provision infrastructure from Java code, Kubernetes Operators are the way to go.

In this interactive panel discussion, our panelists will be discussing best practices of running Flink SQL jobs reliably in production environments, its inherent challenges and pitfalls.

In this talk we will walk you through an end-to-end streaming SQL pipeline that consumes data from multiple Kafka sources, applies enrichment information from Kudu and executes a simple alerting logic.

One of the biggest challenge in streaming application development is making sure your pipeline does exactly what it is supposed to do. The combination of different data sources, sinks and complex application behavior such as time based functionality or interaction with external systems doesn’t make the problem of proper testing any easier.

In this talk we take an in-depth tour of the different job and cluster configuration options available in Flink and how to leverage them to maximize performance, stability, security and observability of your pipeline.