Irene Rodríguez Clavería

When moving data from external systems to Apache Kafka, data balance is not just a matter of organization. It is the real key to the performance of the entire downstream ecosystem. A uniform distribution allows consumers to scale linearly and run efficient filters. However, in practice, external data sources such as relational databases are rarely balanced. This creates the problem of Data Skew and hot partitions in Kafka. These hot partitions are a major issue because they cause high latency in your pipelines. When some partitions have much more data than others, the total time to process information increases and the system becomes slow. In New Relic, we faced this challenge while processing massive volumes of data. In this session, we will present the architecture we designed using Flink CDC for ingestion and Apache Paimon as the main tool for load redistribution. We will explain how we managed to reduce skew using different partitioning strategies and the use of dynamic and static buckets in Paimon before the final delivery to Kafka. This approach ensures that data is already balanced when it reaches the Kafka producers. Attendees will learn practical patterns to transform…