Luke Chen

Cross-cluster replication is essential for disaster recovery, geographic distribution, and data migration. Today, MirrorMaker 2 is the standard tool for this job, but it comes with well-known trade-offs: a separate Connect cluster to deploy and monitor, lossy offset translation that causes duplicate processing during failover, etc. KIP-1279 takes a different approach. Cluster Mirroring embeds cross-cluster replication directly into the Kafka broker runtime, eliminating external infrastructure entirely. But getting this right is harder than it looks. The design had to solve several subtle problems: - Unclean leader elections on either cluster can silently diverge replica logs, yet failover and failback must remain consistent. - Transactional records carry producer state that becomes stale the moment a mirror topic turns writable, yet idempotent producers must be able to resume cleanly. - Leader epochs on source and destination can drift apart, yet consumers must be able to fetch from their last committed offset without falling into infinite metadata refresh loops. - Source clusters can run versions as old as Kafka 2.1, long before leader epochs or modern fetch features existed, yet…

Apache Kafka v4.0 introduces significant changes that will affect how you deploy, configure, and operate your Kafka clusters. In addition to the famous "zookeeper removal", what else does Kafka v4.0 bring to us? As a Kafka user, what changes will impact my existing kafka cluster? As a Flink user, am I safe from the kafka v4.0 upgrade when adopting flink kafka connector? In this session, I'll go through all the important changes in Apache Kafka v4.0, like the log4j2 upgrade(KIP-653), the next-generation rebalance protocol(KIP-848), the async consumer, the Eligible Leader Replicas(KIP-996), some default configuration changes(KIP-1030), and many component/API deprecation and removal...etc. Not just introducing these changes, most importantly, I'll let you know what they impact your existing cluster, for both Kafka and Flink. Like in the change of KIP-996, will it change the existing unclean leader election mechanism? like in the change of KIP-848, will the Flink kafka connector need to adopt the new async consumer for the new rebalance protocol..., like in the removal of old client protocol API versions(KIP-896), will it be incompatible to my existing kafka clients or Flink Kafka…