Wayne Yeung

In this session, we will share our journey of modernizing a 40-year-old mainframe legacy system at BEC Financial Technologies, a financial tech provider of core banking platform for over 20 banks in Denmark. We will discuss how we leveraged Kafka to enable real-time data streaming from our mainframe to the new Salesforce platform, creating new business opportunities. Our presentation will cover the transition from traditional end-of-day batch processes to real-time data synchronization, highlighting the challenges and solutions we encountered. We will delve into the importance of DevOps in managing Kafka topics and the implementation of a Kappa architecture to handle both massive spikes and usual real-time data volumes. Key patterns such as event-carried state transfer, compacted topics, and change data capture (CDC) will be explored, along with our data reconciliation mechanisms to ensure consistency between DB2 and Kafka. We will also share lessons learned from our experience, including mistakes to avoid, such as relying on centralized components for data transformation and not using a schema registry. Additionally, we will discuss the benefits of using Kafka for both online…