Saravanan R

In this session, Team Yubi demonstrates how an intelligent streaming data pipeline leveraging Apache Kafka creates a unified analytical platform to deliver near real-time insights from a centralized Redshift data warehouse. Business operations teams face challenges approving large-ticket trades due to fragmented data across multiple systems managed by different teams. Fetching and reconciling this data often involves writing complex queries—expertise many operations teams lack—leading to delays in due diligence and decision-making. To solve this, we built a robust streaming data pipeline that centralizes disparate data sources into Redshift. The pipeline uses Apache Kafka for streaming, Kubernetes for scalability, dbt for data transformations, and Redshift WLM with data sharing for optimized query execution. Our custom Kafka sink connectors process data efficiently in two modes—snapshot (replicating the source RDS) and CDC (capturing incremental changes)—within a single flush cycle. This approach keeps the warehouse up-to-date, reduces ETL loads, lowers infrastructure costs, and enables quick data refresh cycles. The unified platform also lays the foundation for AI-based…