Jark Wu

As analytical and AI/ML workloads grow in complexity, traditional streaming storage like Kafka struggles to efficiently store and serve structured data and multimodal AI data.

This topic will share our newly developed next-generation streaming storage, Fluss, designed specifically for streaming analytics. We will focus on how Fluss addresses the pain points faced by traditional streaming storage used in streaming analytics.

Kafka and Flink SQL have been widely used together in streaming processing scenarios, becoming a defacto standard paradigm for building streaming warehouses and real-time analytics.

Flink SQL is a powerful tool for stream processing that allows users to write SQL queries over streaming data. However, building a streaming SQL engine is not an easy task.

Change Data Capture (CDC) has become a popular pattern to capture committed changes from a database and propagate those changes to downstream consumers, for example to keep multiple datastores in sync and avoid common pitfalls such as dual writes.

During last two major versions (1.9 & 1.10), Apache Flink community spent lots of effort to improve the architecture for further unified batch & streaming processing.