Jingsong Li

Paimon has widely practiced its ability to stream updated data into the lake in the industry. Combined with Flink, it can complete real-time updates of data into the lake at the minute level, and can also complete automatic schema evolution, with strong usability.

Apache Paimon has widely practiced its ability to stream updated data into the lake in the industry. Combined with Flink, it can complete real-time updates of data into the lake at the minute level, and can also complete automatic schema evolution, with strong usability.

Alibaba Cloud has extensive practice in using Flink and Paimon (as well as Iceberg) to build streaming lake warehouses, Data Lake Formation (DLF) provides REST metadata services in lake format, and combines products such as Flink, Spark, StarRocks to build a unified storage and multi computing lake warehouse.

Introduction Apache Flink, Apache Flink CDC, and Apache Paimon are the three musketeers of the Streaming Lakehouse. In this session, we will see the history of Apache Paimon, why it was created, and what problems it solves.

Apache Paimon is a Data lake format that enables building a Realtime Lakehouse Architecture with Flink and Spark for both streaming and batch operations. It entered the incubator last year and officially graduated as TLP this year.

In Flink 1.11, the combination of stream computing and hive batch data warehouse brings the ability of Flink stream processing real-time and exactly-once to the offline data warehouse.