Hongyue Zhang

Apache Iceberg is a popular table format for managing large analytical datasets. But replicating iceberg tables at scale can be a daunting task—especially when dealing with its hierarchical metadata.

Data quality plays a crucial role in data engineering to enable efficient and insightful data pipelines at scale. In this session, we will leverage Apache Iceberg as the scalable table format with ACID guarantee, Apache Toree’s interactive computation capabilities and orchestrate the automated data workflow on Apache Airflow.

This session explores the integrated use of Apache Toree, YuniKorn, Spark, and Airflow to create efficient, scalable data pipelines. We will start by discussing how Apache Toree provides an interactive analysis environment with Spark via Jupyter Notebook.

Catalog provides unified, secure access to structured data. Apache Iceberg is a new open source table format, which enables ACID-compliance and scales for petabyte datasets.

Catalog provides unified, secure access to structured data. Apache Iceberg is a new open source table format, which enables ACID-compliance and scales for petabyte datasets. While the same Hive metastore can be used for cataloguing when migrating from Hive to Iceberg table, its convenience is quickl