Talat Uyarer

While Apache Iceberg has long supported Change Data Capture (CDC) via incremental changelog scans, this functionality was historically limited to Copy-on-Write (CoW) tables. For users leveraging Merge-on-Read (MoR) to handle high-frequency updates and deletes, a significant gap remained: the inability to accurately track row-level changes when positional or equality delete files were involved.

While Apache Iceberg leads in data interoperability, its reliance on absolute paths complicates essential enterprise tasks like replication, migration, and archival. These operations require dataset portability, yet absolute paths create significant cost and complexity.

At Palo Alto Networks, We are using Beam on Dataflow for 10K+ jobs. Beam has good abstraction run on multiple runner. For multi Cloud Provider use case We developed a fully managed stream processing platform on Flink running on K8s to power thousands of stream processing pipelines in production...

At Palo Alto Networks we heavily rely on Avro, using it as the primary storage format and use Beam Row as in memory. We de/serialize billions Avro records per second. One day we realized Avro Row conversion routines consume much of CPU time. Then the story begins ….

Cortex Data Lake collects, transforms and integrates your enterprise’s security data to enable Palo Alto Networks solutions. We build streaming infrastructure for our customers. I will share share our architecture and experience while building that infrastructure.