Doron Porat

We land 200+ billion events a day, around 40,000 Avro files per minute, into Apache Iceberg, queryable in under a minute. Our first production system did this on Flink with Paimon. It worked, but at this fan-out and freshness target we kept paying for the same things: tiny files flooding the table, compaction and stats running as after-the-fact jobs that re-read the data, one hot partition pinning a single task slot, and elasticity gated by checkpoint and state redistribution. None of these are bugs. They are the honest cost of a general-purpose, keyed-state engine pointed at a problem that is really just file fan-in to a table. So we rebuilt it in Rust as a set of independent, single-purpose services: writers that just flush Parquet, compactors that just merge and sort, and one committer that is the only component touching Iceberg metadata. Nothing shares state, each scales on its own signal, and losing a worker only re-delivers its own in-flight files. Owning the whole path also let us fold in things a bolt-on engine cannot: inline statistics computed in the same write pass, clustering and LSM-style compaction on ingest, dedup, and read-free metadata commits. This is a blueprint…