Shubham Baldava

AI agents now out-query humans 29:1, and their median query scans 7K rows, not 200K. That workload wants cheap embedded engines like DuckDB and PyIceberg reading Iceberg directly. But most Kafka and CDC pipelines write updates as equality deletes, a format PyIceberg can't read, Snowflake and Databricks reject, and Iceberg v4 just prohibited (spec vote passed Aug 2026). Your streaming table is "interoperable" only for Spark. This talk covers how we made an open-source CDC-to-Iceberg writer emit positional deletes and v3 deletion vectors at streaming speed. Iceberg has no primary-key index, so we built one: a per-table LSM (Pebble) key-to-row-location index, checkpointed against the table's snapshot, that survives Spark compaction and Trino writes to the same table, and migrates legacy equality deletes in one atomic commit. You'll leave with a 2026 engine-by-delete-format matrix, the write/read/compaction cost of each format, and a decision framework: query engines, then format version, then delete mode. By attending this session, your contact information may be shared with the sponsor for relevant follow up for this event only.