Roman Kolesnev

The CDC pattern was created to make analytical tables reflect operational changes. Today, Iceberg is where those changes are materialized for querying but CDC-powered pipelines into Iceberg are still under adopted. This is because CDC requires a mutability that is still clunky in Iceberg. Pipelines from Kafka topics to Iceberg tables lead to stale data, metadata bloat, small file proliferation, and delete file explosions that degrade query performance and incur large maintenance overhead. This talk explores an alternative: using CDC as the foundation for a zero-copy, composable data architecture. Instead of continuously ingesting data into Iceberg via ETL, we compose a queryable view that spans CDC topics in Kafka and post CDC materialised tables in Iceberg. Data for this view is assembled at query-time, not ahead-of-time, sidestepping a number of the Iceberg issues described above. In this talk we’ll discuss the advantages of such an approach including: real-time access to fresh data, reduced infrastructure overhead, and simpler data flows as well as the trade-offs including the impact on edge case query performance. Finally, we’ll go under the hood with a technical deep dive…

Over a decade ago, Martin Kleppmann's Turning the Database Inside Out reshaped how we think about data systems, putting the event stream at the heart of storage and computation. That vision inspired a generation of systems built atop Kafka, Flink, and event-driven materializations. But what if we never finished what Martin started? This talk takes the next leap, reimagining not just the transaction log, but the entire database through the lens of streaming. We'll keep Kafka as our canonical source of truth, but enrich it with the missing primitives: long-term storage, indexes, and projections. To achieve this, we'll move beyond Kafka's simple produce/consume model and embrace Apache Iceberg as the new foundation for durable, queryable event data. This architecture collapses the fragile ETL sprawl and unifies real-time and historical data into a single, coherent system. Answering questions from "what's happening right now?" all the way back to "what happened at the beginning of time?". You'll leave seeing the database (and the stream) in a whole new light.

An organisation's data has traditionally been split between the operational estate, for daily business operations, and the analytical estate for after-the-fact analysis and reporting. The journey from one side to the other is today a long and torturous one. But does it have to be? In the modern data stack Apache Kafka is your defacto standard operational platform and Apache Iceberg has emerged as the champion of table formats to power analytical applications. Can we leverage the best of Iceberg and Kafka to create a powerful solution greater than the sum of its parts? Yes you can and we did! This isn't a typical story of connectors, ELT, and separate data stores. We've developed an advanced projection of Kafka data in an Iceberg-compatible format, allowing direct access from warehouses and analytical tools. In this talk, we'll cover: * How we presented Kafka data for Iceberg processors without moving or transforming data upfront—no hidden ETL! * Integrating Kafka's ecosystem into Iceberg, leveraging Schema Registry, consumer groups, and more. * Meeting Iceberg's performance and cost reduction expectations while sourcing data directly from Kafka. Expect a technical deep dive into…