Ismaël Mejía

Apache Iceberg, Hudi & Delta Lake are leading open-source projects providing decoupled storage with advanced transaction & metadata layers, known as table formats, in cloud storage. When data is stored in a distributed file system, these formats offer similar features: a table abstraction over files that includes schema, commit history, partitions & column statistics.

When Data Engineers think about unstructured data, basically the first idea that comes to mind is those pesky legacy files we need to transform to extract into some ‘good old’ structured table somewhere. But the recent improvements on Machine Learning and the growing popularity of Large Language Mod

Apache Avro is the de-facto standard for serializing structured data: Big data engines, streaming platforms, and data lakes use it to optimise storage or transmission of data.Avro is stable and mature, and usually "just works", but there's a lot going on behind the scenes.

TPC-DS is the de-facto SQL-based benchmark framework used to measure database systems and Big Data processing frameworks. Beam introduced an early TPC-DS implementation last year but so far we have not started to use it to measure the state of the performance of Beam.

In this talk we’ll provide an overview of the current and upcoming developments in the Apache Beam Runners. Based off that, we will look into new use cases and patterns. Finally, we want to brainstorm about new items for the roadmap, and encourage users and developers to share their ideas.