Ahmed Abualsaud

For a long time, Beam SQL lagged behind other frameworks like Spark or Flink because it lacked support for any hierarchical metadata management. This point of friction limited Beam SQL’s interoperability, scalability, and ease of use within modern data architectures. No more! In this session,...

Traditionally, converting a Parquet-based data lake to Iceberg required a hidden tax of rewriting every single data file. For organizations managing petabyte-scale datasets, this compute overhead and the associated cloud bill are often dealbreakers. This talk introduces a more efficient path...

Join us for a comprehensive overview of Apache Beam’s Iceberg connector. This session will cover its current features and support across multiple SDKs (Java, Python, YAML, SQL). We’ll delve into key design decisions and share what’s coming next. We’ll also demonstrate how to use the connector in...

Harness the power of cross-language transforms by combining the best of Java and Python in your data processing workflows. Discover how to seamlessly integrate Java transforms into your Python pipelines using the SDK’s newest utilities. With just a few lines, you can also automatically generate...

Recent efforts have made it much easier for a pipeline written in one SDK to discover and use a transform developed in another SDK. This is a walk-through of how to take a Java transform, develop a corresponding SchemaTransform for it, then discover and use it in a Python SDK pipeline.