Md Nahidul Kibria

At HelloFresh, we process millions of events daily through Kafka, powering everything from recipe recommendations to delivery logistics. But our data teams faced a persistent challenge: making this streaming data queryable for analytics, AI/ML, and business intelligence without drowning in operational complexity. This talk walks through our technical journey of building a self-service "Shareable Datasets" platform that automatically transforms Kafka topics into queryable Apache Iceberg tables in S3. The Problem We Solved Our legacy approach used Kafka Connect to sink data as Parquet files to S3. While functional, it created significant friction: partition management required manual intervention or expensive Glue Crawlers, schema evolution was painful, and small file accumulation degraded query performance. Data producers couldn't easily debug their own data, and downstream consumers lacked confidence in data freshness and quality. Our Architecture We evaluated multiple approaches—Glue Crawlers, custom Lambda functions, Airflow-based partition management, Flink/Spark—before committing to a Kafka Connect to Iceberg architecture. Key technical decisions included: Kafka Connect with…