Max Schultze

At HelloFresh, speed is our currency. But with 70,000+ tables generated across decentralized domains, our rapid growth threatened to outpace our ability to govern it. We reached a critical inflection point where the “data mesh” approach began to look like a “data mess” — fragmented across AWS Glue, obscured by complex IAM policies, and creating bottlenecks where data access took days to approve. This session unpacks the strategic pivot from reactive catalog management to a unified, scalable governance model on Databricks Unity Catalog. We will move beyond the technical migration to discuss the architectural shifts required to balance autonomy with control. You will learn how we dismantled the trade-off between security and agility, replacing brittle object-based permissions with a streamlined privilege model that empowers data owners. We will share our blueprint for a zero-downtime migration (using Hive Metastore Federation) and the cultural changes required to make governance stick.

Table formats like Delta Lake and Iceberg have been game changers for pushing lakehouse architecture into modern Enterprises. The acquisition of Tabular added Iceberg to the Databricks ecosystem, an open format that was already well supported by processing engines across the industry. At HelloFresh we are building a lakehouse architecture that integrates many touchpoints and technologies all across the organization. As such we chose Iceberg as the table format to bridge the gaps in our decentralized managed tech landscape. We are leveraging Unity Catalog as the Iceberg REST catalog of choice for storing metadata and managing tables. In this talk we will outline our architectural setup between Databricks, Spark, Flink and Snowflake and will explain the native Unity Iceberg REST catalog, as well as catalog federation towards connected engines. We will highlight the impact on our business and discuss the advantages and lessons learned from our early adopter experience.