Sebastiaan Leysen

This presentation outlines the results of large, real-world implementation project undertaken by EFSA and delaware. In this project we focused on rolling out an elaborate governance model to enable different data engineering teams to develop, industrialize and share data products in a reliable manner. Inspired by the data mesh principles, we established a decentralized governance model that gives the data engineering teams the autonomy to build further on each other's data products (centered around Delta Sharing for cross-team data sharing) while enforcing all teams to adhere to common policies and shared development best-practices (including the DevOps workflow). This session caters to IT professionals, architects, and decision-makers looking for practical insights on how to realize a decentralized platform governance model in Databricks.

EFSA is the European agency providing independent scientific advice on existing and emerging risks across the entire food chain. On 27/03/2021 a new EU regulation (EU 2019/1381) has been enacted, requiring EFSA to significantly increase the transparency of its risk assessment processes towards all citizens. To comply with this new regulation, delaware has been supporting EFSA in undergoing a large Digital Transformation program. We have been designing and rolling-out a modern data platform running on Azure and powered by Databricks. This platform acts as a central control tower brokering data between a variety of applications. It is built around modularity principles, making it adaptable and versatile while keeping the overall ecosystem aligned w.r.t. changing processes and data models. At the heart of the platform lie two important patterns: 1. An Event Driven Architecture (EDA): enabling an extremely loosely coupled system landscape. By centrally brokering events near real-time, consumer applications can react immediately to events from producer applications as they occur. Event producers are decoupled from consumers via a publisher/subscribe mechanism. 2. A central data store built around a lakehouse architecture. The lakehouse collects, organizes and serves data across all stages of the data processing cycle, all data types and all data volumes. Events streams from the EDA layer feed into the store as curated data blocks and are complemented by other sources. This store in turn feeds into APIs, reporting and applications, including the new Open EFSA portal: a public website developed by delaware hosting all relevant scientific data, updated in near real-time. At delaware we are very excited about this project and proud of what we have achieved with EFSA so far.