Giancarlo Costa

In 2020, Delaware implemented a state-of-the-art, event-driven architecture for EFSA, enabling a highly decoupled system landscape, presented at the Data&AI Summit 2021. By centrally brokering events in near real-time, consumer applications react instantly to events from producer applications as they occur. Event producers are decoupled from consumers via a publisher/subscriber mechanism. Over the past years, we noticed some drawbacks. The processing of these custom events, primarily aimed for process integration weren’t covering all edge cases, the data quality was not always optimal due to missing events and we needed to create a complex logic for SCD2 tables. Lakeflow Connect allows us to extract the data directly from the source without the complex architecture in between, avoiding data loss and thus, data quality issues, and with some simple adjustments, an SCD2 table is created automatically. Lakeflow Connect allows us to create more efficient and intelligent data provisioning.

This presentation outlines the results of large, real-world implementation project undertaken by EFSA and delaware. In this project we focused on rolling out an elaborate governance model to enable different data engineering teams to develop, industrialize and share data products in a reliable manner. Inspired by the data mesh principles, we established a decentralized governance model that gives the data engineering teams the autonomy to build further on each other's data products (centered around Delta Sharing for cross-team data sharing) while enforcing all teams to adhere to common policies and shared development best-practices (including the DevOps workflow). This session caters to IT professionals, architects, and decision-makers looking for practical insights on how to realize a decentralized platform governance model in Databricks.