Ceren Morey

Data contracts are agreements between the people who produce data (upstream systems, pipelines, teams) and the people who consume it (analytics, machine learning, applications). Data contracts define what a dataset should look like, how fresh it should be, who can access it, and what quality it must meet. The problem is that most data contracts in enterprises live in the wrong place: architecture blogs, Confluence pages, YAML files nobody reads after the first sprint. A markdown file that says "order_id must not be null" does not stop a null from being written. A Confluence page that says "this table is updated hourly" does not alert anyone when the pipeline fails. This session focuses on the technical enforcement layer of a data contract. Using Azure Databricks Unity Catalog and Lakeflow Spark Declarative Pipelines, I will demonstrate how six of the data contract attributes: schema, quality, usage policies, security, SLAs, and data description can be defined once, enforced automatically, monitored continuously, and versioned alongside your pipeline code in Git. At the end of this session, you will leave with a GitHub repository containing contract templates and a practical framework for turning your existing governance policies into objects that Databricks can actually enforce. The session is for data engineers and platform teams working with Azure Databricks who want to bring their data contracts into the codebase.

Building trust in your data starts with making quality visible, testable, and automatable because without reliable data, even the most advanced platforms fall short. Data quality is a critical component of any data engineering workflow, yet it often becomes a complex and time-consuming challenge. This session will be a practical, hands-on walkthrough of setting up data quality checks with Soda Core, an open-source framework, in a Microsoft Fabric environment, from scratch to automation. With Soda’s low-code, declarative approach, implementing quality checks becomes far more approachable, even for teams with limited engineering resources. Combined with Fabric’s own low-code experience, this makes for a powerful match and an accessible way to bring quality checks into daily workflows. I will demonstrate how to define data quality checks including both built-in tests like data freshness and uniqueness, as well as custom checks tailored to specific business logic. Further demonstrate how to securely connect to Lakehouse tables, automate and monitor validations as part of data pipelines, all within a setup that’s suitable for production environments. We will also explore multiple quality check execution options: - From VS Code via CLI using YAML configuration - As a scheduled Azure Function for automated scanning - Embedded within Spark notebooks in Fabric By the end of this session, you’ll walk away with a practical blueprint for: - Integrating Soda Core into Fabric - Running various data quality checks at different stages of the pipeline - Monitoring and operationalizing quality metrics - Building trust in your data with a minimal learning curve This session is ideal for data engineers, platform teams, and analytics professionals who want a hands-on, scalable approach to solving a foundational challenge in modern data platforms.