Aleksandar Dragojevic

OpenTelemetry has become the industry standard for collecting traces, metrics, and logs from modern distributed systems. However, operating observability platforms at enterprise scale often requires proprietary tools and fragmented data pipelines. In this session, we present an end-to-end architecture for enterprise observability using OpenTelemetry and ZeroBus with Databricks as the analytics and visualization backbone. We will show how OpenTelemetry telemetry streams can be ingested through ZeroBus and stored in Delta Lake, enabling scalable processing and long-term retention of traces, metrics, and logs. We will demonstrate how to build a custom Databricks App to visualize flame graphs, explore distributed traces, and correlate metrics and logs in a unified interface - delivering capabilities similar to proprietary platforms but powered by the Lakehouse.

In this presentation, we'll show how we achieved a unified development experience for teams working on Mercedes-Benz Data Platforms in AWS and Azure. We will demonstrate how we implemented Azure to AWS and AWS to Azure data product sharing (using Delta Sharing and Cloud Tokens), integration with AWS Glue Iceberg tables through UniForm and automation to drive everything using Azure DevOps Pipelines and DABs. We will also show how to monitor and track cloud egress costs and how we present a consolidated view of all the data products and relevant cost information. The end goal is to show how customers can offer the same user experience to their engineers and not have to worry about which cloud or region the Data Product lives in. Instead, they can enroll in the data product through self-service and have it available to them in minutes, regardless of where it originates.