Oleksandra Bovkun

You’ve instrumented your GenAI application, captured your MLflow traces using OpenTelemetry standards and stored them in Unity Catalog. Now what? Too often, observability stops at ingestion, treating traces like passive, isolated logs instead of valuable corporate data. This session looks past the initial setup to show the analytical power unlocked when agent telemetry lives inside a governed lakehouse. By merging your trace tables with existing enterprise data, you can move past micro-debugging and start running true business intelligence on your artificial intelligence. Key takeaways: - Cross-domain data ingestion (join MLflow trace tables with Zendesk/Salesforce data to correlate agent performance with ticket volume/CSAT) - SQL-driven agent analytics (isolate systemic tool failures, latency bottlenecks, infinite loops across thousands of runs) - Data-driven evaluation loops (harvest production logs for targeted LLM eval and fine-tuning)

Evaluating multi-agent systems requires a fundamental shift in perspective: we are no longer just testing model outputs, but the integrity of the coordination layer. Instead of just getting one bad response, we get a complex palette of hidden — and not-so-hidden — issues like poisoned shared memory, sub-optimal decision patterns and distributed hallucinations. In this session, we’ll cover how to identify these glitches with MLflow observability, how to move from passive monitoring to active intervention, and which Databricks components can actually prevent the chaos from happening in the first place. The goal of this talk isn't to give you a recipe for every possible disaster, but to broaden your thinking in a world where agents are becoming more autonomous — and significantly more creative with their mistakes.