Emeli Dral

AI agents and multi-step workflows are powerful, but testing them can be tricky. This talk explores practical ways to test these complex systems — like running multi-step simulations, checking tool calls, and using LLMs for evaluation.

LLM systems are powerful, but it can be challenging to ensure their reliable and effective operation in production. In this talk, we will explore continuous testing, one of the critical components for LLM safety.

How can you evaluate your production models when the data is not structured and you have no labels? To start, by tracking patterns and changes in the input data and model outputs.

Proper monitoring of machine learning models in production is essential to avoid performance issues. Setting up monitoring can be easy for a single model, but it often becomes challenging at scale or when you face alert fatigue based on many metrics and dashboards.