Elliott Botwick

Transform agent debugging from guesswork into a measurable process. Using Snowflake CoCo's agent-optimization skill and native Snowflake Cortex Agent Evaluations, you'll curate evaluation data sets from production logs, run benchmarks with metrics such as answer_correctness and tool_selection_accuracy, analyze failure patterns and iterate on agent instructions — all from the CLI. You'll practice the closed-loop workflow: benchmark → analyze failures → improve → validate. Walk away with a repeatable methodology for taking any Cortex Agents from prototype to production readiness.

To gain trust in Generative AI applications, a robust observability framework is needed. In this session, we will demonstrate how users can instrument an AI application with Trulens, an open source observability framework, to trace each component of their application and use LLM-as-a-Judge evaluations to identify and improve points of failure. Attendees will learn about best practices in enabling observability in their AI workflows and common feedback functions used for evaluation. We will also explore using the observability framework as a mechanism to compare application versions to determine which version is best fit for production use.