Pavithra Rao

We spent a decade learning to ship models reliably. Agents break most of those assumptions: outputs are nondeterministic, behavior depends on tools and context and “good” is defined by domain experts, not a binary test. AgentOps is what MLOps becomes when your system can reason, call tools and act on its own. In 20 minutes, we’ll trace the jump from MLOps to LLMOps to AgentOps and distill it into a practical operating model on Databricks: choosing the right architecture tier (managed AgentBricks vs. code-first), making evaluation a first-class citizen with MLflow LLM judges, governing tools through Unity Catalog and promoting agents across environments with Databricks Asset Bundles. You’ll leave with the stakeholder map, anti-patterns, and DevOps principles needed to move an agent from a demo that impresses to a system production can depend on.

This presentation explores how Databricks' Data Intelligence Platform supports the development and deployment of responsible AI in credit decisioning, ensuring fairness, transparency and regulatory compliance. Key areas include bias and fairness monitoring using Lakehouse Monitoring to track demographic metrics and automated alerts for fairness thresholds. Transparency and explainability are enhanced through the Mosaic AI Agent Framework, SHAP values and LIME for feature importance auditing. Regulatory alignment is achieved via Unity Catalog for data lineage and AIBI dashboards for compliance monitoring. Additionally, LLM reliability and security are ensured through AI guardrails and synthetic datasets to validate model outputs and prevent discriminatory patterns. The platform integrates real-time SME and user feedback via Databricks Apps and AI/BI Genie Space.