Austin Choi

Vibe coding is fun until it hits production. In this session, we benchmark two agents running the identical LLM on identical enterprise workloads — one armed with raw SQL and good intentions, one grounded in the Databricks Data Intelligence stack: a Unity Catalog semantic layer, governed metric views, UC functions with LLM-readable docstrings, and Vector Search over enriched business context. We run the benchmark live, on real questions including the ambiguous, multilingual ones your users actually ask. Then we step back and look at what the trajectories tell us about where production agent cost actually comes from — and why the platform underneath your agents matters more than the model running on top of it.

A production-ready GenAI application is more than the framework itself. Like ML, you need a unified platform to create an end-to-end workflow for production quality applications.Below is an example of how this works on Databricks: - Data ETL with Lakeflow Declarative Pipelines and jobs - Data storage for governance and access with Unity Catalog - Code development with Notebooks - Agent versioning and metric tracking with MLflow and Unity Catalog - Evaluation and optimizations with Mosaic AI Agent Framework and DSPy - Hosting infrastructure with monitoring with Model Serving and AI Gateway - Front-end apps using Databricks Apps In this session, learn how to build agents to access all your data and models through function calling. Then, learn how DSPy enables agent interaction with each other to ensure the question is answered correctly. We will demonstrate a chatbot, powered by multiple agents, to be able to answer questions and reason answers the base LLM does not know and very specialized topics.ow and very specialized topics.

Writing prompts for our GenAI applications is long, tedious, and unmaintainable. A proper software development lifecycle requires proper testing and maintenance, something incredibly difficult to do on a block of text. Our current prompt engineering best practices have largely been manual trial and error, testing which of our prompts work well in certain situations. This process worsens as our prompts become more complex, adding multiple tasks and functionality within one long singular prompt. Enter DSPy, your PROGRAMATIC way of building GenAI Applications. Learn how DSPy allows you to modularize your prompt into modules and enforce typing through signatures. Then, utilize state of the art algorithms to optimize the prompts and weights against your evaluation datasets, just like machine learning! We will compare DSPy to a restaurant to help illustrate and demo DSPy’s capabilities. It's time to start programming, rather than prompting, again!