Brian Law

As LLM applications move into production, redundant calls quietly drive up latency, cost, and complexity. Traditional caching falls short — exact string matches miss the majority of real-world queries where users ask the same question in different ways.In this session, we introduce semantic caching: a smarter approach that retrieves responses based on intent, not syntax. You'll learn how to build a production-ready semantic cache gateway using Lakebase and MLflow to capture paraphrased and context-aware queries. We'll walk through why exact-match caching breaks down at scale, how to design a low-latency caching layer with a FastAPI gateway on Databricks Apps, and how to incorporate conversational context for multi-turn accuracy — plus the observability and governance patterns that make it production-safe.Walk away with a deployable architecture and proven patterns to cut LLM costs by up to 80% while improving response speed and reliability.

The explosive growth in token consumption has made deploying LLMs at scale increasingly complex. Teams must balance latency, cost, and reliability. Databricks Model Serving is designed to absorb these operational challenges across the full model spectrum: proprietary APIs, open-source foundation models, fine-tuned variants, and fully custom architectures. In this talk, we'll go under the hood of how we built token- and cache-aware routing, intelligent scheduling, and autoscaling to deliver efficient, resilient inference at scale. We'll then show how we extended this infrastructure with high-throughput batch support through the ai_query function in Databricks SQL, bringing LLM capabilities natively into data pipelines. Finally, we'll explore how we enable customers to serve their own post-trained/finetuned models or custom transformer-based architectures. Whether you're optimizing costs, scaling to millions of requests, or bringing a custom architecture to production, this session offers a clear picture of what production-grade LLM serving looks like end to end.