Charles Frye

Inference optimization is a grind. The big wins like quantization affect model quality, triggering numerous, expensive quality evals. Both host and accelerator performance improvement is a game of inches. And the engines have more flags than an Erasmus alumni meetup in Brussels.

Open weights models and open source inference servers have made massive strides in the year since we last got together at AIE World's Fair. Where once we had only pirated LLaMA 2 weights and Transformers, we now have an embarrassment of riches. In fact, we have too many choices! What's an AI engineer looking to self-host inference to do?

Every programmer needs to know a few things about hardware, like processors, memory, and disks. Due to AI systems' extreme demand for mathematical processing power, AI engineers need to know a few things about GPUs -- the world's most popular high-throughput mathematical co-processor.

In this talk, AI Engineer Charles Frye will discuss the stack for running your own LLM inference service. We’ll cover: compute options like CPUs, GPUs, TPUs, & LPUs; model options like Qwen & LLaMA; inference server options like TensorRT-LLM, vLLM, & SGLang; and observability options like the OTel stack, LangSmith, W&B Weave, & Braintrust.

Optional introductory course for AI Engineers, free for all Summit attendees. Advanced knowledge of AI Engineering, led by instructor Charles Frye of the massively popular Full Stack LLM Bootcamp. Overview: - Part I: Running Inference - What is the workload?