Philip Kiely

Coding agents are a dominant use case for LLMs, but they tend to use generalized inference setups even at large scale. Like any optimization problem, in inference you can get better results with better constraints.

Four-bit quantization has a bad reputation for destroying model quality. While it’s true that post-training quantization in 4-bit integer formats makes models noticeably worse, new microscaling data formats like MXFP4 and NVFP4 deliver on the promise of fast low-precision inference without meaningful quality loss.

Everyone wants faster inference, but no one wants to compromise the quality of their model outputs. FP8 quantization offers 30-50% lower latencies for inference on large models, but must be applied carefully to maintain quality. Recently, NVIDIA Blackwell GPUs introduced new microscaling number...

Do you want to learn how to serve models like DeepSeek and Qwen with SOTA speeds on launch day? SGLang is an open-source fast serving framework for LLMs and VLMs that generates trillions of tokens per day at companies like xAI, AMD, and Meituan.

How do you get time to first byte (TTFB) below 150 milliseconds for voice models -- and scale it in production? As it turns out, open-source TTS models like Orpheus have an LLM backbone that lets us use familiar tools and optimizations like TensorRT-LLM and FP8 quantization to serve the models with low latency.

GPUs & Inference: TensorRT-LLM is the highest-performance model serving framework, but it can have a steep learning curve when you’re just getting started. We run TensorRT and TensorRT-LLM in production and have seen both the incredible performance gains it offers and the hurdles to overcome in getting it up and running.