Burkhard Ringlein

Today, vLLM is the de-facto industry standard for serving Large Language Models and is widely adopted in production. However, for most of the past, vLLM’s state-of-the-art performance was largely dependent on hand-written CUDA or HIP kernels. These kernels have typically been carefully optimized...

Today, vLLM (part of pytorch) is the de-facto industry standard for serving Large Language Models. vLLM is increasingly being adopted in production and can be executed on NVIDIA GPUS, AMD GPUs, as well as custom accelerators like AWS Inferentia. However, for most of the past, vLLM’s...