Ekin Karabulut

As inference demand explodes, new techniques to optimize these deployments have emerged. One such technique is disaggregated inference, which splits inference into differently optimized workloads (e.g. prefill and decode) on separate workers. The theory is straightforward–better GPU utilization,...

Data transfer is slow -- so in AI and HPC, data locality matters. As workloads scale, optimizing where and how to run data-heavy workloads in Kubernetes becomes critical. Yet this area remains underexplored. The CNCF Batch Subproject shares findings from our work on data-locality-aware scheduling...

Traffic is spiking to your ML application. Your autoscaler kicks in. But instead of serving more requests, your new replicas are stuck downloading massive model weights, loading them onto GPUs, and warming up inference engines like vLLM. Minutes pass, response latency spikes, making your application unusable. You haggle with DevOps to overprovision capacity so your application remains reliable.