Mateusz Charytoniuk

Last year, I shared Paddler, an open-source LLM load balancer. A year of community feedback and building Poet (a static site generator with AI features) on top of it taught me what actually matters when self-hosting LLMs.

Paddler is an open-source llama.cpp load balancer designed to address unique challenges that Large Language Models pose. Typical balancing algorithms like round-robin or least-connections are not the most efficient approaches.