Swastik Roy

Every time we rank an LLM, we're building a model of human preference — yet for most queries there's no ground truth, only a person's sense of which answer is better, shifting with expertise, culture, and expectations.

Benchmarks drive LLM development decisions, yet their signals are fundamentally fragile due to theoretical limitations critical at scale. Current evaluation spans pointwise metrics, pairwise preferences (Arena, MT-Bench), and agentic evaluations, extending to multi-modal models and autonomous agents.