Sergey Zyrianov

As organizations race to deploy Large Language Models, a critical infrastructure challenge emerges: how do you run a 70B+ parameter model when it exceeds your available hardware? The standard answer—buy bigger GPUs—isn't always feasible due to cost, availability, or deployment constraints. This talk presents an alternative: pipeline-parallel inference using Apache Kafka as the coordination layer, turning existing distributed infrastructure into an LLM serving platform. The core architecture partitions transformer layers across multiple nodes, where each node holds a fraction of the model weights. Inference flows as a streaming pipeline: Node 1 processes the first N layers and publishes the resulting hidden state tensor to a Kafka topic. Node 2 consumes this message, processes the next N layers, and publishes downstream—continuing until the final node produces the output token. This pattern transforms Kafka from a data streaming platform into a distributed compute orchestrator for AI workloads. This is experimental work exploring the boundaries of what's possible. CPU-only inference introduces significant latency—seconds per token rather than milliseconds—making it impractical for…

Timers are a cornerstone of any software system, yet traditional implementations often rely on in-memory solutions or RDBMS dependencies. In this talk, I’ll present a unique approach that leverages Kafka alone to power timer functionality—eliminating the need for RDBMS and embracing a distributed architecture. Using Kafka Streams, I’ll demonstrate how to efficiently schedule delayed work at web scale, enabling resilient and scalable microservices.