Jakob Edding

For half a dozen years, we’ve been running Kafka Streams applications at bakdata, constantly balancing throughput, stability, and operational overhead. A recurring pain has been frequent or slow rebalances that negatively impact the latency and throughput of Streams apps. While rebalancing is necessary to optimally distribute tasks - such as when members join or leave a group - frequent rebalances can be disruptive. Incremental cooperative rebalancing (Kafka 2.4+) reduces partition movement and even keeps processing running during rebalancing (Kafka 2.5) but does not fully alleviate the pain. With growing experience, we optimized configs of our Streams apps for specific app characteristics to avoid frequent and slow rebalances that stall processing. Such characteristics include high-latency record processing, application statefulness, or dynamic horizontal scaling. The new Streams rebalance protocol introduced in KIP-1071 makes assignment of Streams tasks a first-class citizen in the Kafka protocol. It promises to make rebalances less disruptive to the processing. As part of our migration efforts to the new protocol, we adjusted our established Streams configs where necessary.…

This talk presents a performance-tuned Apache Kafka pipeline for generating embeddings on large-scale text data streams. To store embeddings, our implementation supports various vector databases, making it highly adaptable to many applications. Text embeddings are fundamental for semantic search and recommendation, representing text in high-dimensional vector spaces for efficient similarity search using approximate k-nearest neighbors (kNN). By storing these embeddings and providing semantic search results given a query, vector databases are central to retrieval-augmented generation systems. We present our Kafka pipeline for continuously embedding texts to enable semantic search on live data. We demonstrate its end-to-end implementation while addressing key technical challenges: - First, the pipeline performs text chunking to adhere to the maximum input sequence length of the embedding model. We use an optimized overlapping text chunking strategy to ensure that context is maintained across chunks. - Using HuggingFace’s Text Embeddings Inference (TEI) toolkit in a lightweight, containerized GPU environment, we achieve efficient, scalable text embedding computation. TEI supports a…