Ademir Spahic

This presentation details how our platform enablement team developed a centralized Kafka library, empowering developers to build applications with ease. Faced with inconsistent Kafka processing approaches across teams, we created a common library, inspired by the multi-threaded consumer approach described in this Confluent blog post: https://www.confluent.io/blog/kafka-consumer-multi-threaded-messaging (re-implemented in Kotlin). We'll share our challenges, successes, and the current state of this library, now used in over 20 services. Initially, varying team approaches to Kafka processing led to inconsistencies and duplicated effort. Our team recognized the need for standardization. Internal library simplifies Kafka development, promotes best practices, and centralizes key functionalities. It wraps the kafka-clients library, offering simple interfaces for building Kafka consumers and producers that integrate with our Confluent clusters, schema registry, and Avro serialization. Core feature is its multi-threaded consumer implementation, enabling efficient consumption from multiple partitions. We'll share the technical hurdles we encountered during development, discussing our…