Jose Manuel Cristobal

This is a story of a team who was at the verge of becoming a victim of his own success, with a massive adoption of a technology and the challenge to maintain a decent service quality, while keeping the infrastructure stable and reliable. Implementing multi-tenancy in Kafka is not too complex when the number of use cases sharing the cluster is low. A central team can operate the infrastructure, taking care of the heavy lifting and creating required assets on demand. This is true until adoption starts growing and the solution becomes a problem. You are a bottleneck and any service request piles up until an agent can resolve it, increasing resolution times and frustration at the same pace. Also, the amount of mistakes committed when you are doing everything by hand is very high, provoking toil and unexpected side effects and operational complexities. In this talk, we'll explain how we reverted the tendency implementing a non opinionated, vendor-agnostic self-service solution, delegating completely the responsibility to maintain assets to our stakeholders (topics, permissions, schemas, connectors) and reducing resolution times for any of these activities several orders of magnitude,…