Manu Manjunath

At Atlassian, we have numerous inter-connected services whose shards are deployed on data centers across the globe. These services together participate in a complex stream-based ‘data migration’ workflow. To enable this, within the Platform org, we built a scalable, resilient and globally consistent Orchestrator. This Orchestrator leverages Kafka’s State store, Kafka streams and Kafka Connect. It provides a “service mesh” equivalent for Kafka-integrated services, enabling seamless coordination and communication between different “steps” of the workflow. The architecture allows different shards and nodes of services to enter and exit the service mesh, specify tenants, allocate usage-based quota to callers and so on. Central to our solution are 5 things: 1️⃣ A State store based context management that’s globally consistent 2️⃣ An SDK that services can readily integrate. This SDK seamlessly abstracts out Kafka for application developers. 3️⃣ A dynamic registry of services which not only catalogs the available services but also maintains an up-to-date map of service deployments across data centers, their health and their usage. 4️⃣ The orchestrator's intelligent routing algorithms…