David Yun

When operating 6 platforms with a team of 3 and actively developing the 7th, near-zero ops is no longer a luxury: it is a necessity. In this talk, we deep dive into Dropbox's standard event processing platform: EventBus. Today EventBus processes hundreds of billions of events per day across 50+ use cases, serving as the backbone for data pipelines, search indexing, and AI ingestion company-wide. The platform abstracts away the complexities of operating Kafka consumers and transforms Kafka consumption into familiar stateless remote procedure calls focusing on business logic. Users of the platform are able to self-onboard use cases that consume millions of messages per second, and everyone still gets to sleep at night. EventBus carries the golden standard in platform reliability at Dropbox, with 0 SEVs incurred since GA >1 year ago despite having 0 resources dedicated for KTLO. This talk covers advanced topics such as going beyond partition-level parallelism while retaining message key ordering, how we deal with hot partitions, protecting tenants from noisy neighbor issues, and how we are able to operate cost effectively on top of being operationally light. Whether you already run…

The core theme of the talk is to build upon existing parallel consumer work that allows for message key level parallelism while retaining ordering guarantees without provisioning additional partitions. We extend the principles by decomposing messages into smaller sub-messages, allowing for messages with the same key to be processed simultaneously while still retaining ordering guarantees. The sub-message parallel consumer allows for faster time to market, at lower latency and cost versus existing methods presented in literature. This talk walks through a very real scenario we experienced when scaling up Dropbox Dash (AI assistant): the deadline to onboard a customer is tomorrow morning, but the backlog needs 2 weeks to finish processing due to poor key choice in early stages of development leading to every message ending up on the same partition. I will recap/summarize existing topics to set the context: 1. Conventional Kafka parallelism (partition level) 2. Message key level parallelism using techniques discussed for the Confluent Parallel Consumer I will also present the additional constraints that we face in our own system: 1. It is not feasible to change the producer quickly…