Vahid Hashemian

Cross-cluster Kafka replication is one of the most CPU- and network-expensive workloads in a large streaming platform. Apache Kafka's built-in MirrorMaker decompresses every batch, deserializes every record, then re-batches and re-compresses on the destination - burning cycles to reproduce bytes that already existed on the source. At Pinterest's scale, those cycles added up to entire fleets of machines doing nothing but encoding and decoding. For years, we ran a "shallow" MirrorMaker that forwarded entire compressed RecordBatches verbatim, preserving compression, timestamps, and CRCs, and only rewriting the few header fields that must change on the destination. The catch? It required substantial changes to Kafka broker internals. Every Kafka upgrade meant weeks of re-porting a moving target. Enter NeoShallowMirror, our new shallow MirrorMaker. It tells Kafka a very careful lie: that it is producing and consuming individual records one at a time. In reality, whole compressed RecordBatches are shuttled from source to destination untouched, and Kafka's internal accounting is kept perfectly consistent with the fiction. The result: we cut MirrorMaker CPU by an order of magnitude and…