Thomas Thornton

Slack's previous batch data replication system cost millions of dollars per year and introduced up to 48 hours of latency. To solve both problems, we replaced it with a real-time Change Data Capture (CDC) pipeline built on Debezium, Vitess, Apache Kafka, and Apache Iceberg. That migration was just the beginning. What followed was four years of operating, scaling, and hardening these pipelines across thousands of shards and petabytes of data in production. This talk is a lessons-learned walkthrough of what actually breaks when you run CDC and Kafka Connect at scale. We'll briefly cover the architecture, then focus on the operational challenges that apply to any streaming deployment: - Reliability: How an unexpected interaction caused out-of-memory failures in production, and the delivery pattern we adopted to prevent unbounded memory growth. - Data loss prevention: A subtle edge case in Kafka Connect silently dropped records for all connectors. How we detected & fixed it and what to monitor for your own pipelines. - Load balancing: Why static connection assignment leaves throughput on the table, and how periodic stream rotation with resource-aware routing distributes work more…

Slack was burning cash on batch data replication, with full-table restores causing multi-day latency. To slash both costs and lag, we overhauled our data infrastructure—replacing batch jobs with Change Data Capture (CDC) streams powered by Debezium, Vitess, and Kafka Connect. We scaled to thousands of shards and streamed petabytes of data. This talk focuses on the open source contributions we made to build scalable, maintainable, and reliable CDC infrastructure at Slack. We'll show how we cut snapshotting time—from weeks to hours—for our largest table, half a petabyte in size and spread across hundreds of shards. You’ll learn how to apply our optimizations in Debezium, tune Kafka Connect configs, and maximize throughput. We’ll also cover how we tackled one of streaming’s most elusive challenges: detecting accurate time windows. By contributing a binlog event watermarking system to Vitess and Debezium, we made it possible to ensure correctness in a distributed system with variable lag. Finally, we’ll show you how to detect & prevent data loss in your own pipelines by applying the fixes we contributed to Kafka Connect and Debezium, which addressed subtle edge cases we uncovered in…