Paul Brebner

Apache Kafka's tiered storage is not just a new feature but a major architectural shift that enables virtually unlimited storage. Traditionally designed for fast, high-throughput real-time streaming, Kafka now also supports more extensive data retention and replay capabilities which support a variety of use cases.

Imagine having a powerful, customizable model that brings the end-to-end flow of records through your Kafka applications and clusters to life. Picture a tool that allows you to swiftly and affordably understand and predict the performance, scalability, and resource demands of your entire system.

Over the last 7 years, I’ve been developing, scaling and benchmarking realistic Kafka demonstration applications running on Instaclustr’s managed Kafka service. Over this period, the number and size of Kafka clusters we manage have increased, and we have developed extensive experience with Kafka operational performance.

When I started as the Instaclustr Technology Evangelist 7 years ago, I already had a background in computer science R&D and thought I knew a few things about architecting complex distributed systems. But it was still challenging to learn multiple new Apache (and other) Big Data technologies and build and scale realistic demonstration applications for domains such as IoT/logistics, fintech, anomaly

In the 1st part of this talk, we'll build a real-time Drone Delivery demonstration application using a combination of two open-source technologies: Uber’s Cadence (for stateful, scheduled, long-running workflows), and Apache Kafka (for fast streaming).

Apache Kafka, Apache Cassandra and Kubernetes are open source big data technologies enabling applications and business operations to scale massively and rapidly.