Sandon Jacobs

Would you build a REST API and make users guess the request and response formats? - What's the domain model? - Name the operation… POST? GET? PUT? PATCH? - What does this error code even mean? - Is the documentation - if it even exists - accurate and relevant? Yet that's exactly what many data streaming deployments do. Apache Kafka's flexible, unopinionated serialization is a huge strength, but it's also a potential risk. Even if your event streams carry structured data, enforcing that structure, managing its evolution, and validating its contents aren't hard requirements. This leaves consumer applications preprocessing every event - even ones with no business value. Designing structured events with a schema (Apache Avro and Google Protobuf) models your business domain and promotes safe evolution practices. Schema-aware serializers enforce event structure before it's written to the stream, while consumers leverage those shared schemas to deserialize on the other end. Build-time validation catches schema problems before code ever reaches review - one last line of defense against poison data. Through all of it, the goal is the same: treat your Kafka topics with the same rigor you'd…

Your shiny, new, cutting-edge RAG microservice is only as smart as its context. And if that context is refreshed by a slow, batch-driven job, your AI is essentially answering today’s critical questions by consulting yesterday’s equivalent of a stale newspaper. It’s time to transition your RAG architecture from batch dependence to streaming certainty. Let’s discuss a “streams-first” approach to building data pipelines with fresh context. We’re using Apache Kafka and Apache Flink to build the always-on knowledge backbone your RAG microservices deserve. We’ll focus on the foundational engineering practices that guarantee reliability and access to real-time data: * Kafka as the data substrate: Data streams based on a fault-tolerant, high-throughput source of truth to capture every critical change across your organization. * Flink’s Real-Time Prep: Leveraging Flink for stateless transformation, stateful contextual enrichment and streamlined chunking—performing the heavy lifting as data arrives. * Production-Grade Guardrails: Implementing crucial patterns like Exactly-Once Semantics (EOS) for data consistency and establishing a Dead Letter Queue (DLQ) strategy for reliable error…

The “traditional” consumer group coordination in Apache Kafka assigns each partition of a topic to a member of a consumer group, providing a powerful combination of ordering and scalability. Sometimes ordering is not of the essence, and we would rather treat events as individual units of work. Enter KIP-932 - aka “Queues for Kafka” - enabling multiple consumers in the group to process from the same topic-partition. Every time I have spoken on this topic, I get the same questions: What are some use cases and when do I use it? So let’s take some time to identify and explore a couple of use cases. We’ll walk through code samples for these scenarios and how we can validate the behavior. When we wrap up, you’ll have a better idea of how and where to get started using queues the Kafka way.

Data streaming engineers need tooling to efficiently provision, maintain, and evolve the data stream platform. The Confluent Terraform Provider does just that, providing human-readable infrastructure-as-code to build a Confluent Cloud environment in a matter of minutes. In this session, we’ll start from a blank canvas and create a new environment - complete with an Apache KafkaⓇ cluster, stream governance, and processing with Flink. Next we’ll create Kafka topics, define data contracts and determine how to transform our input data. We won’t forget about security and access controls - so let’s create service accounts with the necessary roles and permissions. Finally, we’ll set it all in motion by streaming events into Kafka and querying the output of our new data pipeline. When we’re done, you’ll have the tools needed to build and maintain your data streaming platform. Let’s do this!