Sage Pierce

At Indeed, we help people get jobs. Our team supports this mission by ingesting data about employers’ hiring needs, enabling them to manage this data, and transforming it into searchable job advertisements—commonly known as job posts. The many different asynchronous processes involved in solving these and related challenges can be complex, widely diverse, and in many cases, I/O dependent. Consequently, our team implements a wide swath of asynchronous use cases on top of Kafka. This includes conventional idiomatic use cases, but just as important, use cases that are historically tricky on top of Kafka’s architecture. Such use cases involve “treating Kafka like a queue” and implementing long-running (i.e. minutes, hours) per-record consumption tasks. This talk will dive into how we at Indeed tackle these challenges with Kafka and Atleon, an OSS project, as well as explore the following topics: - The difficulties involved with “treating Kafka like a queue” - How “share groups”, introduced by KIP-932, may (or may not) address needs for higher consumer concurrency - What reliable queue-like consumption without loss of ordering looks like with vanilla Kafka consumer groups -…

At Indeed, we help people get jobs. Our team supports this mission by ingesting data about employers’ hiring needs, enabling them to manage this data, and transforming it into searchable job advertisements—commonly known as job posts. Creating job posts is a complex and I/O-bound process, requiring enrichment from multiple bounded contexts. Adding to this complexity, many downstream systems must be notified in real-time when any part of a job post’s data changes. To meet these demands, we implemented a system that produces and maintains an Event Driven View (EDV)—a materialized, denormalized representation of job posts that stays up-to-date as changes occur across the business. This view is powered by a novel Ingest–Materialize–Index (IMI) stream topology, which enables us to scale processing while preserving strong observability and reliability guarantees. This talk will explore how we build and operate EDVs using the IMI architecture: - We’ll break down each IMI stage and show how clean separation of concerns enables performance and observability. - We’ll show how we use micro-batching, structured concurrency, and I/O pipelining to manage I/O-bound enrichment at scale. - We’ll…