Eliel Lima Oliveira

The day start with one problem: How to have content from a CMS system to reflect into multiple systems, especially in our global search (knauf.com) “right away”! That’s easy if you can wait (welcome back to 1920s). Today, milliseconds can mean the difference between a happy customer (in our case, editor) and one lost to frustration. Why is that impressive? ~200 editors working and ~3.3 millions of connections a day!!! Here is where the stream helps us with (near) real-time data processing. Our system efficiently integrates Contentful CMS, Confluent Kafka, and Apache Flink to create a real-time data pipeline that captures, processes, and analyzes content updates with lightning-fast speed and precision. Our goal was clear: create a reactive data processing system that ensures any content changes in the Contentful CMS are immediately reflected in other applications, especially our global search functionality via Algolia indexes. We wanted users to have access to the most up-to-date information, no delays allowed. How did we achieve this? Leveraging Contentful webhooks to detect content modifications and trigger a Spring Boot application, which acts as a Kafka producer, transforming…