K R Avinash Upadhyaya

AI-powered devices are generating more data at the edge, but sending everything to the cloud is expensive and difficult when connectivity is unreliable. This session explores how Apache Kafka® can run closer to these devices, either as a gateway or an embedded server. We’ll look at single-node setups designed for unreliable networks, support for protocols like MQTT and HTTP, and ways to process and reduce data locally before sending it upstream. The approach is simple: stream, buffer, aggregate, and forward. We’ll also cover how to connect multiple edge locations to a central system.

REST-based Request-response APIs are the lifeblood of most architectures. API transactions, specifically involving the CUDs (Creates, Updates, Deletes) are great sources of events and/or change data. This is also true of various RPC style APIs. However, because of the client-server nature, such events need to be published by the service, using an "outboxing" mechanism and subsequently disseminated to pub-sub event broker, most often Apache Kafka. This involves code changes to the application and in addition, a more expansive solution like CDC (change-data-capture) to collect those changes from the outbox table. In this session, we will present an alternative pattern that leverages standard API proxies / gateway solutions to process and push events to Kafka, as they occur in the API data plane at real-time. We will address the following important questions during this Show Me How session: 1. How to source event data from APIs (and API gateways) such as Kong and Istio 2. How to synthesize event data into the desired form using Kafka Streams 3. How to source change feeds against aggregates using Kafka Streams state stores 4. Considerations and tradeoffs for this architecture 5. Key…