RJ Atwal

Data today lives in two worlds: at rest, where you can query it, and in motion, streaming through systems like Apache Kafka®, built for throughput and ordering, not for querying. Bridging them means copying data out of the stream through connectors and stream processors into a warehouse or OLAP database, and every hop adds latency, a new copy, and one more thing that can break. This does not work for agents. Agents decide what to ask at runtime, chain dozens of retrievals per task, and act on what they read, so latency stacks up and stale data leads to wrong actions and consequences. Now lightning queries close that gap: a stream-native OLAP engine that queries streaming data directly, without ever copying it out. In this talk, you'll walk away with a working model for designing a stream-native query engine, one that holds fresh log records and historical columnar data in one place and answers queries consistently across both, with no gap between what just happened and what you can query. You'll see how lightning queries serve ad-hoc filters and lookups without pre-built cubes while staying low-latency, and the API design choices that made that possible. Most importantly, you’ll…