Blake Shaw

The exponential growth of data demands a paradigm shift in how we discover, create, and evolve data products. This presentation introduces a novel Agentic Data Mesh architecture where AI agents take on proactive roles in the data product development lifecycle. Unlike traditional approaches, our agents don't just process data; they think about it. We envision a system where specialized AI agents, empowered by real-time streaming data from Confluent, proactively identify opportunities for new data products based on business needs, data patterns, and existing data assets. These Discovery Agents will propose data product definitions to human subject matter experts for approval, acting as intelligent co-creators. Upon approval, Creation Agents will leverage Confluent Tableflow to seamlessly transform Kafka topics into managed Apache Iceberg tables, ensuring schema evolution, ACID compliance, and time-travel capabilities. This automated creation extends to higher-order data products, where agents autonomously combine, refine, and process existing topics with Apache Flink, continually enriching the data landscape. Furthermore, Analysis Agents will exploit the robust capabilities of…