Chad Harris

Most Kafka troubleshooting still means clicking through a web console or piecing together kafka-*.sh commands. Kpow, the Kafka monitoring and management platform, now ships a single binary that brings it all to the terminal: a CLI that makes any Kafka task one command, a real-time Terminal UI that mirrors the web console, and Agent Skills that let Claude Code, Codex and GitHub Copilot do the digging for you.

Most teams can draw their data architecture on a whiteboard but can't answer a basic governance question: when a value in a dataset looks wrong, where did it come from, and what downstream tables and models does it affect? In a stack spanning Kafka Connect, Flink, Spark, and Iceberg, lineage breaks at every engine boundary. Tracing a bad value backward through four engines in production can take weeks. Catching the break early, costs minutes. This session is about closing that gap. As data becomes more relevant for analytics and AI, that lack of visibility becomes a trust and compliance problem, not only an operational one. This session demonstrates how to build a lineage graph across all four engines using OpenLineage and Marquez, all open source. We begin by demonstrating how to capture lineage from the streaming source without modifying data records. We then fan a single Kafka topic out across parallel pipelines: an S3 sink connector for raw archival, a real-time Flink DataStream job, a Flink SQL job writing to Iceberg, and a downstream Spark batch job that processes the Iceberg tables. We’ll show where the stack yields true column-level lineage versus dataset-level only, using…