Avishek Neupane

A streaming platform has two sides: application developers who want real-time data without becoming stream-processing experts, and platform engineers who must keep thousands of tenant pipelines isolated, observable, and—ideally—boring to operate. At Goldsky, we built a multi-tenant declarative streaming platform twice: first as Mirror on Flink, then as Turbo on Streamling, our DataFusion-based engine. The engine changed, but the goal did not: make streaming applications easy for developers to build and maintain, while keeping infrastructure and support manageable for a small platform team. This talk is a production case study of what we learned from rebuilding both the engine and the control plane. On the DevEx side, we built an end-to-end workflow around the declarative YAML: pre-submit validation, built-in OpenTelemetry, live inspect tooling to preview data as it flows through a pipeline, audit history of what changed between versions, and clearer error ownership—plus LLM-assisted root-cause hints and fix recommendations. Together, these features turn streaming from a black box into something developers can review, debug, and ship with confidence. On the platform side, we’ll…