Nicoleta Lazar

Real-time systems don’t just process data — they accumulate it. As datasets grow, teams must balance low-latency “hot” data with cost-efficient “cold” storage, without sacrificing correctness or performance. In this talk, we’ll share a practical architecture combining Apache Flink for streaming, Apache Spark for historical processing, Apache Iceberg as the cold data lake layer, StarRocks as a unified OLAP and federation engine, and Airflow for orchestration. We’ll explore how to tier data effectively, age it safely, handle late events, deduplicate across batch and streaming boundaries, and maintain strong query performance across hot and cold layers. We’ll focus on real-world challenges, trade-offs, and architectural patterns for building scalable streaming-first data platforms.

As analytics ecosystems grow more diverse, organisations increasingly need to query data across warehouses, data lakes, and operational systems without excessive movement or duplication. Query federation has become essential by enabling unified SQL access and intelligent pushdown into heterogeneous sources.