Satish Duggana

Abstract: At Uber, the EVA platform that drives substantial advancements in our real-time analytics capabilities, empowering various business use cases across marketing, engineering, data science, and operations and internal use cases around metrics, logs & query analytics. The platform features Apache Kafka for realtime data transport, Apache Flink for stream processing, Spark for batch processing, HDFS for deep storage needs, and Apache Pinot as the core analytics engine. Additionally, it features internal service Neutrion for Presto-like queries on Pinot and metadata service for dataset management. Core Theme As part of the talk, we cover the matured architecture for realtime analytics ecosystem powering Uber’s usecases that serve up to 10s of thousands of queries/sec, several million writes/sec and host up to tens of Petabytes of Pinot datasets. We also cover two critical business and observability usecase. Technical Depth 1. Realtime processing and ingestion using AthenaX(SQL based transformation on Flink), Flink and Kafka to provide analytics on realtime data. 2. Realtime Analytics powered by Apache Pinot to serve analytics at high QPS with sub-second latency 3. Disaster…

Apache Kafka introduced tiered storage(KIP-405) feature that provides isolation of storage and processing in a broker. It allows the segments to be available based on configurable remote retention based on time and size.