Scott Schenkein

Delta Lake and Iceberg excel at large-scale analytics, but they are not optimized for sub-second point lookups or full-text search. In operational workloads — such as security analytics, log investigation and telemetry — retrieval queries that require full scans or high-cardinality filtering can take minutes to hours, when seconds or milliseconds are required. I built IndexTables, an embedded, Tantivy-based indexing layer that runs directly inside Spark executors, enabling up to 1000× faster query performance while preserving the Lakehouse model. This session explores the architecture: object-storage-hosted indexes with ACID transactions, millisecond-latency aggregations over billions of rows, native time-series bucketing for efficient GROUP BY analytics and NVMe-backed caching with proactive pre-warming. Attendees will learn how Capital One is using IndexTables to complement Delta and Iceberg, creating lakehouse architectures that support both analytical and retrieval-heavy workloads.

The Delta Lake architecture promises to provide a single, highly functional, and high-scale copy of data that can be leveraged by a variety of tools to satisfy a broad range of use cases. To date, most use cases have focused on interactive data warehousing, ETL, model training, and streaming. Real-time access is generally delegated to costly and sometimes difficult-to-scale NoSQL, indexed storage, and domain-specific specialty solutions, which provide limited functionality compared to Spark on Delta Lake. In this session, we will explore the Delta data-skipping and optimization model and discuss how Capital One leveraged it along with Databricks photon and Spark Connect to implement a real-time web application backend. We’ll share how we built a highly-functional and performant security information and event management user experience (SIEM UX) that is cost effective.