Chase Thomas

Cboe's engineering team had a requirement to replace a legacy system that relied on proprietary wire protocols and end-of-day batch ETL. The goal was to provide internal teams with near real-time access to market data, requiring both high-ingest throughput and immediate data availability for queries. Using the new Snowpipe Streaming High-Performance Architecture, Cboe built a pipeline that directly ingests market data into Snowflake. The production workload achieved the following metrics: Ingest throughput: The pipeline processes over 100 TB of uncompressed data daily. Row velocity: The ingestion rate is over 190 billion rows per day.

Take an in-depth look at our next-generation real-time data ingestion service, designed to address the challenges of today and scale to the needs of tomorrow. We will share Snowflake's vision on streaming ingest, then dive into how we enhanced the current Snowpipe Streaming architecture, with multi-language support, ingest-time clustering and transformation capabilities. We’ll demo unified ingestion endpoints to deliver seamless, dynamic throughput — from trickle to high throughputs — without any manual reconfiguration. Whether you’re seeking more flexibility or building real-time analytics without performance bottlenecks, Snowpipe Streaming works for any scale.

A lot of enterprise customers use multiple accounts in their AWS environment and often have a central data processing account that serves as a hub for many other consumer accounts. They also need to have a cross-region disaster-recovery strategy in place.