Matthew Rocklin

Pangeo is an ecosystem of OSS tools like Xarray, Dask, Jupyter, Zarr, and others used in large scale geospatial analysis. This talk tells the history of how these tools came together and where they're going today.

Distributed systems are neat to demo, but hard to use in reality. This talk goes through lessons learned running 100,000s of Dask clusters and 1,000,000,000s of Python functions for users in critical production settings across many companies and research groups.

Spark, Dask, DuckDB, and Polars are popular dataframe tools used at large scale. We benchmark these across a variety of scales (10 GiB to 10 TiB) on both local and cloud architectures with the standard TPC-H benchmark.

We're in a new era of dataframe development. Libraries like Arrow, Polars, DuckDB, Vaex, Modin, and others stretch the bounds of performance on what we think can be done with tabular data in Python.

Learn how to run the Python data science ecosystem in parallel with Dask in this hands-on tutorial. Dask is an open source library for parallel computing in Python with deep integration with common Python libraries like numpy, pandas, xgboost, pytorch, xarray, and of course Jupyter.

This talk discusses the challenges and options to scale Python with Dask on distributed hardware. We particularly focus on how Dask gets deployed on cluster resource managers like Kubernetes, Yarn, and the cloud today.