Driss Guessous

FlexAttention democratized attention research by letting researchers prototype custom attention variants in PyTorch without hand-written CUDA. Over 1,000 repos have adopted it, and dozens of papers cite it. But flexibility came at a cost: FlexAttention achieved only ~60% of FlashAttention-3's...

These sessions give participants an opportunity to meet some of the developers leading PyTorch to foster collaboration, gather feedback, and inspire contributions and collaboration . PyTorch core modules (e.g. torch.autograd, torch.optim, torch.nn) form the foundation for most AI research and...

FlexAttention is a novel compiler-driven programming model that allows implementing the majority of attention variants in a few lines of idiomatic PyTorch code. Since its release in PyTorch 2.5.0, many ML researchers have utilized it to customize their attention kernels without writing kernel code....

The introduction of MicroScaling formats (MX) by the Open Compute Project has led to native hardware support by major hardware vendors. This talk will explore the end-to-end workflow of utilizing MX formats on modern hardware, highlighting the performance benefits and challenges associated with...