Jeffrey Wan

These sessions give participants an opportunity to meet some of the developers leading PyTorch to foster collaboration, gather feedback, and inspire contributions and collaboration . PyTorch core modules (e.g. torch.autograd, torch.optim, torch.nn) form the foundation for most AI research and...

A Case Study of Torch.compile-induced Timeouts in Distributed - Simon Fan & Brian Hirsh, Meta Activation Checkpoint 2: A New Graph-capture-based Activation Checkpoint - Jeffrey Wan, Meta Automatic Parallelism in PyTorch - Will Constable & Francisco Massa, Meta Breaking Less: Tackling Graph Breaks...

Activation checkpointing is a commonly used technique to reduce memory usage during model training by reducing the number of activations saved for backward. Instead of keeping tensors needed for backward alive until they are used in gradient computation during backward, those tensors are recomputed...