Philip Bontrager

Long gone are the days where the entire lifecycle of a parameter consists of being initialized on a GPU, updated on the same GPU, and then finally saved to your local SSD. As models and training pipelines scale in size and complexity, a single parameter will get sharded, resharded, streamed across...

PyTorch Libraries provide building blocks (data processing transforms, modeling components, loss functions, etc.) on top of PyTorch as well as examples and tutorials on how to use these building blocks for training SoTA Models. In this talk, we’ll provide insights into ongoing work to accelerate...