Hamid Shojanazeri

Here, we talk about large model inference with Torchserve, using PiPPy, Tensor Parallel, challenges of distributed inference and available solutions. Discuss the features that Torchserve provide today for serving LLMs in production today.

Amazon EC2 Trn1 instances, powered by AWS Trainium chips, are purpose-built for high-performance, deep-learning training and offer up to 50 percent cost-to-train savings over equivalent GPU-based instances. In this session, learn about AWS Trainium and Trn1 innovations, the AWS collaboration with PyTorch and Hugging Face, and the successes users have seen.

Attend this chalk talk with Meta PyTorch and AWS to learn about distributed training, accelerated inference, bin-packing models, and scaling with PyTorch on AWS. In addition, hear about relevant large-companies that use AWS infrastructure and services for distributed training and to serve thousands of models with low latency and cost.

By using PyTorch in Amazon SageMaker, you have a flexible deep learning framework combined with a fully managed machine learning solution that allows you to transition seamlessly from research prototyping to production deployment. In this session, hear from the PyTorch team on the latest features and library releases.