Keita Watanabe

Extracting useful throughput from GPU clusters is a discipline: profiling kernels, tuning collectives, keeping the GPUs fed, and handling the failures that grow with scale.

Disaggregated inference separates prefill and decode onto independent GPU pools, eliminating the interference that lets one long prompt stall active generations and letting each phase scale independently.

At scale, GPU clusters fail in ways that single-node debugging does not reveal: NCCL timeouts that are actually storage problems, DataLoader forks that silently hang collectives, and GPUs that pass health checks yet corrupt results.

In this session, we will dive deep into advanced strategies for distributed training, including parallelism and communications collectives, storage, and networking.

Amazon SageMaker HyperPod removes the undifferentiated heavy lifting involved in building and optimizing machine learning (ML) infrastructure for training foundation models (FMs), reducing training time by up to 40%.

Unlock the full potential of large language models (LLMs) on Amazon EKS by optimizing inference performance and cost efficiency. This chalk talk provides practical guidance on deploying and scaling PyTorch-based LLMs on EKS using AWS Inferentia, Karpenter, and KServe.

This workshop explores the end-to-end workflow of managing foundation models (FMs) on Amazon SageMaker HyperPod. The discussion covers four critical aspects: Parameter-Efficient Fine-Tuning (PEFT), model deployment, and serving.

Amazon EC2 offers a broad set of accelerators in the cloud for machine learning. Whether you are using instances powered by NVIDIA GPUs, AWS Trainium, or AWS Inferentia, you may wonder how to manage the software stack.

Generative AI and large language models (LLMs) inspired many organizations to reimagine the experiences they are building for their customers. As these sophisticated LLMs are integrated into more applications, developers are challenged with model serving for high-volume deployments while meeting performance targets.

Distributed training can be daunting for nonexperts because it requires a deep understanding of your architecture, software stack, and runtime configurations to be efficient.

Generative AI and large language models (LLMs) are rapidly disrupting workflows. Inference architectures for these large-scale models need to be carefully constructed to achieve optimal cost performance.