NVIDIA DLI - Model Parallelism: Building and Deploying Large Neural Networks
Train neural networks across multiple servers. Use techniques such as activation checkpointing, gradient accumulation, and various forms of model parallelism to overcome the challenges associated with large-model memory footprint. Capture and understand training performance characteristics to optimize model architecture. Deploy very large multi-GPU models to production using NVIDIA Triton™ Inference Server.