How it works

When a model does not fit in the memory of a single GPU, model parallelism distributes its layers or parameters across multiple GPUs. In pipeline parallelism, each GPU processes a stage of the network. In tensor parallelism, a single operation is split across GPUs.

Communication between GPUs is the bottleneck: data needs to be transferred between stages at each step. Frameworks like DeepSpeed and Megatron-LM implement these techniques in an optimized way.

Model parallelism is often combined with data parallelism (3D parallelism) to scale training of models with hundreds of billions of parameters.

Practical example

A 100 billion parameter model does not fit on an 80 GB GPU. With model parallelism across 4 GPUs, each one stores ~25 billion parameters. The forward pass passes data from one GPU to the next.

Texto
GPU 0: camadas 1-20
GPU 1: camadas 21-40
GPU 2: camadas 41-60
GPU 3: camadas 61-80

With DeepSpeed, the configuration is done via JSON:

JSON
{
  "tensor_parallel": {"size": 4},
  "pipeline_parallel": {"size": 2}
}

What it's for

Train and serve models that exceed the memory of a single GPU. It is the foundation of systems like Megatron-LM and DeepSpeed. It complements data parallelism in large scale training.