Model parallelism
4 methods in the atlas attack this one problem. They are rivals: each wins something the others do not.
Phrasings that mean this problem
Model parallelism
llm-training-alignment
- Tensor parallelismRow-column weight shardingstandardllm-training-alignment
- Pipeline parallelismMicro-batch bubble reductionstandardllm-training-alignment
- Sequence parallelismActivation sharding along tokensspecialistllm-training-alignment
- Expert parallelismToken routing across devicesspecialistllm-training-alignment