xrsrke/pipegoose
Large scale 4D parallelism pre-training for 🤗 transformers in Mixture of Experts *(still work in progress)*
This project helps machine learning engineers efficiently train large-scale transformer models, especially multi-modal Mixture of Experts (MoE) models. It takes existing Hugging Face transformer models and training scripts as input, and outputs significantly faster distributed model training, leveraging techniques like data and tensor parallelism. The primary users are ML engineers and researchers working with state-of-the-art large language models.
No commits in the last 6 months.
Use this if you are pre-training large transformer-based multi-modal Mixture of Experts models and need to scale training across multiple GPUs or machines efficiently.
Not ideal if you are working with small models or non-transformer architectures, or if you do not have access to multiple GPUs for distributed training.
Stars
87
Forks
19
Language
Python
License
MIT
Category
Last pushed
Dec 14, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/xrsrke/pipegoose"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
openvinotoolkit/nncf
Neural Network Compression Framework for enhanced OpenVINOâ„¢ inference
huggingface/optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers...
NVIDIA/Megatron-LM
Ongoing research training transformer models at scale
huggingface/optimum-intel
🤗 Optimum Intel: Accelerate inference with Intel optimization tools
eole-nlp/eole
Open language modeling toolkit based on PyTorch