Spico197/MoE-SFT
🍼 Official implementation of Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts
This project helps machine learning engineers and researchers optimize how they train Mixture-of-Experts (MoE) models. It takes various instruction datasets (like creative writing, coding, or math problems) and intelligently adjusts which data the model sees during training. The output is a more efficient and better-performing MoE model for a range of tasks.
No commits in the last 6 months.
Use this if you are training Mixture-of-Experts models and want to improve their performance and efficiency by dynamically managing your training data.
Not ideal if you are working with standard, non-MoE large language models or do not have access to diverse, instruction-tuned datasets.
Stars
41
Forks
—
Language
Python
License
Apache-2.0
Category
Last pushed
Sep 29, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Spico197/MoE-SFT"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
EfficientMoE/MoE-Infinity
PyTorch library for cost-effective, fast and easy serving of MoE models.
raymin0223/mixture_of_recursions
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation...
AviSoori1x/makeMoE
From scratch implementation of a sparse mixture of experts language model inspired by Andrej...
thu-nics/MoA
[CoLM'25] The official implementation of the paper
jaisidhsingh/pytorch-mixtures
One-stop solutions for Mixture of Expert modules in PyTorch.