AviSoori1x/makeMoE
From scratch implementation of a sparse mixture of experts language model inspired by Andrej Karpathy's makemore :)
This project helps machine learning engineers understand and build a 'sparse mixture of experts' language model from the ground up. It takes a dataset of text (like Shakespeare's writings) as input and outputs a trained model capable of generating new text in a similar style. It's intended for engineers or researchers working with language models who want to explore advanced architectures.
793 stars. No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher who wants to learn the inner workings of sparse mixture of experts (MoE) architectures for language models through a clear, from-scratch implementation.
Not ideal if you are a practitioner looking for a pre-trained, high-performance language model for immediate use in applications.
Stars
793
Forks
92
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Oct 30, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/AviSoori1x/makeMoE"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
EfficientMoE/MoE-Infinity
PyTorch library for cost-effective, fast and easy serving of MoE models.
raymin0223/mixture_of_recursions
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation...
thu-nics/MoA
[CoLM'25] The official implementation of the paper
jaisidhsingh/pytorch-mixtures
One-stop solutions for Mixture of Expert modules in PyTorch.
CASE-Lab-UMD/Unified-MoE-Compression
The official implementation of the paper "Towards Efficient Mixture of Experts: A Holistic Study...