MoonshotAI/MoBA
MoBA: Mixture of Block Attention for Long-Context LLMs
This project helps AI engineers and researchers improve how large language models (LLMs) handle very long texts. It takes an existing LLM, trains it with a new attention mechanism, and produces a more efficient model that can process much longer inputs without a significant performance hit. This is for professionals building and deploying advanced LLMs who need to scale their models for extensive context understanding.
2,076 stars. No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher developing large language models and need to efficiently scale their ability to understand and process extremely long documents or conversations.
Not ideal if you are looking for a ready-to-use solution to immediately apply to an existing, pre-trained LLM without additional training, as MoBA requires continued training to achieve its benefits.
Stars
2,076
Forks
136
Language
Python
License
MIT
Category
Last pushed
Apr 03, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/MoonshotAI/MoBA"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
EfficientMoE/MoE-Infinity
PyTorch library for cost-effective, fast and easy serving of MoE models.
raymin0223/mixture_of_recursions
Mixture-of-Recursions: Learning Dynamic Recursive Depths for Adaptive Token-Level Computation...
AviSoori1x/makeMoE
From scratch implementation of a sparse mixture of experts language model inspired by Andrej...
thu-nics/MoA
[CoLM'25] The official implementation of the paper
jaisidhsingh/pytorch-mixtures
One-stop solutions for Mixture of Expert modules in PyTorch.