kyegomez/Mixture-of-MQA
An implementation of a switch transformer like Multi-query attention model
This is an advanced neural network architecture designed for developers building large-scale AI models. It processes sequences of data, such as text or other sequential inputs, to produce learned representations or predictions, improving efficiency and scalability. AI/ML engineers and researchers who are working on complex natural language processing or sequence modeling tasks would find this useful.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher developing transformer-based models and need to process very long sequences more efficiently.
Not ideal if you are a data scientist looking for an off-the-shelf model for immediate use or if you are not comfortable with deep learning model architecture.
Stars
8
Forks
—
Language
Python
License
MIT
Category
Last pushed
Feb 20, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/kyegomez/Mixture-of-MQA"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
lucidrains/x-transformers
A concise but complete full-attention transformer with a set of promising experimental features...
kanishkamisra/minicons
Utility for behavioral and representational analyses of Language Models
lucidrains/simple-hierarchical-transformer
Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
lucidrains/dreamer4
Implementation of Danijar's latest iteration for his Dreamer line of work
Nicolepcx/Transformers-in-Action
This is the corresponding code for the book Transformers in Action