kyegomez/SimplifiedTransformers
SimplifiedTransformer simplifies transformer block without affecting training. Skip connections, projection parameters, sequential sub-blocks, and normalization layers are removed. Experimental results confirm similar training speed and performance.
This project offers a simplified approach to building and training AI models, specifically those based on transformer architectures. It helps machine learning engineers and researchers by taking standard transformer model configurations and reducing their complexity. The output is a more streamlined and efficient transformer model that maintains training speed and performance while consuming fewer computational resources.
Use this if you are a machine learning engineer or researcher looking to experiment with more efficient and stable transformer architectures for your AI models.
Not ideal if you need to strictly adhere to traditional transformer block designs or are looking for a pre-trained model rather than a simplified architecture.
Stars
15
Forks
4
Language
Python
License
MIT
Category
Last pushed
Feb 06, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/kyegomez/SimplifiedTransformers"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
huggingface/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in...
kyegomez/LongNet
Implementation of plug in and play Attention from "LongNet: Scaling Transformers to 1,000,000,000 Tokens"
pbloem/former
Simple transformer implementation from scratch in pytorch. (archival, latest version on codeberg)
NVIDIA/FasterTransformer
Transformer related optimization, including BERT, GPT
ARM-software/keyword-transformer
Official implementation of the Keyword Transformer: https://arxiv.org/abs/2104.00769