liangyuwang/Tiny-Megatron
Tiny-Megatron, a minimalistic re-implementation of the Megatron library
This project helps machine learning engineers and researchers understand and implement distributed training strategies for large language models. It takes a PyTorch model and an HPC cluster configuration as input, and outputs a functionally identical model that can be trained efficiently across multiple GPUs or nodes. It's designed for those learning how to scale deep learning models for faster training or to fit larger models into memory.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher looking to learn about or implement tensor, data, or 2D hybrid parallelism strategies for training large language models in PyTorch.
Not ideal if you need a production-ready library with advanced features like pipeline parallelism or optimizer state sharding, or if you are not comfortable with PyTorch and distributed training concepts.
Stars
23
Forks
3
Language
Python
License
Apache-2.0
Category
Last pushed
Sep 01, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/liangyuwang/Tiny-Megatron"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
LowinLi/transformers-stream-generator
This is a text generation method which returns a generator, streaming out each token in...
ystemsrx/mini-nanoGPT
One-click training of your own GPT. Training a GPT has never been easier for beginners. /...
jaymody/picoGPT
An unnecessarily tiny implementation of GPT-2 in NumPy.
kyegomez/AttentionGrid
A network of attention mechanisms at your fingertips. Unleash the potential of attention...
kamalkraj/minGPT-TF
A minimal TF2 re-implementation of the OpenAI GPT training