cliang1453/SAGE
No Parameters Left Behind: Sensitivity Guided Adaptive Learning Rate for Training Large Transformer Models (ICLR 2022)
This project helps machine learning engineers and researchers fine-tune large transformer models more efficiently. By using a sensitivity-guided adaptive learning rate, it takes a pre-trained transformer model and training data (like GLUE benchmark tasks) and produces a fine-tuned model that trains faster and potentially achieves better performance. This is for professionals working with state-of-the-art natural language processing or related deep learning models.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher looking to fine-tune large transformer models like BERT or RoBERTa with improved training speed and performance.
Not ideal if you are looking for a general-purpose machine learning library or if you are not working specifically with large transformer architectures.
Stars
29
Forks
2
Language
Python
License
MIT
Category
Last pushed
Feb 09, 2022
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/cliang1453/SAGE"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
peremartra/optipfair
Structured pruning and bias visualization for Large Language Models. Tools for LLM optimization...
VainF/Torch-Pruning
[CVPR 2023] DepGraph: Towards Any Structural Pruning; LLMs, Vision Foundation Models, etc.
horseee/LLM-Pruner
[NeurIPS 2023] LLM-Pruner: On the Structural Pruning of Large Language Models. Support...
CASIA-LMC-Lab/FLAP
[AAAI 2024] Fluctuation-based Adaptive Structured Pruning for Large Language Models
princeton-nlp/LLM-Shearing
[ICLR 2024] Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning