fattorib/ZeRO-transformer
Two implementations of ZeRO-1 optimizer sharding in JAX
This helps machine learning engineers train very large transformer models that would otherwise exceed memory limits, even on powerful hardware. You input your model and training configurations, along with your dataset (typically on a GCP bucket). The output is a trained transformer model, which can then be used for various natural language processing tasks. It's designed for ML engineers working with large-scale language models.
No commits in the last 6 months.
Use this if you need to train transformer models with over a billion parameters on hardware like a TPU v3-32 and are encountering out-of-memory errors.
Not ideal if you are working with smaller models that fit within standard GPU memory or if you prefer not to use JAX for your deep learning projects.
Stars
14
Forks
—
Language
Python
License
MIT
Category
Last pushed
Jun 11, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/fattorib/ZeRO-transformer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ThilinaRajapakse/simpletransformers
Transformers for Information Retrieval, Text Classification, NER, QA, Language Modelling,...
jsksxs360/How-to-use-Transformers
Transformers 库快速入门教程
google/deepconsensus
DeepConsensus uses gap-aware sequence transformers to correct errors in Pacific Biosciences...
Denis2054/Transformers-for-NLP-2nd-Edition
Transformer models from BERT to GPT-4, environments from Hugging Face to OpenAI. Fine-tuning,...
abhimishra91/transformers-tutorials
Github repo with tutorials to fine tune transformers for diff NLP tasks