cheneydon/efficient-bert
This repository contains the code for the paper in Findings of EMNLP 2021: "EfficientBERT: Progressively Searching Multilayer Perceptron via Warm-up Knowledge Distillation".
This project helps machine learning engineers and researchers create smaller, faster, and more efficient language models for tasks like text classification or question answering. You provide large text datasets (like Wikipedia and BooksCorpus) and specific task datasets (like GLUE or SQuAD). The output is a highly optimized language model that performs well while using fewer computational resources. It's ideal for those working on natural language processing applications where speed and efficiency are critical.
No commits in the last 6 months.
Use this if you need to develop compact and performant BERT-like language models for deployment in resource-constrained environments or for faster inference.
Not ideal if you are a beginner looking for a simple, out-of-the-box language model or if you don't have experience with pre-training and fine-tuning large language models.
Stars
33
Forks
4
Language
Python
License
—
Category
Last pushed
Jun 14, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/cheneydon/efficient-bert"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
airaria/TextBrewer
A PyTorch-based knowledge distillation toolkit for natural language processing
sunyilgdx/NSP-BERT
The code for our paper "NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original...
princeton-nlp/CoFiPruning
[ACL 2022] Structured Pruning Learns Compact and Accurate Models https://arxiv.org/abs/2204.00408
kssteven418/LTP
[KDD'22] Learned Token Pruning for Transformers
georgian-io/Transformers-Domain-Adaptation
:no_entry: [DEPRECATED] Adapt Transformer-based language models to new text domains