BERT Model Implementations Transformer Models

PyTorch and framework-specific implementations of BERT and BERT-variant architectures (RoBERTa, DistilBERT, etc.), including pretraining, finetuning libraries, and language-specific BERT models. Does NOT include task-specific applications (NER, classification, QA), downstream finetuning notebooks, or non-BERT transformer implementations.

There are 73 bert model implementations models tracked. 6 score above 50 (established tier). The highest-rated is Tongjilibo/bert4torch at 66/100 with 1,335 stars.

Get all 73 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=bert-model-implementations&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	Tongjilibo/bert4torch An elegent pytorch implement of transformers	66	Established	1,335	Python
2	nyu-mll/jiant jiant is an nlp toolkit	59	Established	1,674	Python
3	lonePatient/TorchBlocks A PyTorch-based toolkit for natural language processing	53	Established	160	Python
4	monologg/JointBERT Pytorch implementation of JointBERT: "BERT for Joint Intent Classification...	51	Established	738	Python
5	grammarly/gector Official implementation of the papers "GECToR – Grammatical Error...	51	Established	955	Python
6	appvision-ai/fast-bert Super easy library for BERT based NLP models	50	Established	1,920	Python
7	sagorbrur/bangla-bert Bangla-Bert is a pretrained bert model for Bengali language	47	Emerging	83	Jupyter Notebook
8	voidful/TFkit 🤖📇 handling multiple nlp task in one pipeline	46	Emerging	57	Python
9	gitabtion/BertBasedCorrectionModels PyTorch impelementations of BERT-based Spelling Error Correction Models. ...	44	Emerging	279	Python
10	dccuchile/beto BETO - Spanish version of the BERT model	44	Emerging	502	—
11	sagorbrur/bntransformer Bengali transformer using transformers	44	Emerging	22	Python
12	backprop-ai/backprop Backprop makes it simple to use, finetune, and deploy state-of-the-art ML models.	44	Emerging	241	Python
13	iPieter/RobBERT A Dutch RoBERTa-based language model	43	Emerging	207	Jupyter Notebook
14	JetRunner/BERT-of-Theseus ⛵️The official PyTorch implementation for "BERT-of-Theseus: Compressing BERT...	43	Emerging	315	Python
15	gitabtion/SoftMaskedBert-PyTorch 🙈 An unofficial implementation of SoftMaskedBert based on huggingface/transformers.	43	Emerging	97	Python
16	menon92/BangalASR Transformer based Bangla Speech Recognition \| Encoder Decoder Architecture	42	Emerging	57	Jupyter Notebook
17	ymcui/PERT PERT: Pre-training BERT with Permuted Language Model	41	Emerging	367	—
18	Ethan-yt/guwenbert GuwenBERT: 古文预训练语言模型（古文BERT） A Pre-trained Language Model for Classical...	41	Emerging	555	—
19	JulesBelveze/bert-squeeze 🛠️ Tools for Transformers compression using PyTorch Lightning ⚡	40	Emerging	85	Python
20	nlpaueb/greek-bert A Greek edition of BERT pre-trained language model	38	Emerging	148	Python
21	dbmdz/berts DBMDZ BERT, DistilBERT, ELECTRA, GPT-2 and ConvBERT models	38	Emerging	159	—
22	alexa/ramen A software for transferring pre-trained English models to foreign languages	37	Emerging	19	Python
23	rdenadai/BR-BERTo Transformer model for Portuguese language (Brazil pt_BR)	37	Emerging	16	Python
24	cakshat/AlloyBERT Introducing AlloyBERT: a transformer encoder-based model for predicting...	36	Emerging	12	Python
25	bnosac/golgotha Contextualised Embeddings and Language Modelling using BERT and Friends using R	36	Emerging	47	R
26	retarfi/language-pretraining Pre-training Language Models for Japanese	36	Emerging	50	Python
27	TayeeChang/keras_transformers the implement of transformer family such as bert, alber, roberta, nezha, etc.	35	Emerging	7	Python
28	Beomi/exbert-transformers exBERT on Transformers🤗	35	Emerging	10	Python
29	shahrukhx01/bert-probe BERT Probe: A python package for probing attention based robustness to...	34	Emerging	18	Jupyter Notebook
30	isaacus-dev/emubert-creator The training code behind EmuBert, the largest open-source masked language...	33	Emerging	3	Python
31	Beomi/KcBERT-Finetune KcBERT/KcELECTRA Fine Tune Benchmarks code (forked from...	33	Emerging	47	Python
32	HeegyuKim/language-model 한국어 언어 모델 학습을 위한 프로젝트(Flax, Pytorch with Huggingface Accelerate)	32	Emerging	32	Jupyter Notebook
33	psychbruce/FMAT 😷 The Fill-Mask Association Test (FMAT): Measuring Propositions in Natural Language.	32	Emerging	16	R
34	asiff00/Bengali-Sentence-Error-Correction Fine-tune mBart 50 for Bengali Sentence Error Correction	31	Emerging	4	Jupyter Notebook
35	AshutoshDongare/softskill-NER Fine tuning 🤗 transformer model for softskill NER task	31	Emerging	3	Jupyter Notebook
36	DomHudson/bert-in-production A collection of resources on using BERT (https://arxiv.org/abs/1810.04805 )...	31	Emerging	96	—
37	gitabtion/ConvBert-PyTorch 🤗An unofficial PyTorch implementation of ConvBert based on huggingface/transformers.	30	Emerging	17	Python
38	PlanTL-GOB-ES/lm-biomedical-clinical-es Official source for Spanish pretrained biomedical and clinical language...	30	Emerging	26	Python
39	phkhanhtrinh23/spelling_correction_project This spelling correction project helps people fix English spelling mistakes....	29	Experimental	18	Python
40	YRL-AIDA/RuTaBERT RuTaBERT is a framework for solving column type and property annotation...	29	Experimental	7	Python
41	haozhg/lmd Language Model Decomposition: Quantifying the Dependency and Correlation of...	28	Experimental	10	Python
42	sagorbrur/fillblank Fill The Blank	27	Experimental	2	Jupyter Notebook
43	lcl-hse/heptabot A full-text error corrector for English based on transformers and deep learning	26	Experimental	10	Jupyter Notebook
44	shreydan/masked-language-modeling Transformers Pre-Training with MLM objective — implemented encoder-only...	25	Experimental	6	Jupyter Notebook
45	LennartKeller/roberta2longformer Convert pretrained RoBerta models to various long-document transformer models	25	Experimental	11	Python
46	ilanaliouchouche/KANBert Implementation of an Encoder only MoE usable as an Embedding Model,...	24	Experimental	2	Python
47	joshstephenson/MorphemeSegmentation This is a survey of morpheme segmentation techniques including 2 baselines...	23	Experimental	3	Python
48	Vincentiv/BERT_Finetuning_from_scratch Notebook on finetuning BERT	22	Experimental	2	Jupyter Notebook
49	Thisen-Ekanayake/HelaBERT A compact BERT (6-layer) masked language model trained from scratch on a...	21	Experimental	—	Jupyter Notebook
50	sappho192/ffxiv-ja-ko-translator Japanese→Korean translator model specialized in Final Fantasy XIV based on...	21	Experimental	11	C#
51	sfp932705/simple_bert A pure pytorch from scratch implementation of BERT	21	Experimental	—	Python
52	RichardScottOZ/geoscience-transformers-for-predictive-mapping-of-critical-minerals First pass paper implementation	21	Experimental	—	Python
53	tejasvaidhyadev/ALBERT.jl ALBERT(A Lite BERT for Self-Supervised Learning of Language Representations)...	20	Experimental	7	Julia
54	SumitM0432/XLM-RoBERTa-for-Textual-Entailment A multilingual model XLM- RoBERTa for the textual entailment of sequence...	20	Experimental	6	Jupyter Notebook
55	viktor-shcherb/vive_la_ner The default way to fine-tune BERT is wrong. Here is why	19	Experimental	4	Jupyter Notebook
56	mhmdsabry/BERT_with_Residual_vs_Highway Comparing between residual stream and highway stream in transformers(BERT) .	19	Experimental	3	Python
57	DiFronzo/Multilingual-Models mBERT and XLM-R for encodeing of Scandinavian languages	19	Experimental	3	Python
58	teticio/inBERTolate Hit your word count by using BERT to pad out your essays!	19	Experimental	3	Python
59	gaolichen/simplebert A simple implementation of transformer models with tensorflow/keras.	17	Experimental	1	Python
60	cbstanley/dp-bert Differential privacy with BERT model	17	Experimental	1	Python
61	Sean652039/Token-Masking Token Masking Regularization	16	Experimental	—	Python
62	mdmmn378/spell-magic Transformer Based Seq2Seq Model for Bangla Spell Correction	13	Experimental	—	Jupyter Notebook
63	seoyeon9646/MLM-data-augmentation Masked Language Modeling for data augmentation	13	Experimental	9	Python
64	davydantoniuk/grammarfix-bot Fine-tuned a Hugging Face transformer model for grammar correction.	12	Experimental	1	Jupyter Notebook
65	sumuzhao/Investigate-BERT-Non-linearity-Commutativity Investigate BERT on Non-linearity and Layer Commutativity	11	Experimental	4	Python
66	MojammelHossain/coref_model BERT for Coreference Resolution	11	Experimental	—	Python
67	ardimento/bugfix This repository contains the implementation for predicting bug-fixing time...	11	Experimental	—	Python
68	MawadaMhd/BERT This repository houses all resources related to the Bidirectional Encoder...	11	Experimental	—	Jupyter Notebook
69	gitabtion/bb-corrector Bert-based chinese spelling error corrector. 基于Bert的中文文本纠错工具。	11	Experimental	—	—
70	MystikHub/space-efficient-bert Code repository for the research paper "Space Efficient Transformer Neural Network"	11	Experimental	—	C++
71	many-facedgod/BERT-PyTorch A PyTorch implementation of BERT proposed by Devlin et al.	10	Experimental	2	Python
72	stas1f1/techdebt-project A project that utilizes fine-tuning of CodeBERT and CodeT5 to detect bad...	10	Experimental	2	Jupyter Notebook
73	atherfawaz/BERT-RoBERTa Training the Bidirectional Encoder Representations from Transformers (BERT)...	10	Experimental	2	Jupyter Notebook