Model Evaluation Diagnostics Transformer Models

Tools for systematically evaluating, diagnosing, and benchmarking transformer models across NLI, WSD, and other NLP tasks using standard test sets and evaluation frameworks. Does NOT include general model training, fine-tuning without evaluation focus, or language-specific model overviews.

There are 57 model evaluation diagnostics models tracked. 1 score above 50 (established tier). The highest-rated is LoicGrobol/zeldarose at 51/100 with 28 stars.

Get all 57 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=model-evaluation-diagnostics&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	LoicGrobol/zeldarose Train transformer-based models.	51	Established	28	Python
2	CPJKU/wechsel Code for WECHSEL: Effective initialization of subword embeddings for...	49	Emerging	89	Python
3	yuanzhoulvpi2017/zero_nlp 中文nlp解决方案(大模型、数据、模型、训练、推理)	49	Emerging	3,783	Jupyter Notebook
4	minggnim/nlp-models A repository for training transformer based models	49	Emerging	2	Jupyter Notebook
5	IntelLabs/nlp-architect A model library for exploring state-of-the-art deep learning topologies and...	49	Emerging	2,935	Python
6	MahmoudWahdan/dialog-nlu Tensorflow and Keras implementation of the state of the art researches in...	46	Emerging	100	Jupyter Notebook
7	yuanzhoulvpi2017/quick_sentence_transformers sentence-transformers to onnx 让sbert模型推理效率更快	46	Emerging	166	Python
8	ukairia777/tensorflow-nlp-tutorial tensorflow를 사용하여 텍스트 전처리부터, Topic Models, BERT, GPT, LLM과 같은 최신 모델의 다운스트림...	45	Emerging	575	Jupyter Notebook
9	soldni/pyterrier_sentence_transformers Create PyTerrier compatible dense indices using any sentence_transformers model	43	Emerging	6	Python
10	HarderThenHarder/transformers_tasks ⭐️ NLP Algorithms with transformers lib. Supporting Text-Classification,...	41	Emerging	2,412	Jupyter Notebook
11	g8a9/ferret A python package for benchmarking interpretability techniques on Transformers.	39	Emerging	215	Python
12	sinanuozdemir/oreilly-bert-nlp This repository contains code for the O'Reilly Live Online Training for BERT	36	Emerging	32	Jupyter Notebook
13	Azure/nlp-samples Japanese NLP sample codes	36	Emerging	10	Shell
14	ManashJKonwar/NLP-Transformers Transformer (BERT, GPT2, etc.) based Training Module for popular NLP tasks	34	Emerging	9	Python
15	rajaswa/indic-syntax-evaluation Vyākarana: A Colorless Green Benchmark for Syntactic Evaluation in Indic Languages	32	Emerging	15	Jupyter Notebook
16	shunk031/allennlp-shiba-model AllenNLP integration for Shiba: Japanese CANINE model	32	Emerging	12	Python
17	ropensci/pangoling An R package for estimating the log-probabilities of words in a given...	31	Emerging	12	R
18	prajjwal1/generalize_lm_nli Code for the paper EMNLP 2021 workshop paper "Generalization in NLI: Ways...	31	Emerging	34	Jupyter Notebook
19	VirtualRoyalty/gan-plus-nlp Generative adversarial approach to most popular NLP tasks	31	Emerging	4	Jupyter Notebook
20	matteomedioli/BERT-KG Enriching Language Models Representations via Knowledge Graphs Regularisation	31	Emerging	3	Python
21	Nickil21/weakly-supervised-parsing Official Code for our Findings of ACL 2022 paper: Co-training an...	31	Emerging	4	Python
22	stevezheng23/fewshot_nlp_pt Few-shot NLP in PyTorch	31	Emerging	4	Python
23	CyberAgentAILab/japanese-nli-model This repository provides the code for Japanese NLI model, a fine-tuned...	30	Emerging	6	Jupyter Notebook
24	th789/mbr-for-nmt Characterizing the performance of minimum Bayes risk (MBR) decoding for...	30	Emerging	2	Jupyter Notebook
25	ai-forever/model-zoo NLP model zoo for Russian	29	Experimental	50	—
26	yucc2018/share 一些代码实践分享。	28	Experimental	22	Jupyter Notebook
27	Beomi/transformers-language-modeling Train 🤗transformers with DeepSpeed: ZeRO-2, ZeRO-3	28	Experimental	23	Python
28	proycon/deepfrog An NLP-suite powered by deep learning	28	Experimental	19	Rust
29	TRISTAN-ORF/RiboTIE Scripts and instructions to apply RiboTIE on Ribo-seq data	27	Experimental	19	—
30	ishan00/meta-learning-for-multi-task-multilingual Official Repository for the paper titled "Meta-Learning for Effective...	26	Experimental	9	Python
31	DFKI-NLP/gevalm Code and data for the paper "Evaluating German Transformer Language Models...	25	Experimental	7	Python
32	hppRC/simple-simcse-ja Exploring Japanese SimCSE	24	Experimental	69	Python
33	zhestyatsky/MCL-WiC Research on Multilingual and Cross-lingual Word-in-Context Disambiguation	23	Experimental	4	Jupyter Notebook
34	SapienzaNLP/xl-wsd-code Code to train and test Word Sense Disambiguation models based on different...	22	Experimental	15	Python
35	princeton-nlp/MultilingualAnalysis Repository for the paper titled: "When is BERT Multilingual? Isolating...	21	Experimental	13	Python
36	RobinSmits/Dutch-NLP-Experiments This repository contains a number of experiments with Multi Lingual...	20	Experimental	5	Python
37	brihijoshi/granular-similarity-COLING-2020 Code for the paper "The Devil is in the Details: Evaluating Limitations of...	20	Experimental	8	Jupyter Notebook
38	iamlxb3/UMAMGT Code for the publication of LREC'22	19	Experimental	3	Jupyter Notebook
39	HannaAbiAkl/PSYCHIC The official repository for the PSYCHIC model	19	Experimental	3	Jupyter Notebook
40	TRISTAN-ORF/RiboTIE_article Scripts run to produce the RiboTIE paper	19	Experimental	3	Shell
41	skomban/seq-unscrambler Unscrambles shuffled letters in a word sequence.	18	Experimental	2	Python
42	SambhawDrag/XLNet.jl A Julia-based implementation of XLNet: A Generalized Autoregressive...	17	Experimental	1	Julia
43	bglid/haitian-creole-nlu Project designed to reimplement and build upon CreoleVal's Reading...	17	Experimental	1	Python
44	loubnabnl/canine-mednli CANINE for Medical Natural Language Inference on MedNLI data, as part of the...	17	Experimental	1	Python
45	aarnetalman/nli-with-transformers Fine-tune transformers with NLI data	15	Experimental	—	Python
46	mhdr3a/transformers-diagnostics Model Evaluation using SuperGLUE Diagnostic Dataset	13	Experimental	—	Python
47	DudalaShrujana/nlp-transformers-toolkit ModularNLP pipeline utilizing Hugging Face Transformers for Sentiment...	13	Experimental	—	Python
48	mhdr3a/transformers-snli Model Evaluation using SNLI Development Set	11	Experimental	—	Python
49	tranquoctrinh/huggingface-transformers-examples Fine-tuning (or training from scratch) the library models for language...	11	Experimental	—	Python
50	mhdr3a/transformers-hans Adversarial evaluation of model performances [Updated]	11	Experimental	—	Python
51	mhdr3a/transformers-wanli Model Evaluation using WANLI Test Set	11	Experimental	—	Python
52	MatteoFasulo/Transformers-Sentence-Reconstruction Sentence Reconstruction using Transformer Model	11	Experimental	—	Jupyter Notebook
53	Evfidiw/LMs_NLU Exploring different language models on text classification tasks.	11	Experimental	3	Python
54	chloeskt/nlp_ensae Final project of the Machine Learning for Natural Language Processing at...	11	Experimental	3	Jupyter Notebook
55	Mrpatekful/dialogue-graph Enriching pre-trained language models with knowledge graphs for dialogue generation.	11	Experimental	—	Python
56	FaresGh1997/Contexto_3lang implementation of Contexto (guess the word game) for three diffrent...	10	Experimental	2	Jupyter Notebook
57	LeonardoEmili/neural-wsd Neural WSD with Transformers and candidate masking	10	Experimental	2	Python