NLP Model Interpretability NLP Tools

Tools and frameworks for explaining, visualizing, and understanding the decisions of NLP and ML models through techniques like feature attribution, concept activation vectors, attention analysis, and model-agnostic explanations. Does NOT include general model evaluation, performance metrics, or bias detection frameworks without interpretability focus.

There are 36 nlp model interpretability tools tracked. 4 score above 50 (established tier). The highest-rated is rmovva/HypotheSAEs at 64/100 with 77 stars.

Get all 36 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=nlp&subcategory=nlp-model-interpretability&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	rmovva/HypotheSAEs HypotheSAEs: hypothesizing interpretable relationships in text datasets...	64	Established	77	Jupyter Notebook
2	interpretml/interpret-text A library that incorporates state-of-the-art explainers for text-based...	56	Established	432	Python
3	fdalvi/NeuroX A Python library that encapsulates various methods for neuron interpretation...	54	Established	106	Python
4	jalammar/ecco Explain, analyze, and visualize NLP language models. Ecco creates...	54	Established	2,088	Jupyter Notebook
5	alexdyysp/ESIM-pytorch 中国高校计算机大赛--大数据挑战赛	41	Emerging	37	Jupyter Notebook
6	MultiplEYE-COST/wg1-experiment-implementation In this repository we keep the code for the implementation of the...	40	Emerging	9	Python
7	NeuroLIAA/reading-et Eye-tracking during reading of short stories	40	Emerging	6	Python
8	adaamko/POTATO XAI based human-in-the-loop framework for automatic rule-learning.	39	Emerging	49	Jupyter Notebook
9	RiccardoSpolaor/Verbal-Explanations-of-Spatio-Temporal-Graph-Neural-Networks-for-Traffic-Forecasting An eXplainable AI system to elucidate short-term speed forecasts in traffic...	38	Emerging	24	Jupyter Notebook
10	octanove/expats EXPATS: A Toolkit for Explainable Automated Text Scoring	36	Emerging	23	Python
11	ymcui/mrc-model-analysis Multilingual Multi-Aspect Explainability Analyses on Machine Reading...	33	Emerging	7	Python
12	mohsenfayyaz/DecompX DecompX: Explaining Transformers Decisions by Propagating Token...	32	Emerging	19	Jupyter Notebook
13	ravipatelxyz/nlp-ethics In depth evaluation of the ETHICS utilitarianism task dataset. An assessment...	30	Emerging	2	Jupyter Notebook
14	jwliao1209/Explainable-NLP 2022 AI CUP Explainable Information Tagging Competition for Natural Language...	30	Emerging	2	Python
15	robinvanschaik/interpret-flair A small repository to test Captum Explainable AI with a trained Flair...	30	Emerging	26	Jupyter Notebook
16	hint-lab/doctrack Dataset for EMNLP'23 Paper "DocTrack: A Visually-Rich Document Dataset...	28	Experimental	11	—
17	synapticore-io/ethics-model A modern, modular PyTorch framework for ethical text analysis, manipulation...	28	Experimental	3	Python
18	MichiganNLP/micromodels Micromodels -- A framework for accurate, explainable, data efficient, and...	27	Experimental	14	Python
19	fursovia/tcav_nlp "Interpretability Beyond Feature Attribution: Quantitative Testing with...	26	Experimental	8	Jupyter Notebook
20	avijit-thawani/numeracy-literacy Code for paper: "Numeracy Enhances the Literacy of Language Models"	23	Experimental	4	Python
21	christianwarmuth/explainable-predictive-process-monitoring-with-text On the Potential of Textual Data for Explainable Predictive Process...	20	Experimental	5	Jupyter Notebook
22	mainlp/gaze-guided-text-generation Code and data for the paper "Controlling Reading Ease with Gaze-Guided Text...	19	Experimental	—	Jupyter Notebook
23	amcrisan/interactive-model-cards An experimental project that examines whether interactivity can augment...	19	Experimental	4	Python
24	MortadhaMannai/XAI_ConstrainedAttentionVerifier Code for the NLDB 2023 paper. Work partially funded by grant...	19	Experimental	4	Jupyter Notebook
25	shresthasingh1501/legal_document_analysis Legal document analysis using BERT and FlanT5	19	Experimental	16	Jupyter Notebook
26	christophsk/classifier-lit PAIR Code's Language Interpretability Tool (LIT) for Text Classification	17	Experimental	1	Python
27	StonyBrookNLP/irene [ACL 2021] IrEne: Interpretable Energy Prediction for Transformers	13	Experimental	11	Python
28	pradippramanick/coexp-iros24 Code and data for the IROS 2024 paper - Multimodal Coherent Explanation...	13	Experimental	4	Python
29	EBAnO-Ecosystem/Text-EBAnO-Express T-EBAnO: Explaining deep learning black-box models for Natural Language Processing.	12	Experimental	8	Jupyter Notebook
30	shreyas-kowshik/nlp4if Code for the runners up entry on the English subtask on the...	12	Experimental	6	Python
31	danchern97/tda4la This is an official repository for "Acceptability Judgements via Examining...	12	Experimental	6	Jupyter Notebook
32	meiyor/DeepGaze-Text-Embedding-Map DeepGaze + Text-Embedding-Map project developed in Cardiff University -...	12	Experimental	7	Python
33	DIME-XAI/dime-xai Implementation of Dual Interpretable Model-agnostic Explanations for Rasa...	11	Experimental	2	Python
34	yang-su2000/State-Space-Interpretability Investigation of state space model interpretability using SHAP (SHapley...	11	Experimental	3	Jupyter Notebook
35	MortadhaMannai/XAIPalette-Exploring-Image-Structure-via-Color-Distillation-for-Explainable-AI This repository contains the PyTorch implementation of XAIPalette, a novel...	11	Experimental	4	Python
36	lennox55555/Legal-BERT-RLHF This web app is part of a research project to identify and address biases in...	10	Experimental	2	Jupyter Notebook