Vision Transformer Optimization ML Frameworks

Official implementations and research papers focused on improving Vision Transformer architectures through efficiency enhancements, dynamic token pruning, hierarchical designs, and architectural innovations. Does NOT include general computer vision frameworks, multimodal models, or non-transformer-based vision approaches.

There are 103 vision transformer optimization frameworks tracked. 8 score above 50 (established tier). The highest-rated is Jittor/jittor at 59/100 with 3,221 stars. 1 of the top 10 are actively maintained.

Get all 103 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=ml-frameworks&subcategory=vision-transformer-optimization&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Framework	Score	Tier	Stars	Language
1	Jittor/jittor Jittor is a high-performance deep learning framework based on JIT compiling...	59	Established	3,221	Python
2	berniwal/swin-transformer-pytorch Implementation of the Swin Transformer in PyTorch.	58	Established	859	Python
3	zhanghang1989/ResNeSt ResNeSt: Split-Attention Networks	58	Established	3,264	Python
4	NVlabs/FasterViT [ICLR 2024] Official PyTorch implementation of FasterViT: Fast Vision...	54	Established	907	Python
5	ViTAE-Transformer/ViTPose The official repo for [NeurIPS'22] "ViTPose: Simple Vision Transformer...	53	Established	1,957	Python
6	sniklaus/pytorch-pwc a reimplementation of PWC-Net in PyTorch that matches the official Caffe version	51	Established	654	Python
7	microsoft/CvT This is an official implementation of CvT: Introducing Convolutions to...	51	Established	602	Python
8	gaohuang/MSDNet Multi-Scale Dense Networks for Resource Efficient Image Classification （ICLR...	50	Established	461	Lua
9	Khrylx/AgentFormer [ICCV 2021] Official PyTorch Implementation of "AgentFormer: Agent-Aware...	49	Emerging	309	Python
10	tobna/WhatTransformerToFavor Github repository for the paper Which Transformer to Favor: A Comparative...	49	Emerging	33	Python
11	innat/DOLG-TensorFlow Implementation of Deep Orthogonal Fusion of Local and Global Features in TensorFlow 2	48	Emerging	26	Jupyter Notebook
12	google-research/big_transfer Official repository for the "Big Transfer (BiT): General Visual...	47	Emerging	1,538	Python
13	richzhang/PerceptualSimilarity LPIPS metric. pip install lpips	47	Emerging	4,185	Python
14	Renumics/mesh2vec Turn CAE mesh data => aggregated element feature vectors for ML	47	Emerging	15	KFramework
15	iduta/pyconv Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual...	46	Emerging	331	Python
16	vra/dinov2-retrieval A cli program of image retrieval using dinov2	46	Emerging	79	Python
17	alon-albalak/TLiDB Transfer Learning in Dialogue Benchmarking Toolkit	45	Emerging	14	Python
18	bwconrad/vit-finetune Fine-tuning Vision Transformers on various classification datasets	45	Emerging	115	Python
19	clovaai/rexnet Official Pytorch implementation of ReXNet (Rank eXpansion Network) with...	45	Emerging	451	Python
20	walsvid/CoordConv Pytorch implementation of "An intriguing failing of convolutional neural...	45	Emerging	163	Jupyter Notebook
21	PracticumAI/transfer_learning Transfer learning is a powerful method allowing you to repurpose an AI model...	45	Emerging	3	Jupyter Notebook
22	VicenteVivan/geo-clip This is an official PyTorch implementation of our NeurIPS 2023 paper...	45	Emerging	330	Python
23	raoyongming/DynamicViT [NeurIPS 2021] [T-PAMI] DynamicViT: Efficient Vision Transformers with...	45	Emerging	651	Jupyter Notebook
24	Yangzhangcst/Transformer-in-Computer-Vision A paper list of some recent Transformer-based CV works.	44	Emerging	1,435	—
25	bryanlimy/V1T [TMLR 2023] V1T: Large-scale mouse V1 response prediction using a Vision Transformer	44	Emerging	23	Jupyter Notebook
26	LeapLabTHU/DAT Repository of Vision Transformer with Deformable Attention (CVPR2022) and...	44	Emerging	925	Python
27	ShirAmir/dino-vit-features Official implementation for the paper "Deep ViT Features as Dense Visual...	44	Emerging	464	Python
28	kampta/DeepLayout PyTorch implementation of "LayoutTransformer: Layout Generation and...	44	Emerging	165	Python
29	thuml/Xlearn Transfer Learning Library	43	Emerging	463	Jupyter Notebook
30	mit-han-lab/offsite-tuning Offsite-Tuning: Transfer Learning without Full Model	42	Emerging	387	Python
31	jwr1995/dc1d A 1D implementation of a deformable convolutional layer in PyTorch with a few tricks.	42	Emerging	46	Python
32	htdt/hyp_metric Hyperbolic Vision Transformers: Combining Improvements in Metric Learning \|...	42	Emerging	209	Python
33	chenhaoxing/SSFormers This repository is the code of the paper "Sparse Spatial Transformers for...	42	Emerging	49	Python
34	fkodom/yet-another-retnet A simple but robust PyTorch implementation of RetNet from "Retentive...	42	Emerging	106	Python
35	dongkyunk/DOLG-pytorch Unofficial PyTorch Implementation of "DOLG: Single-Stage Image Retrieval...	40	Emerging	135	Python
36	intel/transfer-learning Libraries and tools to support Transfer Learning	40	Emerging	20	Python
37	ChristophReich1996/MaxViT PyTorch reimplementation of the paper "MaxViT: Multi-Axis Vision...	40	Emerging	164	Python
38	baraline/convst Implementation of the Random Dilated Shapelet Transform algorithm along with...	40	Emerging	35	Python
39	AaltoVision/DGC-Net A PyTorch implementation of "DGC-Net: Dense Geometric Correspondence Network"	39	Emerging	206	Jupyter Notebook
40	amazon-science/semi-vit PyTorch implementation of Semi-supervised Vision Transformers	39	Emerging	61	Python
41	NVlabs/FAN Official PyTorch implementation of Fully Attentional Networks	39	Emerging	480	Python
42	DavidLandup0/deepvision PyTorch and TensorFlow/Keras image models with automatic weight conversions...	38	Emerging	42	Python
43	daniel-code/TubeViT An unofficial implementation of TubeViT in "Rethinking Video ViTs: Sparse...	37	Emerging	94	Python
44	FrancescoSaverioZuppichini/ViT Implementing Vi(sion)T(transformer)	37	Emerging	453	—
45	SunghwanHong/Cost-Aggregation-transformers Official implementation of CATs	37	Emerging	134	Python
46	apple/parameterized-transforms torchvision-based transforms that provide access to parameterization	37	Emerging	16	Python
47	NU-CUCIS/CrossPropertyTL Cross-property Deep Transfer Learning	36	Emerging	9	Jupyter Notebook
48	YifanXu74/Evo-ViT Official implement of Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision...	35	Emerging	74	Python
49	iduta/coconv [ICCV W] Contextual Convolutional Neural Networks...	35	Emerging	14	Python
50	ViTAE-Transformer/ViTAE-Transformer The official repo for [NeurIPS'21] "ViTAE: Vision Transformer Advanced by...	35	Emerging	281	Python
51	GuanRunwei/Awesome-Vision-Transformer-Collection Variants of Vision Transformer and its downstream tasks	34	Emerging	257	—
52	JoanaR/multi-mode-CNN-pytorch A PyTorch implementation of the Multi-Mode CNN to reconstruct Chlorophyll-a...	34	Emerging	10	Jupyter Notebook
53	MosbehBarhoumiRAI/VITON-PRE-PROCESSING This repository contains the initial implementation of pre-processing for...	34	Emerging	52	Jupyter Notebook
54	pavlo-melnyk/mlgp-embedme The official implementation of the "Embed Me If You Can: A Geometric...	34	Emerging	9	Jupyter Notebook
55	xiusu/ViTAS Code for ViTAS_Vision Transformer Architecture Search	33	Emerging	51	Python
56	shikishima-TasakiLab/Involution-PyTorch Unofficial PyTorch reimplemention of the paper "Involution: Inverting the...	33	Emerging	21	C++
57	AnkurDeria/MFT Pytorch implementation of Multimodal Fusion Transformer for Remote Sensing...	33	Emerging	237	Jupyter Notebook
58	insitro/ContextViT Contextual Vision Transformers for Robust Representation Learning	32	Emerging	15	Python
59	graldij/transformer-fusion Official repository of the "Transformer Fusion with Optimal Transport"...	32	Emerging	31	Python
60	benbergner/cropr A token pruning method that accelerates ViTs for various tasks while...	32	Emerging	27	Python
61	shashankvkt/DoRA_ICLR24 This repo contains the official implementation of ICLR 2024 paper "Is...	31	Emerging	95	Python
62	paulgavrikov/CNN-Filter-DB A database of over 1.4 billion 3x3 convolution filters extracted from...	31	Emerging	34	Jupyter Notebook
63	ViTAE-Transformer/ViTAE-VSA The official repo for [ECCV'22] "VSA: Learning Varied-Size Window Attention...	30	Emerging	158	Python
64	jman4162/PyTorch-Vision-Transformers-ViT Explore fine-tuning the Vision Transformer (ViT) model for object...	30	Emerging	7	Python
65	nerminnuraydogan/vision-transformer Vision Transformer explanation and implementation with PyTorch	30	Emerging	67	Jupyter Notebook
66	altndrr/vic Code implementation of our NeurIPS 2023 paper: Vocabulary-free Image Classification	30	Emerging	107	Python
67	billpsomas/simpool This repo contains the official implementation of ICCV 2023 paper "Keep It...	29	Experimental	101	Python
68	Rishit-dagli/Transformer-in-Transformer An Implementation of Transformer in Transformer in TensorFlow for image...	29	Experimental	43	Jupyter Notebook
69	mako443/Text2Pos-CVPR2022 Code, dataset and models for our CVPR 2022 publication "Text2Pos"	28	Experimental	54	Python
70	alantess/transformer Implementation of a modified vision transformer on the crypto market space	28	Experimental	14	Python
71	EthanBnntt/tinygrad-vit A minimalist implementation of the ViT (Vision Transformer) model, using tinygrad	27	Experimental	15	Python
72	ViTAE-Transformer/LeMeViT The official repo for [IJCAI'24] "LeMeViT: Efficient Vision Transformer with...	26	Experimental	53	Python
73	RohanG9929/LoFTR-in-Tensorflow Code for our re-implementation of "LoFTR: Detector-Free Local Feature...	26	Experimental	8	Jupyter Notebook
74	materight/RepNet-pytorch A PyTorch port with pre-trained weights of RepNet, from "Counting Out Time:...	26	Experimental	40	Python
75	PegHeads-Inc/PegHeads-Tutorial-4 TRANSFER LEARNING: TO CREATE A PRE-TRAINED MODEL	25	Experimental	6	Jupyter Notebook
76	EmPasLab/ExMobileVIT ExMobileViT: Lightweight Classifier Extension for Mobile Vision Transformer	25	Experimental	5	Python
77	janaalbader28/Waste-Classification-ViT Exploring the use of Vision Transformers (ViT) for waste classification	24	Experimental	1	Jupyter Notebook
78	suous/RecNeXt RecConv: Efficient Recursive Convolutions for Multi-Frequency Representations	24	Experimental	19	Python
79	sanket-poojary-03/Fine-tuning-ViVit Python script to fine tune Open source Video Vision Transformer (ViVit)...	23	Experimental	14	Python
80	lizhh268/FSSUWNet [IJCNN 2025 Oral] Official implementation of paper: FSSUWNet: Mitigating the...	23	Experimental	3	—
81	WalterSimoncini/fungivision Library implementation of "No Train, all Gain: Self-Supervised Gradients...	23	Experimental	40	Python
82	BobMcDear/vit-pytorch PyTorch implementation of the vision transformer	22	Experimental	17	Python
83	zhouchenlin2096/Awesome-Transformer-for-Vision-Recognition A comprehensive paper list of Transformer & Attention for Vision Recognition...	22	Experimental	20	—
84	chinefed/convolutional-set-transformer Official implementation of the Convolutional Set Transformer (Chinello &...	22	Experimental	11	Jupyter Notebook
85	zs1314/Fraesormer 【ICME2025 Oral】Offical Pytorch Code for "Fraesormer: Learning Adaptive...	21	Experimental	11	Python
86	rentainhe/ViT.pytorch The Pytorch reimplementation of Vision Transformer	21	Experimental	10	Jupyter Notebook
87	Tejeshyewale/transfer_learning_in_Deeplearning This project demonstrates image classification using transfer learning with...	21	Experimental	—	Jupyter Notebook
88	EvgenyKashin/non-leaking-conv Implementation of Spectral Leakage and Rethinking the Kernel Size in CNNs in Pytorch	21	Experimental	14	Jupyter Notebook
89	Atharv279/Transfer-Learning Files containing projects related to Transfer Learning	21	Experimental	—	Jupyter Notebook
90	AliKHaliliT/MobileViViT MobileViViT, a higher dimensional adaptation of MobileViT	21	Experimental	3	Python
91	jiaowoguanren0615/DINOV2-Pytorch This is a warehouse for DinoV2-models, based pytorch framework.	20	Experimental	5	Python
92	dabane-ghassan/int-lab-book Foveated Spatial Transformers	20	Experimental	6	Jupyter Notebook
93	MohammadRoodbari/Image-Classification image classification with fine tuning the BEiT vision transformer on CIFAR 10 dataset	20	Experimental	6	Jupyter Notebook
94	VikramRangarajan/SIEDD A fast coordinate-based neural video encoder	20	Experimental	3	Python
95	nick8592/ViT-Classification-CIFAR10 This repository contains an implementation of the Vision Transformer (ViT)...	20	Experimental	6	Jupyter Notebook
96	lucasjvds/ViT-for-Dark-Matter-Morphology Under the international Google Summer of Code program, the project...	19	Experimental	3	Jupyter Notebook
97	OSU-MLB/ViT_PEFT_Vision [CVPR'25 (Highlight)] Lessons and Insights from a Unifying Study of...	18	Experimental	46	Jupyter Notebook
98	aimagelab/TransFusion Official codebase of "Update Your Transformer to the Latest Release:...	15	Experimental	23	Python
99	iijumanaAhmed/Waste-Classification-ViT Exploring the use of Vision Transformers (ViT) for waste classification	13	Experimental	—	Jupyter Notebook
100	techsup93/CIFAR10-CNN-vs-ViT 🔍 Comparing CNN vs Vision Transformer (ViT) on CIFAR-10 with GPU T4 \| Deep...	13	Experimental	6	Python
101	sntsemilio/Transfer-learning A machine learning project focused on transfer learning techniques using...	13	Experimental	—	Jupyter Notebook
102	mahshid1378/SwinTransformerPytorch Implementation of the Swin Transformer in PyTorch. and use Article:...	11	Experimental	—	Python
103	justanhduc/involution A Pytorch CUDA/C++ JIT implementation with Python wrapper of Involution	10	Experimental	2	Cuda