Sparse Attention Optimization Transformer Models

There are 26 sparse attention optimization models tracked. 1 score above 70 (verified tier). The highest-rated is fla-org/flash-linear-attention at 76/100 with 4,549 stars. 1 of the top 10 are actively maintained.

Get all 26 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=sparse-attention-optimization&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	fla-org/flash-linear-attention 🚀 Efficient implementations of state-of-the-art linear attention models	76	Verified	4,549	Python
2	thu-ml/SageAttention [ICLR2025, ICML2025, NeurIPS2025 Spotlight] Quantized Attention achieves...	57	Established	3,213	Cuda
3	thu-ml/SpargeAttn [ICML2025] SpargeAttention: A training-free sparse attention that...	54	Established	956	Cuda
4	fla-org/flame 🔥 A minimal training framework for scaling FLA models	52	Established	355	Python
5	foundation-model-stack/fms-fsdp 🚀 Efficiently (pre)training foundation models with native PyTorch features,...	52	Established	282	Python
6	NX-AI/mlstm_kernels Tiled Flash Linear Attention library for fast and efficient mLSTM Kernels.	44	Emerging	87	Jupyter Notebook
7	skylight-org/sparse-attention-hub Advancing the frontier of efficient AI	43	Emerging	54	Python
8	XunhaoLai/native-sparse-attention-triton Efficient triton implementation of Native Sparse Attention.	41	Emerging	269	Python
9	NVIDIA/Star-Attention Efficient LLM Inference over Long Sequences	40	Emerging	392	Python
10	NimbleEdge/sparse_transformers Sparse Inferencing for transformer based LLMs	39	Emerging	216	Python
11	Infini-AI-Lab/vortex_torch Vortex: A Flexible and Efficient Sparse Attention Framework	38	Emerging	49	Python
12	Bruce-Lee-LY/flash_attention_inference Performance of the C++ interface of flash attention and flash attention v2...	37	Emerging	43	C++
13	zhenyi4/ssa Official repository for "SSA: Sparse Sparse Attention by Aligning Full and...	36	Emerging	10	Python
14	Relaxed-System-Lab/Flash-Sparse-Attention 🚀🚀 Efficient implementations of Native Sparse Attention	36	Emerging	983	Python
15	Bruce-Lee-LY/decoding_attention Decoding Attention is specially optimized for MHA, MQA, GQA and MLA using...	35	Emerging	46	C++
16	egaoharu-kensei/flash-attention-triton Cross-platform FlashAttention-2 Triton implementation for Turing+ GPUs with...	34	Emerging	21	Python
17	jlamprou/Infini-Attention Efficient Infinite Context Transformers with Infini-attention Pytorch...	27	Experimental	86	Python
18	nanowell/Q-Sparse-LLM My Implementation of Q-Sparse: All Large Language Models can be Fully...	26	Experimental	34	Python
19	wesleyscholl/drex 🦀 The transformer is a brilliant hack scaled past its limits. DREX is what...	23	Experimental	1	Python
20	AstrolexisAI/MnemoCUDA Expert streaming inference engine for MoE models larger than VRAM — run...	22	Experimental	—	C
21	aymanelrody/FlashMLA ⚡ Optimize attention mechanisms with FlashMLA, a library of advanced sparse...	22	Experimental	—	C++
22	NAME0x0/OMNI PERSPECTIVE v2 — A 1.05 trillion parameter sparse Mixture-of-Experts...	22	Experimental	1	Rust
23	XunhaoLai/ring-sliding-window-attention Ring sliding window attention implementation with flash attention	22	Experimental	9	Python
24	kamalrss88/FlashMLA 🚀 Accelerate attention mechanisms with FlashMLA, featuring optimized kernels...	21	Experimental	—	C++
25	BICLab/MetaLA Offical implementation of "MetaLA: Unified Optimal Linear Approximation to...	21	Experimental	35	Python
26	HassanJbara/lin-attn-zoo Pure PyTorch implementations of popular linear attention models	13	Experimental	—	Python

Comparisons in this category

flash-linear-attention and SageAttention (76 vs 57) flash-linear-attention and Flash-Sparse-Attention (76 vs 36) flash-linear-attention and Star-Attention (76 vs 40) flash-linear-attention and flame (76 vs 52) flash-linear-attention and flash_attention_inference (76 vs 37) flash-linear-attention and ring-sliding-window-attention (76 vs 22) SageAttention and SpargeAttn (57 vs 54)