zzbright1998/SentenceKV
Official implementation of "SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching" (COLM 2025). A novel KV cache compression method that organizes cache at sentence level using semantic similarity.
This project helps large language model (LLM) developers and researchers make their models run more efficiently, especially with long texts. It takes the text an LLM processes and organizes its internal memory (KV cache) by grouping semantically similar sentences, leading to reduced memory use and faster response times. The end-user is an LLM developer or researcher who is working on optimizing model performance for applications that involve understanding and generating long-form content.
No commits in the last 6 months.
Use this if you are a developer or researcher building or experimenting with LLMs and need to reduce their memory footprint and speed up inference, especially for tasks involving long input texts.
Not ideal if you are a general user looking for an off-the-shelf LLM application, rather than a developer looking to optimize an LLM's underlying performance.
Stars
11
Forks
1
Language
Python
License
—
Category
Last pushed
Sep 29, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/zzbright1998/SentenceKV"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ModelCloud/GPTQModel
LLM model quantization (compression) toolkit with hw acceleration support for Nvidia CUDA, AMD...
intel/auto-round
🎯An accuracy-first, highly efficient quantization toolkit for LLMs, designed to minimize quality...
pytorch/ao
PyTorch native quantization and sparsity for training and inference
bodaay/HuggingFaceModelDownloader
Simple go utility to download HuggingFace Models and Datasets
NVIDIA/kvpress
LLM KV cache compression made easy