Hsu1023/DuQuant
[NeurIPS 2024 Oral🔥] DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs.
This project helps machine learning engineers and researchers optimize large language models (LLMs) for deployment. It takes existing LLMs, like those from the LLaMA or Mistral series, and applies a dual transformation technique. The output is a "quantized" LLM that uses less memory and computes faster, while maintaining its performance on tasks like text generation or question answering.
180 stars. No commits in the last 6 months.
Use this if you need to deploy large language models more efficiently on hardware with limited resources, such as edge devices or mobile platforms, without significantly compromising accuracy.
Not ideal if you are a general user looking for a pre-trained, ready-to-use LLM without needing to optimize its underlying architecture.
Stars
180
Forks
17
Language
Python
License
MIT
Category
Last pushed
Oct 03, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Hsu1023/DuQuant"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
bitsandbytes-foundation/bitsandbytes
Accessible large language models via k-bit quantization for PyTorch.
intel/neural-compressor
SOTA low-bit LLM quantization (INT8/FP8/MXFP8/INT4/MXFP4/NVFP4) & sparsity; leading model...
dropbox/hqq
Official implementation of Half-Quadratic Quantization (HQQ)
OpenGVLab/OmniQuant
[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.
VITA-Group/Q-GaLore
Q-GaLore: Quantized GaLore with INT4 Projection and Layer-Adaptive Low-Rank Gradients.