dmis-lab/Outlier-Safe-Pre-Training
[ACL 2025] Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Models
This project helps machine learning engineers and researchers create more efficient Large Language Models (LLMs) from scratch. By using 'Outlier-Safe Pre-Training,' you can train new LLMs that perform well even when compressed to very small sizes (4-bit quantization), without losing accuracy. It takes raw text data for pre-training and produces a robust, quantization-ready LLM.
Use this if you are developing or training new Large Language Models and need them to be highly efficient and deployable on systems with limited computing resources, like edge devices.
Not ideal if you are looking to compress an already fully-trained LLM, as this method focuses on making models robust to quantization during their initial training phase.
Stars
35
Forks
4
Language
Python
License
—
Category
Last pushed
Nov 04, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/dmis-lab/Outlier-Safe-Pre-Training"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ModelTC/LightCompress
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs,...
p-e-w/heretic
Fully automatic censorship removal for language models
Orion-zhen/abliteration
Make abliterated models with transformers, easy and fast
YerbaPage/LongCodeZip
LongCodeZip: Compress Long Context for Code Language Models [ASE2025]
locuslab/wanda
A simple and effective LLM pruning approach.