asahi417/lm-vocab-trimmer
Vocabulary Trimming (VT) is a model compression technique, which reduces a multilingual LM vocabulary to a target language by deleting irrelevant tokens from its vocabulary. This repository contains a python-library vocabtrimmer, that remove irrelevant tokens from a multilingual LM vocabulary for the target language.
When working with large multilingual language models for tasks in a single language, this tool helps you reduce the model's size. It takes a pre-trained multilingual language model and a target language, then outputs a smaller, more efficient model optimized for that specific language. This is ideal for machine learning engineers or researchers deploying models for single-language applications.
No commits in the last 6 months.
Use this if you need to reduce the size of a multilingual language model for better performance or lower computational costs when working with a specific target language.
Not ideal if your application requires the full multilingual capabilities of the original large model or if you are not working with pre-trained models.
Stars
63
Forks
5
Language
Python
License
MIT
Category
Last pushed
Oct 25, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/asahi417/lm-vocab-trimmer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ModelTC/LightCompress
[EMNLP 2024 & AAAI 2026] A powerful toolkit for compressing large models including LLMs, VLMs,...
p-e-w/heretic
Fully automatic censorship removal for language models
Orion-zhen/abliteration
Make abliterated models with transformers, easy and fast
YerbaPage/LongCodeZip
LongCodeZip: Compress Long Context for Code Language Models [ASE2025]
locuslab/wanda
A simple and effective LLM pruning approach.