bminixhofer/zett
Code for Zero-Shot Tokenizer Transfer
This project helps machine learning engineers and researchers adapt existing large language models (LLMs) to use different tokenizers without extensive retraining. You input a pre-trained LLM and a desired tokenizer, and it outputs a new version of the LLM that works seamlessly with that tokenizer. This is designed for developers working on natural language processing tasks who need flexibility with model architectures and tokenization strategies.
143 stars. No commits in the last 6 months.
Use this if you need to make a pre-trained large language model compatible with a different tokenizer than it was originally trained with, especially for multilingual applications or specific domain texts.
Not ideal if you are a non-technical user simply looking to use an LLM out-of-the-box, or if you need to train a brand-new language model from scratch.
Stars
143
Forks
11
Language
Python
License
—
Category
Last pushed
Jan 14, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/bminixhofer/zett"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
jncraton/languagemodels
Explore large language models in 512MB of RAM
microsoft/unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
haizelabs/verdict
Inference-time scaling for LLMs-as-a-judge.
albertan017/LLM4Decompile
Reverse Engineering: Decompiling Binary Code with Large Language Models
bytedance/Sa2VA
Official Repo For Pixel-LLM Codebase