Tokenization Libraries NLP Tools
There are 4 tokenization libraries tools tracked. The highest-rated is AndersonBY/deepseek-tokenizer at 49/100 with 42 stars.
Get all 4 projects as JSON
curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=nlp&subcategory=tokenization-libraries&limit=20"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
| # | Tool | Score | Tier |
|---|---|---|---|
| 1 |
AndersonBY/deepseek-tokenizer
DeepSeek Tokenizer is an efficient and lightweight tokenization library with... |
|
Emerging |
| 2 |
cahya-wirawan/rwkv-tokenizer
A fast RWKV Tokenizer written in Rust |
|
Emerging |
| 3 |
dakofler/simple_tokenizers
Tokenizers is a collection of tokenization implementations focused on... |
|
Experimental |
| 4 |
SiriPrathikantam/custom-tokenize
A Python tokenizer project with regex, vocab building, and logging. |
|
Experimental |