Tokenizer Libraries NLP Tools
There are 2 tokenizer libraries tools tracked. The highest-rated is Mbeeee111/tokenizer.cpp at 27/100 with 1 stars.
Get all 2 projects as JSON
curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=nlp&subcategory=tokenizer-libraries&limit=20"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
| # | Tool | Score | Tier |
|---|---|---|---|
| 1 |
Mbeeee111/tokenizer.cpp
📦 Optimize tokenization in C++ for HuggingFace models with a fast,... |
|
Experimental |
| 2 |
NotShrirang/marathi-tokenizer
🖋️ A sleek, BPE-powered tokenizer that understands the richness of Marathi. |
|
Experimental |