Tokenization Libraries Transformer Models
There are 2 tokenization libraries models tracked. The highest-rated is Akibkhan/LLM-Tokenizer at 13/100 with 0 stars.
Get all 2 projects as JSON
curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=tokenization-libraries&limit=20"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
| # | Model | Score | Tier |
|---|---|---|---|
| 1 |
Akibkhan/LLM-Tokenizer
The Llm-Tokenizer project is a lightweight, efficient tokenizer designed for... |
|
Experimental |
| 2 |
deltartificial/tokenizer
Compute long files token lengths for different LLM models, built in Rust. |
|
Experimental |