Text Tokenization Libraries Transformer Models
There are 3 text tokenization libraries models tracked. The highest-rated is NLPOptimize/flash-tokenizer at 45/100 with 509 stars.
Get all 3 projects as JSON
curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=text-tokenization-libraries&limit=20"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
| # | Model | Score | Tier |
|---|---|---|---|
| 1 |
NLPOptimize/flash-tokenizer
EFFICIENT AND OPTIMIZED TOKENIZER ENGINE FOR LLM INFERENCE SERVING |
|
Emerging |
| 2 |
bminixhofer/tokenkit
A toolkit implementing advanced methods to transfer models and model... |
|
Emerging |
| 3 |
briesearch/token-masks
Masked language model with Positional & One-Hot encoding - built using Aurora |
|
Experimental |