mridulsaklani/My_Tokenizer
It is a small model of tokenizer also used by every AI GPT's model to perform tha task how to convert alphabets into a specific assigned token and encoding or decoding
No commits in the last 6 months.
Stars
2
Forks
—
Language
Python
License
—
Category
Last pushed
May 25, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/mridulsaklani/My_Tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
SauravP97/hf-tokenizer-visualizer
Visualize HuggingFace Byte-Pair Encoding (BPE) Tokenizer encoding process
twinnydotdev/toxe
SentencePiece tokenizer for cross-encoders
jawrainey/hfta
Reference implementation: run any huggingface tokenizer in Android (rust).
andikaseptiadi/local-code-model
🛠️ Build a pure Go GPT-style transformer from scratch to grasp the fundamentals of large...
Example69420/splintr
🚀 Boost text processing speed with Splintr, a high-performance BPE tokenizer in Rust that...