Bpe Tokenizers Embedding Tools
There are 2 bpe tokenizers tools tracked. The highest-rated is gweidart/rs-bpe at 44/100 with 37 stars.
Get all 2 projects as JSON
curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=embeddings&subcategory=bpe-tokenizers&limit=20"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
| # | Tool | Score | Tier |
|---|---|---|---|
| 1 |
gweidart/rs-bpe
A ridiculously fast Python BPE (Byte Pair Encoder) implementation written in Rust |
|
Emerging |
| 2 |
taabishhh/LLM_Preprocessing
This project implements a Byte Pair Encoding (BPE) tokenization approach... |
|
Experimental |