Text Tokenization Libraries NLP Tools

There are 17 text tokenization libraries tools tracked. 2 score above 50 (established tier). The highest-rated is guillaume-be/rust-tokenizers at 61/100 with 336 stars and 8,112 monthly downloads.

Get all 17 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=nlp&subcategory=text-tokenization-libraries&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	guillaume-be/rust-tokenizers Rust-tokenizer offers high-performance tokenizers for modern language...	61	Established	336	Rust
2	sugarme/tokenizer NLP tokenizers written in Go language	54	Established	316	Go
3	elixir-nx/tokenizers Elixir bindings for 🤗 Tokenizers	46	Emerging	100	Elixir
4	openscilab/tocount ToCount: Lightweight Token Estimator	44	Emerging	21	Python
5	reinfer/blingfire-rs Rust wrapper for the BlingFire tokenization library	44	Emerging	15	Rust
6	Scurrra/ubpe Universal (general sequence) Byte-Pair Encoding	43	Emerging	1	Python
7	euskadi31/go-tokenizer A Text Tokenizer library for Golang	39	Emerging	11	Go
8	frothywater/kanade-tokenizer Kanade is a single-layer disentangled speech tokenizer that extracts compact...	37	Emerging	85	Python
9	howl-anderson/PaddleTokenizer 使用 PaddlePaddle 实现基于深度神经网络的中文分词引擎 \| A DNN Chinese Tokenizer by Using PaddlePaddle	32	Emerging	15	JavaScript
10	rjmacarthy/string-tokeniser An implementation of Keras Tokenizer in JavaScript.	31	Emerging	3	JavaScript
11	Usama3627/tokenizer Implementation of BPE Tokenizer in Rust	31	Emerging	3	Rust
12	DePasqualeOrg/swift-tokenizers High-performance tokenizers	24	Experimental	3	Swift
13	Mecanik/Modern-Text-Tokenizer Modern UTF-8 aware C++ tokenizer with vocabulary support, ideal for NLP and...	22	Experimental	12	C++
14	rekram1-node/tokenizer Natural Language Processing (NLP) Tokenization Libary designed for English....	19	Experimental	3	Go
15	mrinalxdev/bpe-cpp implementation of Byte-Pair Encoding (BPE) for subword tokenization,...	13	Experimental	5	C++
16	m-doughty/Raku-Tokenizers Wrapper module for Huggingface Tokenizers	12	Experimental	3	Raku
17	brightjonathan/BPE-Stanford We train and implement a byte-level byte-pair encoding (BPE) tokenizer. In...	11	Experimental	—	Python

Comparisons in this category

rust-tokenizers and tokenizer (61 vs 31) tokenizer and go-tokenizer (54 vs 39)