OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
This is a versatile tool for language processing engineers, machine learning scientists, and data scientists who need to prepare raw text for analysis or model training. It takes raw text as input and breaks it down into individual words or subword units (tokens), which are the building blocks for natural language processing tasks. This allows you to precisely control how text is segmented and processed before it’s fed into your algorithms.
330 stars.
Use this if you need fine-grained control over how text is segmented into tokens, including handling different languages, preserving case, or using subword models like BPE or SentencePiece for machine translation or large language model training.
Not ideal if you just need basic word splitting for simple text analysis and don't require advanced customization or subword segmentation.
Stars
330
Forks
80
Language
C++
License
MIT
Category
Last pushed
Jan 10, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/OpenNMT/Tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
daac-tools/vaporetto
🛥 Vaporetto: Very accelerated pointwise prediction based tokenizer
soaxelbrooke/python-bpe
Byte Pair Encoding for Python!