LanguageMachines/ucto
Unicode tokeniser. Ucto tokenizes text files: it separates words from punctuation, and splits sentences. It offers several other basic preprocessing steps such as changing case that you can all use to make your text suited for further processing such as indexing, part-of-speech tagging, or machine translation. Ucto comes with tokenisation rules for several languages and can be easily extended to suit other languages. It has been incorporated for tokenizing Dutch text in Frog, our Dutch morpho-syntactic processor. http://ilk.uvt.nl/ucto --
This tool helps researchers and developers in natural language processing (NLP) to prepare raw text for analysis. It takes text files as input and precisely separates words from punctuation, splits sentences, and can even change text case. The output is clean, pre-processed text suitable for tasks like building search indexes, part-of-speech tagging, or machine translation.
Use this if you need to precisely break down text into individual words and sentences for linguistic analysis or other text processing applications, especially across multiple languages.
Not ideal if you need a full NLP pipeline with advanced features like semantic analysis or entity recognition, as this focuses only on foundational text segmentation.
Stars
70
Forks
14
Language
C++
License
GPL-3.0
Category
Last pushed
Mar 13, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/LanguageMachines/ucto"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related tools
google/sentencepiece
Unsupervised text tokenizer for Neural Network-based text generation.
daac-tools/vibrato
🎤 vibrato: Viterbi-based accelerated tokenizer
OpenNMT/Tokenizer
Fast and customizable text tokenization library with BPE and SentencePiece support
Systemcluster/kitoken
Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers,...
soaxelbrooke/python-bpe
Byte Pair Encoding for Python!