sefineh-ai/Amharic-Tokenizer
Syllable-aware BPE tokenizer for the Amharic language (አማርኛ) – fast, accurate, trainable.
When working with Amharic text for applications like machine translation or text analysis, you need to break down sentences into smaller, meaningful units. This tool takes raw Amharic text and outputs a list of 'tokens' (sub-word units) or numerical IDs for each token, which can then be used in various language models. It's designed for anyone building or researching natural language processing (NLP) systems for the Amharic language.
Use this if you need a fast and accurate way to prepare Amharic text for use in AI models, either by using a pre-trained model or training your own on custom Amharic data.
Not ideal if your project does not involve the Amharic language or if you need a full-fledged language model rather than just a tokenizer.
Stars
98
Forks
14
Language
Python
License
MIT
Category
Last pushed
Nov 17, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/sefineh-ai/Amharic-Tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
eliben/go-sentencepiece
Go implementation of the SentencePiece tokenizer
mdabir1203/BPE_Tokenizer_Visualizer
A Visualizer to check how BPE Tokenizer in an LLM Works
BobMcDear/minbpe-hs
Byte-level byte pair encoding (BPE) in Haskell
franciszekparma/GBPET
GPT-style language model with Byte Pair Encoding tokenizer, built from scratch in PyTorch.
sajjadh47/bpe-encoder-php
BPE (Byte-Pair Encoding) Encoder Decoder for OpenAI's GPT-2 / GPT-3 Implemented In Pure PHP,...