elixir-nx/tokenizers
Elixir bindings for 🤗 Tokenizers
This is a developer tool that helps Elixir programmers process human language data for machine learning tasks. It takes raw text inputs and converts them into numerical tokens, which are essential for training and running natural language processing models. The primary users are Elixir developers building applications that involve text analysis or understanding.
100 stars. No commits in the last 6 months.
Use this if you are an Elixir developer working with text data and need to convert it into a format suitable for machine learning models.
Not ideal if you are not an Elixir developer or are looking for a high-level, no-code solution for text processing.
Stars
100
Forks
19
Language
Elixir
License
Apache-2.0
Category
Last pushed
May 30, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/elixir-nx/tokenizers"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
guillaume-be/rust-tokenizers
Rust-tokenizer offers high-performance tokenizers for modern language models, including...
sugarme/tokenizer
NLP tokenizers written in Go language
openscilab/tocount
ToCount: Lightweight Token Estimator
reinfer/blingfire-rs
Rust wrapper for the BlingFire tokenization library
Scurrra/ubpe
Universal (general sequence) Byte-Pair Encoding