ash-01xor/bpe.c
Simple Byte pair Encoding mechanism used for tokenization process . written purely in C
This tool helps developers working with natural language by breaking down text into smaller, more manageable pieces called tokens. It takes raw text as input and produces a sequence of numerical IDs representing these tokens, which can then be decoded back into text. It's used by software engineers who are building text-processing applications and need an efficient way to handle language data.
146 stars. No commits in the last 6 months.
Use this if you are a C programmer building a text processing application and need a simple, lightweight way to convert text into numerical tokens for further analysis or model input.
Not ideal if you need a pre-trained tokenizer for a specific language or a tool with extensive language-specific features beyond basic byte pair encoding.
Stars
146
Forks
4
Language
C
License
MIT
Category
Last pushed
Nov 11, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/ash-01xor/bpe.c"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
georg-jung/FastBertTokenizer
Fast and memory-efficient library for WordPiece tokenization as it is used by BERT.
ml-rust/splintr
A high-performance tokenizer (BPE + SentencePiece) built with Rust with Python bindings, focused...
sanderland/script_tok
Code for the paper "BPE stays on SCRIPT"
U4RASD/r-bpe
R-BPE: Improving BPE-Tokenizers with Token Reuse
jmaczan/bpe-tokenizer
Byte-Pair Encoding tokenizer for training large language models on huge datasets