AI21Labs/ai21-tokenizer
AI21's Jamba models tokenizers
This tool helps developers prepare text for use with AI21's Jamba AI models by converting human-readable text into numerical tokens that the models understand. It takes text strings as input and outputs a list of numerical IDs, or vice-versa. AI/ML engineers and data scientists building applications with AI21's Jamba models will use this.
Use this if you are a developer working with AI21's Jamba models and need to convert text into token IDs for model input or interpret model outputs back into human language.
Not ideal if you are not using AI21's Jamba models or are looking for a general-purpose natural language processing library.
Stars
33
Forks
4
Language
Python
License
Apache-2.0
Category
Last pushed
Oct 27, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/AI21Labs/ai21-tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
pkoukk/tiktoken-go
go version of tiktoken
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)