aallam/ktoken
Kotlin multiplatform BPE tokenizer library for OpenAI models
When working with OpenAI models, this tool helps you convert text into numerical 'tokens' and back again. It takes your input text and provides the token representation, or takes tokens and reconstructs the original text. Developers building applications with OpenAI's language models will use this to manage how text is processed and understood by the AI.
Use this if you are a developer integrating OpenAI models into a Kotlin-based application and need to handle text tokenization precisely.
Not ideal if you are an end-user of an AI application or if you're working with AI models other than OpenAI's.
Stars
41
Forks
3
Language
Kotlin
License
MIT
Category
Last pushed
Mar 08, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/aallam/ktoken"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers
pkoukk/tiktoken-go
go version of tiktoken
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)