tiktoken-go and tiktoken-cli
These are ecosystem siblings: the Go binding provides language-specific access to OpenAI's tokenizer, while the CLI wrapper offers a language-agnostic interface to the same underlying tokenization library for use across different programming environments.
About tiktoken-go
pkoukk/tiktoken-go
go version of tiktoken
This is a Go language implementation of OpenAI's tiktoken library. It helps developers working with Go applications to accurately count and tokenize text for use with various OpenAI models, such as GPT-3.5 and GPT-4. You provide text or chat messages, and it outputs the corresponding token IDs and the total token count.
About tiktoken-cli
oelmekki/tiktoken-cli
Simple wrapper around tiktoken to use it in your favorite language.
This tool helps developers who are building applications using OpenAI's GPT API. It takes text input, like a user's prompt or a message, and outputs the exact number of tokens OpenAI will count for that text. This allows developers to accurately predict token usage and avoid costly API errors, regardless of the programming language they're using.
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work