oelmekki/tiktoken-cli
Simple wrapper around tiktoken to use it in your favorite language.
This tool helps developers who are building applications using OpenAI's GPT API. It takes text input, like a user's prompt or a message, and outputs the exact number of tokens OpenAI will count for that text. This allows developers to accurately predict token usage and avoid costly API errors, regardless of the programming language they're using.
No commits in the last 6 months.
Use this if you are building an application with OpenAI's GPT API in a language other than Python and need to accurately count tokens for user prompts or generated content.
Not ideal if you are working with a Python-based application, as you can directly use the 'tiktoken' library, or if you need to calculate tokens for many small messages individually, which can impact performance.
Stars
7
Forks
3
Language
Python
License
—
Category
Last pushed
Jul 08, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/oelmekki/tiktoken-cli"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Higher-rated alternatives
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers
pkoukk/tiktoken-go
go version of tiktoken
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)