botisan-ai/gpt3-tokenizer
Isomorphic JavaScript/TypeScript Tokenizer for GPT-3 and Codex Models by OpenAI.
When working with OpenAI's GPT-3 or Codex models, you often need to understand how text is broken down into 'tokens' or convert tokens back into readable text. This tool takes your text and shows you how the AI model sees it (as tokens) or takes a list of tokens and reconstructs the original text. It's for developers building applications that integrate directly with OpenAI's GPT-3 or Codex APIs.
172 stars. No commits in the last 6 months.
Use this if you are a developer building a JavaScript or TypeScript application that needs to accurately count tokens or convert between text and tokens for OpenAI's GPT-3 or Codex models.
Not ideal if you are an end-user of a GPT-3 application and simply want to understand or optimize your prompts without coding.
Stars
172
Forks
16
Language
TypeScript
License
MIT
Category
Last pushed
Jan 27, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/botisan-ai/gpt3-tokenizer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
pkoukk/tiktoken-go
go version of tiktoken
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)