zkry/tiktoken.el
tiktoken.el is an Emacs Lisp port of OpenAI's tiktoken library for BPE tokenization
This tool helps Emacs users who are building or interacting with Large Language Model (LLM) applications. It takes plain text, like prompts or responses, and converts it into 'tokens'—the fundamental units LLMs process. This allows Emacs users to accurately count tokens, which is crucial for managing costs and understanding LLM input/output limits.
No commits in the last 6 months.
Use this if you are developing LLM applications within Emacs or need to precisely measure the token count of text for OpenAI models.
Not ideal if you are working with LLMs outside of the Emacs environment or do not need to perform BPE tokenization.
Stars
13
Forks
5
Language
Emacs Lisp
License
GPL-3.0
Category
Last pushed
Jan 03, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/zkry/tiktoken.el"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
dqbd/tiktokenizer
Online playground for OpenAPI tokenizers
pkoukk/tiktoken-go
go version of tiktoken
microsoft/Tokenizer
Typescript and .NET implementation of BPE tokenizer for OpenAI LLMs.
lenML/tokenizers
a lightweight no-dependency fork from transformers.js (only tokenizers)