belladoreai/llama-tokenizer-js
JS tokenizer for LLaMA 1 and 2
This is a tool for developers building web applications or Node.js services that use LLaMA 1 or LLaMA 2 language models. It takes text input and accurately tells you how many "tokens" it contains, or converts tokens back into human-readable text. Developers use this to manage the length of user input or model responses, ensuring they fit within LLaMA's context limits.
363 stars. No commits in the last 6 months.
Use this if you are a web developer or Node.js developer working with LLaMA 1 or LLaMA 2 models and need to accurately count tokens directly in the user's browser or your server without relying on external API calls.
Not ideal if you are working with LLaMA 3 models (there's a separate tool for that) or non-LLaMA models like OpenAI's GPT series, as the tokenization will be incompatible and inaccurate.
Stars
363
Forks
24
Language
JavaScript
License
MIT
Category
Last pushed
Jun 27, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/belladoreai/llama-tokenizer-js"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ludwig-ai/ludwig
Low-code framework for building custom LLMs, neural networks, and other AI models
withcatai/node-llama-cpp
Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema...
mudler/LocalAI
:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and...
zhudotexe/kani
kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)
SciSharp/LLamaSharp
A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.