belladoreai/llama3-tokenizer-js
JS tokenizer for LLaMA 3 and LLaMA 3.1
This tool helps developers accurately count tokens in text when working with LLaMA 3 or LLaMA 3.1 language models. It takes plain text as input and outputs the corresponding token count, ensuring that applications using these models can precisely manage text length for features like input limits or cost estimation. It's designed for developers building web or Node.js applications that interact with LLaMA 3 models.
117 stars. No commits in the last 6 months. Available on npm.
Use this if you are a web or Node.js developer building an application that needs to accurately count tokens for LLaMA 3 or LLaMA 3.1 models directly in the browser or server-side JavaScript.
Not ideal if you are working with LLaMA 1, LLaMA 2, OpenAI, or Mistral models, or if you need to train a tokenizer.
Stars
117
Forks
6
Language
JavaScript
License
MIT
Category
Last pushed
Jul 28, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/belladoreai/llama3-tokenizer-js"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ludwig-ai/ludwig
Low-code framework for building custom LLMs, neural networks, and other AI models
withcatai/node-llama-cpp
Run AI models locally on your machine with node.js bindings for llama.cpp. Enforce a JSON schema...
mudler/LocalAI
:robot: The free, Open Source alternative to OpenAI, Claude and others. Self-hosted and...
zhudotexe/kani
kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)
SciSharp/LLamaSharp
A C#/.NET library to run LLM (🦙LLaMA/LLaVA) on your local device efficiently.