TiktokenSharp and SharpToken
About TiktokenSharp
aiqinxuancai/TiktokenSharp
Token calculation for OpenAI models, using `o200k_base` `cl100k_base` `p50k_base` encoding.
This is a C# library that helps developers accurately count tokens for text processed by OpenAI's large language models like GPT-3.5 and GPT-4. It takes a model name or encoding identifier as input, along with your text, and outputs the precise token count. This tool is for C# developers building applications that integrate with OpenAI APIs and need to manage token limits effectively.
About SharpToken
dmitry-brazhenko/SharpToken
SharpToken is a C# library for tokenizing natural language text. It's based on the tiktoken Python library and designed to be fast and accurate.
This is a C# library for developers building applications that interact with large language models (LLMs) like GPT or Claude. It takes natural language text and converts it into numerical "tokens" or vice-versa, which is essential for managing input length and processing text efficiently for AI models. Developers working with .NET applications who need to pre-process or post-process text for LLMs would use this.
Related comparisons
Scores updated daily from GitHub, PyPI, and npm data. How scores work