mit-han-lab/TinyChatEngine
TinyChatEngine: On-Device LLM Inference Library
This project lets you run advanced AI chatbots, like those that write code or describe images, directly on your laptop, car, or robot without needing an internet connection. It takes a large language model and compresses it so it can process your text or image inputs locally, giving you instant replies and better privacy. This is for developers or hobbyists who want to integrate powerful AI features into on-device applications.
944 stars. No commits in the last 6 months.
Use this if you are a developer looking to embed large language model (LLM) or vision language model (VLM) capabilities directly into an application or device for real-time, private, and offline AI interactions.
Not ideal if you are an end-user without programming experience or if you primarily rely on cloud-based AI services with less concern for local execution and data privacy.
Stars
944
Forks
95
Language
C++
License
MIT
Category
Last pushed
Jul 04, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/mit-han-lab/TinyChatEngine"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
jundot/omlx
LLM inference server with continuous batching & SSD caching for Apple Silicon — managed from the...
josStorer/RWKV-Runner
A RWKV management and startup tool, full automation, only 8MB. And provides an interface...
jordanhubbard/nanolang
A tiny experimental language designed to be targeted by coding LLMs
waybarrios/vllm-mlx
OpenAI and Anthropic compatible server for Apple Silicon. Run LLMs and vision-language models...
akivasolutions/tightwad
Pool your CUDA + ROCm GPUs into one OpenAI-compatible API. Speculative decoding proxy gives you...