Notnaton/microllm
My own implementation to run inference on local LLM models
Microllm helps developers run large language models (LLMs) directly on their personal computers, rather than relying on cloud services. It takes a model file in GGUF format as input and allows you to perform basic inference tasks. This is for software developers and researchers who want to test or utilize LLMs privately or offline.
No commits in the last 6 months.
Use this if you are a developer looking for a straightforward, local way to run LLM inference without external dependencies.
Not ideal if you need a full-featured LLM application with advanced capabilities like fine-tuning, complex prompt engineering, or robust token generation out of the box.
Stars
8
Forks
1
Language
Python
License
AGPL-3.0
Category
Last pushed
Sep 03, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Notnaton/microllm"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
PaddlePaddle/FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
mlc-ai/mlc-llm
Universal LLM Deployment Engine with ML Compilation
skyzh/tiny-llm
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny...
ServerlessLLM/ServerlessLLM
Serverless LLM Serving for Everyone.
AXERA-TECH/ax-llm
Explore LLM model deployment based on AXera's AI chips