godaai/llm-inference
Resources for Large Language Model Inference
This project provides a curated collection of resources for improving how large language models (LLMs) generate responses, making them faster and more efficient. It helps machine learning engineers and researchers understand and implement techniques like continuous batching and quantization. You'll find a glossary of key terms, a list of open-source software frameworks, and a selection of important academic papers, all focused on optimizing LLM performance.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher looking to understand, implement, or compare different methods for optimizing the speed and efficiency of large language model inference.
Not ideal if you are a non-technical user simply looking to use an existing LLM, or a developer interested in general-purpose machine learning libraries not specifically focused on LLM inference optimization.
Stars
17
Forks
3
Language
—
License
—
Category
Last pushed
Dec 29, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/godaai/llm-inference"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
thu-pacman/chitu
High-performance inference framework for large language models, focusing on efficiency,...
NotPunchnox/rkllama
Ollama alternative for Rockchip NPU: An efficient solution for running AI and Deep learning...
sophgo/LLM-TPU
Run generative AI models in sophgo BM1684X/BM1688
Deep-Spark/DeepSparkHub
DeepSparkHub selects hundreds of application algorithms and models, covering various fields of...
howard-hou/VisualRWKV
VisualRWKV is the visual-enhanced version of the RWKV language model, enabling RWKV to handle...