harleyszhang/llm_counts
llm theoretical performance analysis tools and support params, flops, memory and latency analysis.
This tool helps AI engineers and researchers understand the theoretical performance limits of large language models (LLMs) on different GPUs. You input an LLM name, GPU type, and various parallelism and batch size settings, and it outputs detailed analysis of parameters, computational load (FLOPS), memory usage, and latency during both prefill and decode stages. It's designed for those optimizing LLM deployments.
115 stars. No commits in the last 6 months.
Use this if you need to predict an LLM's performance and resource consumption on various hardware and parallelism configurations before actual deployment, helping you choose the most efficient setup.
Not ideal if you are looking for actual real-world inference benchmarks from a deployed system, as this provides theoretical analysis rather than empirical measurements.
Stars
115
Forks
10
Language
Python
License
—
Category
Last pushed
Jul 11, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/harleyszhang/llm_counts"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
alibaba/MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering...
xorbitsai/inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source,...
tensorzero/tensorzero
TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM...