KevinLee1110/dynamic-batching
The official repo for the paper "Optimizing LLM Inference Throughput via Memory-aware and SLA-constrained Dynamic Batching"
This project helps optimize the performance of large language model (LLM) inference systems by dynamically adjusting how many requests are processed simultaneously. It takes your existing LLM serving infrastructure and, by continuously monitoring GPU memory and response time targets, automatically configures the optimal batch size. The end result is a significant boost in how many requests your LLM can handle per second and increased overall capacity, without changing your current setup. This is for infrastructure engineers or MLOps teams managing LLM deployments.
No commits in the last 6 months.
Use this if you are deploying large language models and need to maximize their throughput and capacity while consistently meeting service-level agreements for latency.
Not ideal if you are working with smaller models that are not memory-constrained or if your inference workloads are entirely static and predictable.
Stars
17
Forks
2
Language
—
License
Apache-2.0
Category
Last pushed
Mar 17, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/KevinLee1110/dynamic-batching"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
alibaba/MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering...
xorbitsai/inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source,...
tensorzero/tensorzero
TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM...