VectorInstitute/vector-inference
Efficient LLM inference on Slurm clusters.
This tool helps researchers and AI practitioners efficiently deploy and manage large language models (LLMs) on Slurm-managed computing clusters. You provide a model name, and it gives you a URL endpoint to send inference requests to, making it easy to generate text or responses from the model. This is ideal for anyone who needs to run LLM inference at scale within a shared cluster environment.
Use this if you need to quickly get an LLM inference server running on a Slurm cluster and get a live URL to send requests to.
Not ideal if you are working with a single machine or do not have access to a Slurm-managed cluster.
Stars
95
Forks
12
Language
Python
License
MIT
Category
Last pushed
Mar 09, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/VectorInstitute/vector-inference"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
PaddlePaddle/FastDeploy
High-performance Inference and Deployment Toolkit for LLMs and VLMs based on PaddlePaddle
mlc-ai/mlc-llm
Universal LLM Deployment Engine with ML Compilation
skyzh/tiny-llm
A course of learning LLM inference serving on Apple Silicon for systems engineers: build a tiny...
ServerlessLLM/ServerlessLLM
Serverless LLM Serving for Everyone.
AXERA-TECH/ax-llm
Explore LLM model deployment based on AXera's AI chips