llm-semantic-router/vllm-router
vLLM Router
Managing multiple large language models (LLMs) deployed with vLLM can be complex, as each model typically requires its own endpoint. This tool provides a single entry point to access various vLLM models. You send your requests to one unified interface, specifying which model to use, and it routes your request to the correct LLM. This is ideal for AI/ML engineers or DevOps teams deploying and managing LLM services.
No commits in the last 6 months.
Use this if you need to deploy and manage several vLLM-based language models and want a single, unified API endpoint for all of them.
Not ideal if you are running only a single vLLM model or if your application doesn't require a centralized routing layer for LLMs.
Stars
55
Forks
2
Language
Python
License
Apache-2.0
Category
Last pushed
Mar 11, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/llm-semantic-router/vllm-router"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.