npuichigo/openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
This project helps developers integrate custom large language models (LLMs) deployed with NVIDIA's TensorRT-LLM and Triton Inference Server into applications that expect an OpenAI-compatible API. It takes your deployed TensorRT-LLM model and makes it accessible as if it were an OpenAI model, outputting generated text responses. AI/ML engineers and application developers who build with LLMs will find this useful.
219 stars. No commits in the last 6 months.
Use this if you need to expose your high-performance TensorRT-LLM deployments through an OpenAI-like API, especially when integrating with tools like LangChain that are designed for OpenAI's interface.
Not ideal if you are looking for a pre-trained LLM or do not have a TensorRT-LLM model already deployed with Triton Inference Server.
Stars
219
Forks
30
Language
Rust
License
MIT
Category
Last pushed
Aug 01, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/npuichigo/openai_trtllm"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
trymirai/uzu
A high-performance inference engine for AI models
justrach/bhumi
⚡ Bhumi – The fastest AI inference client for Python, built with Rust for unmatched speed,...
lipish/llm-connector
LLM Connector - A unified interface for connecting to various Large Language Model providers
keyvank/femtoGPT
Pure Rust implementation of a minimal Generative Pretrained Transformer
ShelbyJenkins/llm_client
The Easiest Rust Interface for Local LLMs and an Interface for Deterministic Signals from...