toyaix/TritonLLM
LLM Inference via Triton (Flexible & Modular): Focused on Kernel Optimization using CUBIN binaries, Starting from gpt-oss Model
This project helps AI developers and researchers significantly speed up large language model (LLM) inference. It takes a pre-trained LLM, specifically gpt-oss models, and optimizes how quickly it generates text. The output is faster text generation, especially when running multiple queries at once, making it ideal for those building or deploying LLM-powered applications.
Use this if you are a developer or researcher looking to optimize the performance and reduce latency of your LLM applications, particularly when deploying gpt-oss models on NVIDIA GPUs for high-throughput scenarios.
Not ideal if you are an end-user without programming experience, or if you are working with non-gpt-oss models or hardware other than NVIDIA GPUs.
Stars
76
Forks
2
Language
Python
License
MIT
Category
Last pushed
Mar 11, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/toyaix/TritonLLM"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
alibaba/MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering...
xorbitsai/inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source,...
tensorzero/tensorzero
TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM...