darkautism/llmserver-rs
A Rust-based, OpenAI-style API server for large language models (LLMs)
This project helps developers and engineers integrate large language models (LLMs) into their applications, particularly on devices powered by the RK3588 chip. It takes raw text or audio input and, using an OpenAI-compatible API, outputs text responses from the LLM or transcriptions from audio. This tool is for those building AI-powered features for applications running on specific hardware.
Use this if you are a developer looking to deploy and interact with open-source LLMs and speech recognition models on hardware featuring the RK3588 chip, using an OpenAI-like API.
Not ideal if you are an end-user without programming experience or if you need to deploy LLMs on general-purpose cloud servers or consumer GPUs.
Stars
19
Forks
8
Language
Rust
License
—
Category
Last pushed
Mar 04, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/darkautism/llmserver-rs"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Higher-rated alternatives
trymirai/uzu
A high-performance inference engine for AI models
justrach/bhumi
⚡ Bhumi – The fastest AI inference client for Python, built with Rust for unmatched speed,...
lipish/llm-connector
LLM Connector - A unified interface for connecting to various Large Language Model providers
keyvank/femtoGPT
Pure Rust implementation of a minimal Generative Pretrained Transformer
ShelbyJenkins/llm_client
The Easiest Rust Interface for Local LLMs and an Interface for Deterministic Signals from...