Rust LLM Infrastructure LLM Tools

Low-level Rust libraries and tools for building, running, and managing LLMs locally—including model merging, inference engines, tokenization, and architecture implementations. Does NOT include application frameworks, API clients, or higher-level orchestration platforms.

There are 98 rust llm infrastructure tools tracked. 3 score above 50 (established tier). The highest-rated is trymirai/uzu at 59/100 with 1,492 stars. 1 of the top 10 are actively maintained.

Get all 98 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=llm-tools&subcategory=rust-llm-infrastructure&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Tool	Score	Tier	Stars	Language
1	trymirai/uzu A high-performance inference engine for AI models	59	Established	1,492	Rust
2	justrach/bhumi ⚡ Bhumi – The fastest AI inference client for Python, built with Rust for...	54	Established	64	Python
3	lipish/llm-connector LLM Connector - A unified interface for connecting to various Large Language...	52	Established	13	Rust
4	keyvank/femtoGPT Pure Rust implementation of a minimal Generative Pretrained Transformer	49	Emerging	935	Rust
5	ShelbyJenkins/llm_client The Easiest Rust Interface for Local LLMs and an Interface for Deterministic...	45	Emerging	245	Rust
6	rustformers/llm [Unmaintained, see README] An ecosystem of Rust libraries for working with...	44	Emerging	6,152	Rust
7	luckenco/rsai Predictable development for unpredictable models. Let the compiler handle the chaos.	44	Emerging	9	Rust
8	npuichigo/openai_trtllm OpenAI compatible API for TensorRT LLM triton backend	43	Emerging	219	Rust
9	kreuzberg-dev/liter-llm Universal LLM API client — 142+ providers, 11 native language bindings,...	42	Emerging	4	Rust
10	darkautism/llmserver-rs A Rust-based, OpenAI-style API server for large language models (LLMs)	41	Emerging	19	Rust
11	microsoft/aici AICI: Prompts as (Wasm) Programs	41	Emerging	2,064	Rust
12	antirez/gguf-tools GGUF implementation in C as a library and a tools CLI program	41	Emerging	311	C
13	reinterpretcat/qwen3-rs An educational Rust project for exporting and running inference on Qwen3 LLM family	39	Emerging	40	Rust
14	haasonsaas/uranium High-security storage vault for Large Language Model (LLM) weights with...	39	Emerging	2	Rust
15	visualstudioblyat/yule Run AI models locally Prove what ran	37	Emerging	2	Rust
16	mplekh/rust-microgpt Port of Andrej Karpathy's python microGPT to Rust	37	Emerging	36	Rust
17	rosarp/llm-lsp Language Server Protocol for accessing Large Language Models	37	Emerging	42	Rust
18	yigitkonur/cli-batch-requester 10K+ req/s batch API client for LLM endpoints — Rust, async, load-balanced	37	Emerging	19	Rust
19	tmetsch/rusty_llm Rust based AI LLM inference service	36	Emerging	6	Rust
20	Michael-A-Kuykendall/schoolmarm Production-grade GBNF grammar-constrained decoding for LLMs. Zero...	36	Emerging	1	Rust
21	InfraWhisperer/llmtop htop for your LLM inference cluster	36	Emerging	6	Go
22	Lallapallooza/gpt.rs Rust LLM playground: build, train, generate on pluggable backends	36	Emerging	15	Rust
23	GammaTauAI/opentau Using Large Language Models for Repo-wide Type Prediction	34	Emerging	114	Rust
24	Mattbusel/llm_affector An async Rust library for LLM-based content analysis, providing...	33	Emerging	5	Rust
25	fabriziopfannl/llm-autobatch Turn single LLM calls into fast micro-batches. Rust core, Python API.	33	Emerging	4	Rust
26	brontoguana/ktop Terminal system resource monitor for hybrid LLM workloads	32	Emerging	64	Rust
27	rodmarkun/flyllm A Rust library for unifying LLM backends as an abstraction layer with load...	32	Emerging	28	Rust
28	jondot/awesome-rust-llm 🦀 A curated list of Rust tools, libraries, and frameworks for working with...	30	Emerging	543	—
29	yybit/pllm Portable LLM - A rust library for LLM inference	30	Emerging	11	Rust
30	richardanaya/epistemology A simple and clear way of hosting llama.cpp as a private HTTP API using Rust	30	Emerging	26	Rust
31	chenhunghan/mlx-training-rs A CLI in Rust to generate synthetic data for MLX friendly training	30	Emerging	25	Rust
32	codito/arey Simple large language model playground app	30	Emerging	6	Rust
33	sizzlecar/ferrum-infer-rs Rust-native LLM inference engine. Single binary, no Python. Chat locally or...	29	Experimental	1	Rust
34	jaggederest/locque Locque, a dependently-typed LLM first programming language	29	Experimental	3	Haskell
35	SundryAPI/sundry Sundry is an intelligent context provider API designed specifically for...	29	Experimental	9	Rust
36	okayasl/normy Ultra-fast, zero-copy text normalization for Rust NLP pipelines & tokenizers	29	Experimental	3	Rust
37	pwh-pwh/couplet_gen use rust to generate couplet	28	Experimental	8	Rust
38	nkypy/candle-rwkv RWKV models and examples powered by candle.	28	Experimental	24	Rust
39	proj-airi/candle-examples 🦀 Rust powered LLM, Whisper, Embedding inference, backed by 🤗 candle from HuggingFace	28	Experimental	30	Rust
40	antoineMoPa/rust-text-experiments Tiny LLM in rust / candle	28	Experimental	2	Rust
41	netdur/hugind vLLM for poor GPUs	27	Experimental	6	Rust
42	ahoylabs/gguf.js A Javascript library (with Typescript types) to parse metadata of GGML based...	27	Experimental	51	TypeScript
43	paiml/apr-cookbook Examples of .apr format models	27	Experimental	5	Rust
44	Mattbusel/llm-diff Output diffing and versioning for LLM outputs — semantic diff, version...	25	Experimental	2	Rust
45	aprxi/talu Talu is a single-binary, local-first LLM runtime with a Zig core and...	25	Experimental	7	Zig
46	AspadaX/secretary Robustly create/extract structural data with LLMs	25	Experimental	5	Rust
47	hyperpolymath/patallm-gallery Gallery of LLM patterns and implementations	25	Experimental	2	Rust
48	usemarbles/langmail Email preprocessing for LLMs. Fast, typed, Rust-powered.	25	Experimental	1	Rust
49	encomers/nn-yandex-foundation Unified library for working with Yandex Foundation Models. Provides a simple...	25	Experimental	7	Rust
50	matthewhaynesonline/phile Single file llm, but in _rust_. phi + file = phile.	24	Experimental	3	Rust
51	t81dev/ternary-tools file(1) of the ternary age — balanced-ternary-aware GGUF inspector and...	24	Experimental	1	Rust
52	wassemgtk/llm-training-rust llm training rust	24	Experimental	1	Rust
53	TomOstt/BlueOS GPU-first LLM inference runtime in Rust + CUDA. Tiered virtual VRAM,...	23	Experimental	1	Rust
54	blueheron786/cpu_llm A lightweight CPU-friendly neural language model from scratch, with hybrid...	23	Experimental	2	Rust
55	msk/lumine A high-level Rust interface for language models powered by the Candle ML...	23	Experimental	1	Rust
56	lspecian/crabinfer Safe, fast, memory-aware on-device LLM inference SDK for iOS — built in Rust...	23	Experimental	2	Rust
57	Defilan/gguf-parser A Rust library and CLI for parsing GGUF model file headers — extract...	22	Experimental	1	Rust
58	HelgeSverre/sema A Lisp with first-class LLM primitives, implemented in Rust	22	Experimental	1	Rust
59	defai-digital/ax-engine Mac-native Rust inference engine for running larger local GGUF models with...	22	Experimental	—	Rust
60	coconut-os/coconutOS Rust microkernel for GPU-isolated AI inference	22	Experimental	1	Rust
61	greysquirr3l/heretic-rs Abliterate LLMs in pure Rust — zero Python, single static binary, runs on Colab	22	Experimental	—	Rust
62	santino-research/spell A Programming Language Designed for Large Language Models	22	Experimental	3	Rust
63	petlukk/Cougar Fast, dependency-free LLM engine in Rust with custom SIMD kernels	22	Experimental	—	Rust
64	cukas/KERNlang The language LLMs think in. Write one .kern file, ship 7 targets. 70% fewer tokens	22	Experimental	—	TypeScript
65	JuliaMerz/pantry Actor based multi-llm registry + runner.	22	Experimental	20	Rust
66	menezis-ai/LDSI White-box LLM stability benchmark using Kolmogorov complexity, Shannon...	21	Experimental	—	Rust
67	yarenty/modelmux ModelMux is a high-performance Rust proxy server that seamlessly converts...	21	Experimental	—	Rust
68	samkeen/llm-bridge Rust SDK for interacting with various Large Language Model (LLM) APIs	21	Experimental	4	Rust
69	tzervas/axolotl-rs YAML-driven configurable fine-tuning toolkit for LLMs in Rust	21	Experimental	—	Rust
70	chongliujia/fermi-infer The Rust-native inference engine for Small Language Models (SLMs), Run...	21	Experimental	—	Rust
71	qwrtgvdsdf/ternary-tools 🔍 Explore and validate GGUF files effortlessly with ternary-tools, a...	21	Experimental	—	Rust
72	PCfVW/candle-mi Mechanistic interpretability for language models in Rust, built on candle	21	Experimental	—	Rust
73	StepfenShawn/ferris-grad Pytorch-like autograd engine in Rust.	21	Experimental	3	Rust
74	TheRadDani/VectorPrime VectorPrime takes a model file and your hardware, then finds the fastest way...	21	Experimental	—	Rust
75	srijitiyer/alloy A fast Rust CLI for LLM model merging, diffing, and conversion. 10 merge...	19	Experimental	4	Rust
76	tauseefk/streamformers Wrap Rustformers' LLM inference in a stream.	18	Experimental	2	Rust
77	text-yoga/ask WIP browser-based LLM question/answering for the web	18	Experimental	2	Rust
78	ramendrasingla/ml_algorithms_in_rust Creating Machine Learning and Deep Learning Algorithms in Rust	17	Experimental	1	Rust
79	cjroth/neuroscope Real-time "x-ray vision" into LLMs' minds	17	Experimental	3	Rust
80	rhi-zone/sketchpad Deep learning inference in pure Rust using Burn. Image generation (SD, SDXL,...	16	Experimental	1	Rust
81	eren23/synapse Modular LLM inference engine in Rust + Zig SIMD kernels. Runs on desktop...	14	Experimental	—	Rust
82	sanggi-wjg/LLML LLML — Language for Large Model Logic. A programming language optimized for...	14	Experimental	—	Rust
83	Plarturer/llm-distributed-inference High-performance distributed inference engine for LLMs using Rust and CUDA.	14	Experimental	—	Rust
84	danielgrittner/llama2-rs LLaMA2 + Rust	13	Experimental	13	Rust
85	kn0sys/adamo Rust LLM proof-of-concept	13	Experimental	—	Rust
86	kmolerov/llm-temp-scale llm-temp-scale is a multiplatform library for normalizing and converting a...	13	Experimental	—	Rust
87	lipish/llm-providers A unified source of truth for LLM providers, models, pricing, and...	13	Experimental	—	Rust
88	JuniMay/llm.rs An attempt to migrate Karpathy's llm.c to safe rust.	13	Experimental	13	Rust
89	reinterpretcat/zero-depend-pub An educational Rust workspace featuring zero-dependency crates built using...	13	Experimental	—	Rust
90	zTgx/transformer-rust Transformer With Rust & Candle	13	Experimental	—	Rust
91	AshtonVaughan/prismllm Any model. Any hardware. Any size. — Hardware-agnostic LLM inference with...	13	Experimental	—	Rust
92	llmprogram/llmprogram-rs llmprogram is a Rust crate that provides a structured and powerful way to...	13	Experimental	—	Rust
93	gicrisf/microgpt-candle-rs Rust implementation of Karpathy's Microgpt	13	Experimental	—	Rust
94	mrcsparker/guanaco Run local LLMs in Ruby	12	Experimental	1	Rust
95	mbn-code/rusty-OLLAMA read ollama LLM stream using rust	11	Experimental	—	Rust
96	AspadaX/dim Use LLMs for effective and refined vectorizations.	11	Experimental	—	Rust
97	InftyAI/PUMA Aim to be a lightweight, high-performance inference engine for local AI.	11	Experimental	—	Rust
98	Artur-Sulej/ai_devs My Rust solutions to exercises from the AI_Devs course	10	Experimental	1	Rust

Comparisons in this category

llm-connector and llm (52 vs 44) llm-connector and llm_client (52 vs 45) llm-connector and llm-lsp (52 vs 37) llm-connector and llmserver-rs (52 vs 41) llm-connector and liter-llm (52 vs 42)