Model Inference Serving ML Frameworks

Tools and frameworks for deploying, serving, and scaling machine learning models in production environments. Includes model servers, inference optimization, batching, and multi-model serving orchestration. Does NOT include model training frameworks, hyperparameter tuning, or general MLOps platforms.

There are 53 model inference serving frameworks tracked. 3 score above 70 (verified tier). The highest-rated is modelscope/modelscope at 77/100 with 8,784 stars. 3 of the top 10 are actively maintained.

Get all 53 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=ml-frameworks&subcategory=model-inference-serving&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Framework	Score	Tier	Stars	Language
1	modelscope/modelscope ModelScope: bring the notion of Model-as-a-Service to life.	77	Verified	8,784	Python
2	basetenlabs/truss The simplest way to serve AI/ML models in production	76	Verified	1,125	Python
3	Lightning-AI/LitServe A minimal Python framework for building custom AI inference servers with...	72	Verified	3,812	Python
4	deepjavalibrary/djl-serving A universal scalable machine learning model deployment solution	59	Established	248	Java
5	tensorflow/serving A flexible, high-performance serving system for machine learning models	57	Established	6,349	C++
6	labmlai/labml 🔎 Monitor deep learning model training and hardware usage from your mobile phone 📱	53	Established	2,300	Python
7	tobegit3hub/simple_tensorflow_serving Generic and easy-to-use serving service for machine learning models	51	Established	758	JavaScript
8	awslabs/multi-model-server Multi Model Server is a tool for serving neural net models for inference	51	Established	1,025	Java
9	ShannonAI/service-streamer Boosting your Web Services of Deep Learning Applications.	49	Emerging	1,244	Python
10	OrderLab/TrainCheck An Observability Framework for AI Training	49	Emerging	66	Python
11	iitzco/tfserve Serve TF models simple and easy as an HTTP API	49	Emerging	36	Python
12	polyaxon/sdks Polyaxon Clients & Langange SDKS	49	Emerging	14	Java
13	VertaAI/modeldb Open Source ML Model Versioning, Metadata, and Experiment Management	49	Emerging	1,744	Java
14	jrieke/traingenerator 🧙 A web app to generate template code for machine learning	48	Emerging	1,381	Python
15	ELS-RD/transformer-deploy Efficient, scalable and enterprise-grade CPU/GPU inference server for 🤗...	46	Emerging	1,688	Python
16	spotify/zoltar Common library for serving TensorFlow, XGBoost and scikit-learn models in production.	46	Emerging	142	Java
17	Angel-ML/serving A stand alone industrial serving system for angel.	45	Emerging	66	Java
18	reacher-z/gpu-monitor Lightweight NVIDIA GPU monitor — alerts on Slack/Discord/Telegram/20...	45	Emerging	138	Python
19	zooniverse/bajor Azure Batch Job Runner - BaJoR	43	Emerging	2	Python
20	feast-dev/feast-java-old Feast Java Components	41	Emerging	13	Java
21	CODAIT/max-central-repo Central Repository of Model Asset Exchange project. This repository contains...	41	Emerging	218	—
22	alvarobartt/serving-pytorch-models Serving PyTorch models with TorchServe :fire:	41	Emerging	103	Jupyter Notebook
23	rai-project/mlmodelscope MLModelScope is an open source, extensible, and customizable platform to...	40	Emerging	50	JavaScript
24	flipkart-incubator/Hunch Hunch allows users to turn arbitrary machine learning models built using...	40	Emerging	14	Python
25	mKaloer/TFServingCache Distributed model cache for TF Serving	39	Emerging	25	Go
26	ZhigaMason/monitorch A plug-and-use python module to monitor neural network learning.	38	Emerging	6	Python
27	BeyonderXX/tensorflow-serving-tutorial A tutorial of building tensorflow serving service from scratch	38	Emerging	89	C++
28	ParagGhatage/ZeroML ZeroML is a visual-first, end-to-end machine learning platform that lets you...	36	Emerging	12	TypeScript
29	mme/vergeml Machine Learning Environment - alpha version	36	Emerging	339	Python
30	Ifihan/blazerpc A lightweight, framework-agnostic gRPC library for serving machine learning...	35	Emerging	6	Python
31	ovh/serving-runtime Exposes a serialized machine learning model through a HTTP API.	35	Emerging	13	Java
32	kemingy/batching Dynamic Batching for Deep Learning Serving	35	Emerging	9	Go
33	huggingbench/huggingbench Find the optimal model serving solution for 🤗 Hugging Face models 🚀	33	Emerging	45	Python
34	redis-applied-ai/redis-feast-gcp A demo of Redis Enterprise as the Online Feature Store deployed on GCP with...	32	Emerging	15	Jupyter Notebook
35	alvarobartt/tensorflow-serving-streamlit TensorFlow Serving + Streamlit! :sparkles::framed_picture:	32	Emerging	24	Dockerfile
36	tradingAI/runner Job runner for tbase experiments	30	Emerging	2	Go
37	entrpn/serving-model-cards Collection of OSS models that are containerized into a serving container	27	Experimental	16	Python
38	prabhuomkar/bitbeast Experiments with Model Training, Deployment & Monitoring	27	Experimental	40	Python
39	galafis/realtime-ml-serving-api High-performance ML model serving API built with Go and Python, featuring...	26	Experimental	1	Python
40	kazuki-kanaya/obsern Lightweight CLI-based monitoring and notifications for long-running ML jobs....	24	Experimental	1	TypeScript
41	GAISSA-UPC/energy-ml-serving Energy consumption of ML inference with Runtime Engines	21	Experimental	3	Jupyter Notebook
42	mcp-tool-shop-org/backprop CLI-first ML trainer with intelligent resource governance — timeboxed runs,...	21	Experimental	—	TypeScript
43	aakashns/servefastai Serve FastAI models and get a web-based UI with a single line of code	19	Experimental	29	JavaScript
44	danielschulz/aiModelsAtScaleOnRestfulJeeSvcs Delivery Excellence, DevOps: Cloud-native Deployments of Data Science Models...	17	Experimental	1	Java
45	North-Shore-AI/crucible_adversary Adversarial testing and robustness evaluation for the Crucible framework	17	Experimental	—	Elixir
46	jonychoi/neuralverse Beyond the State of the Arts: Share more, Compare more, Edit Easy, Create...	17	Experimental	1	JavaScript
47	North-Shore-AI/crucible_framework CrucibleFramework: A scientific platform for LLM reliability research on the BEAM	17	Experimental	—	Elixir
48	bluebell2505/qrucible Qrucible is a real‑world–aligned, decision‑support system for early‑stage...	13	Experimental	—	Jupyter Notebook
49	A-SHOJAEI/adaptive-inference-router-with-cascade-serving A research-grade adaptive inference routing system that learns to...	13	Experimental	—	Python
50	rsj-cs/distributed-pipeline-capstone Design and optimization of a scalable distributed data processing pipeline...	13	Experimental	—	—
51	scattering-ai/MLDrop MLDrop model serving for Pytorch	11	Experimental	3	—
52	anasLearn/TensorFlow-Serving-API A docker image to serve TensorFlow models through a REST API	11	Experimental	—	C
53	Charl-AI/bmon CLI tool for monitoring system metrics and diagnosing bottlenecks in...	10	Experimental	2	Rust

Comparisons in this category

serving and simple_tensorflow_serving (57 vs 51) simple_tensorflow_serving and tfserve (51 vs 49)