epwalsh/rust-dl-webserver
🦀 Example of serving deep learning models in Rust with batched prediction
This is an example for software engineers who need to deploy deep learning models that generate text, such as a GPT-2 model. It takes in text prompts and outputs generated text, efficiently handling multiple requests by processing them in batches, especially when a GPU is available. It's designed for developers building backend services for AI-powered applications.
No commits in the last 6 months.
Use this if you are a software engineer looking for a robust way to serve text generation models with good performance and stability.
Not ideal if you are a data scientist or end-user looking for a pre-built application to use directly without writing code.
Stars
34
Forks
5
Language
Rust
License
—
Category
Last pushed
Mar 09, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/epwalsh/rust-dl-webserver"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
tracel-ai/burn
Burn is a next generation tensor library and Deep Learning Framework that doesn't compromise on...
sonos/tract
Tiny, no-nonsense, self-contained, Tensorflow and ONNX inference
pykeio/ort
Fast ML inference & training for ONNX models in Rust
elixir-nx/ortex
ONNX Runtime bindings for Elixir
robertknight/rten
ONNX neural network inference engine