ShannonAI/service-streamer
Boosting your Web Services of Deep Learning Applications.
This tool helps developers make their deep learning web services faster and more efficient. It takes individual user requests, groups them into 'mini-batches,' and feeds them to the deep learning model. This process allows the model, especially on GPUs, to handle many requests concurrently, significantly boosting the service's overall speed and responsiveness. It's for machine learning engineers and developers who deploy deep learning models as web services.
1,244 stars. No commits in the last 6 months.
Use this if you are deploying a deep learning model as a web service and want to increase its speed and throughput, especially when using GPUs.
Not ideal if your application doesn't involve deep learning models, or if you are not deploying a web service that requires high-performance inference.
Stars
1,244
Forks
187
Language
Python
License
Apache-2.0
Category
Last pushed
May 13, 2021
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/ml-frameworks/ShannonAI/service-streamer"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
modelscope/modelscope
ModelScope: bring the notion of Model-as-a-Service to life.
basetenlabs/truss
The simplest way to serve AI/ML models in production
Lightning-AI/LitServe
A minimal Python framework for building custom AI inference servers with full control over...
deepjavalibrary/djl-serving
A universal scalable machine learning model deployment solution
tensorflow/serving
A flexible, high-performance serving system for machine learning models