mtszkw/fast-torch
Comparing PyTorch, JIT and ONNX for inference with Transformers
This project helps machine learning engineers and data scientists efficiently compare different methods for deploying PyTorch Transformer models for inference. It takes a pre-trained sentiment analysis model and test text sequences, then measures and compares prediction speeds using standard PyTorch, TorchScript, and ONNX Runtime. The output is a clear performance comparison, indicating the fastest inference method for a given model.
No commits in the last 6 months.
Use this if you need to quickly evaluate and choose the most performant deployment strategy for your PyTorch Transformer models to optimize prediction speed.
Not ideal if you are looking for a general-purpose model training or a detailed guide on model quantization or pruning.
Stars
20
Forks
1
Language
Python
License
—
Category
Last pushed
Feb 22, 2021
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/mtszkw/fast-torch"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
openvinotoolkit/nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
huggingface/optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers...
NVIDIA/Megatron-LM
Ongoing research training transformer models at scale
huggingface/optimum-intel
🤗 Optimum Intel: Accelerate inference with Intel optimization tools
eole-nlp/eole
Open language modeling toolkit based on PyTorch