LowinLi/fastgpt
⚡ boost inference speed of GPT models in transformers by onnxruntime
This tool helps developers who are building applications that use GPT-based models for text generation. It allows you to convert these models into a more efficient format, specifically ONNX, which significantly speeds up how quickly your application can generate text, especially on CPU hardware. You input an existing GPT model from the Hugging Face Transformers library and get back a faster, optimized version that seamlessly integrates into your current text generation code.
No commits in the last 6 months. Available on PyPI.
Use this if you are a developer using Hugging Face Transformers for text generation with GPT models and need to boost inference speed and reduce resource consumption on CPU.
Not ideal if you are not a developer, or if you are not using GPT models from the Hugging Face Transformers library for text generation tasks.
Stars
52
Forks
4
Language
Python
License
MIT
Category
Last pushed
Aug 20, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/LowinLi/fastgpt"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
openvinotoolkit/nncf
Neural Network Compression Framework for enhanced OpenVINO™ inference
huggingface/optimum
🚀 Accelerate inference and training of 🤗 Transformers, Diffusers, TIMM and Sentence Transformers...
NVIDIA/Megatron-LM
Ongoing research training transformer models at scale
huggingface/optimum-intel
🤗 Optimum Intel: Accelerate inference with Intel optimization tools
eole-nlp/eole
Open language modeling toolkit based on PyTorch