Infini-AI-Lab/TriForce
[COLM 2024] TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
This project dramatically speeds up how quickly large language models generate very long text, such as detailed reports or lengthy articles. It takes your prompt and a long-context Llama model, and outputs text generated significantly faster without losing quality. This is ideal for anyone who regularly uses large language models for generating extensive written content and needs to accelerate their workflows.
277 stars. No commits in the last 6 months.
Use this if you need to generate extremely long text sequences using Llama-based models and want to do it much faster, especially with limited GPU resources.
Not ideal if you are working with short text generations, non-Llama models, or if your hardware lacks sufficient PCIe bandwidth for optimal offloading performance.
Stars
277
Forks
17
Language
Python
License
—
Category
Last pushed
Aug 31, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/Infini-AI-Lab/TriForce"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
sgl-project/SpecForge
Train speculative decoding models effortlessly and port them smoothly to SGLang serving.
structuredllm/syncode
Efficient and general syntactical decoding for Large Language Models
SafeAILab/EAGLE
Official Implementation of EAGLE-1 (ICML'24), EAGLE-2 (EMNLP'24), and EAGLE-3 (NeurIPS'25).
romsto/Speculative-Decoding
Implementation of the paper Fast Inference from Transformers via Speculative Decoding, Leviathan...
hao-ai-lab/JacobiForcing
Jacobi Forcing: Fast and Accurate Diffusion-style Decoding