bigscience-workshop/petals
🌸 Run LLMs at home, BitTorrent-style. Fine-tuning and inference up to 10x faster than offloading
This project helps you run and customize powerful large language models (LLMs) like Llama 3.1 or Mixtral on your personal computer, even if you don't have super-expensive hardware. You provide a prompt or data for fine-tuning, and it generates text or a specialized model. It's for researchers, developers, or hobbyists who want to experiment with advanced AI models without needing a supercomputer.
9,997 stars. Used by 1 other package. No commits in the last 6 months. Available on PyPI.
Use this if you want to access and fine-tune massive language models for text generation or specific tasks using your existing computer, leveraging a distributed network of GPUs.
Not ideal if your data is highly sensitive and cannot be processed by a public network, or if you require guaranteed low-latency inference for real-time production systems without contributing GPU resources.
Stars
9,997
Forks
595
Language
Python
License
MIT
Category
Last pushed
Sep 07, 2024
Commits (30d)
0
Dependencies
18
Reverse dependents
1
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/bigscience-workshop/petals"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related models
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
sgl-project/sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
alibaba/MNN
MNN: A blazing-fast, lightweight inference engine battle-tested by Alibaba, powering...
xorbitsai/inference
Swap GPT for any LLM by changing a single line of code. Xinference lets you run open-source,...
tensorzero/tensorzero
TensorZero is an open-source stack for industrial-grade LLM applications. It unifies an LLM...