robert-mcdermott/ollama-batch-cluster
Large Scale Batch Processing with Ollama
This project helps you process a large number of text prompts using an LLM on multiple Ollama servers and GPUs, much faster than a single setup. You provide a file with many prompts and the system returns individual response files or a combined output. This is ideal for researchers, data analysts, or content creators who need to generate many LLM responses efficiently.
No commits in the last 6 months.
Use this if you need to run a large batch of prompts through an Ollama-based LLM and want to use multiple GPUs or servers to speed up the process significantly.
Not ideal if you only have a few prompts to process or are not comfortable setting up and managing multiple Ollama instances and GPU configurations.
Stars
30
Forks
8
Language
Python
License
—
Category
Last pushed
Nov 25, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/robert-mcdermott/ollama-batch-cluster"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related models
anmolg1997/Multi-LoRA-Serve
Multi-adapter inference gateway — one base model, many LoRA adapters per-request,...
kimmmmyy223/llm-batch
🚀 Process JSON data in batches with `llm-batch`, leveraging sequential or parallel modes for...
Rohit2sali/vllm-multi-tenant-llm-gateway
This is vllm multi tenant large language model gateway. This system is created to serve lot of...