srikanth235/benchllama
Benchmark your local LLMs.
This tool helps AI developers and researchers choose the best local large language models (LLMs) for specific tasks. It takes an LLM served via Ollama and an optional dataset, then outputs performance and quality metrics, including pass@k for coding models. It's designed for individuals managing and optimizing local LLM deployments.
No commits in the last 6 months. Available on PyPI.
Use this if you need to compare and select the most effective local LLMs for your applications, especially when evaluating code generation capabilities across different programming languages.
Not ideal if you are working with cloud-based LLMs or models not served through Ollama, or if your primary focus is on fine-tuning models rather than comparative benchmarking.
Stars
53
Forks
3
Language
Python
License
MIT
Category
Last pushed
Aug 26, 2024
Commits (30d)
0
Dependencies
5
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/srikanth235/benchllama"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)