arthur-ai/bench
A tool for evaluating LLMs
This tool helps machine learning engineers and data scientists evaluate the performance of large language models (LLMs) for specific business applications. You provide your LLM outputs and reference answers, and it generates performance scores, helping you compare different models, prompts, or generation settings. It's designed for anyone deploying or integrating LLMs into production systems.
428 stars. No commits in the last 6 months.
Use this if you need to systematically compare different LLMs, optimize prompt engineering, or validate LLM outputs against specific business requirements before deployment.
Not ideal if you are looking for a general-purpose LLM experimentation platform rather than a focused evaluation and benchmarking tool.
Stars
428
Forks
42
Language
TypeScript
License
MIT
Category
Last pushed
May 10, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/arthur-ai/bench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems