OpenGenerativeAI/llm-colosseum
Benchmark LLMs by fighting in Street Fighter 3! The new way to evaluate the quality of an LLM
This project helps evaluate and compare different Large Language Models (LLMs) by having them compete in the video game Street Fighter III. You provide the LLMs you want to test, and the system shows which one performs best based on criteria like speed, smart decision-making, and adaptability. It's designed for researchers or practitioners who need to assess LLM capabilities beyond standard benchmarks.
1,467 stars. No commits in the last 6 months.
Use this if you need a dynamic, real-time method to benchmark the decision-making, speed, and strategic capabilities of different LLMs in a complex, adversarial environment.
Not ideal if you are solely interested in traditional linguistic or factual accuracy benchmarks for LLMs, as this focuses on game-play performance.
Stars
1,467
Forks
178
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Mar 21, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/OpenGenerativeAI/llm-colosseum"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems