stalkermustang/llm-bulls-and-cows-benchmark
A mini-framework for evaluating LLM performance on the Bulls and Cows number guessing game, supporting multiple LLM providers.
This project helps evaluate how well different large language models (LLMs) can play the classic number-guessing game 'Bulls and Cows'. You input a list of LLMs you want to test, and it outputs detailed performance metrics like success rates, how many turns each model takes to win, and how often they fail to follow formatting rules. This is for researchers or practitioners who need to compare the reasoning and in-context learning abilities of various LLMs for decision-making tasks.
237 stars. No commits in the last 6 months.
Use this if you need to systematically benchmark and compare the logical reasoning and memory recall capabilities of different LLMs using a standardized, interactive game.
Not ideal if you are looking for a general-purpose LLM evaluation tool for a wide range of tasks beyond number-guessing games.
Stars
237
Forks
1
Language
HTML
License
MIT
Category
Last pushed
Jan 31, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/stalkermustang/llm-bulls-and-cows-benchmark"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)