mazzzystar/TurtleBench
TurtleBench: Evaluating Top Language Models via Real-World Yes/No Puzzles.
This project helps AI researchers and developers assess how well large language models (LLMs) can reason, specifically with yes/no questions. It takes real-world 'Turtle Soup' puzzles, which require logical deduction rather than factual knowledge, and evaluates an LLM's responses. The output is a clear, quantifiable score indicating how accurately the LLM answered these challenging puzzles, allowing for unbiased comparison of different models.
163 stars. No commits in the last 6 months.
Use this if you need an objective way to benchmark and compare the logical reasoning abilities of various large language models using real, user-generated puzzles.
Not ideal if you are looking to evaluate a language model's ability to recall factual information or generate creative text, as it focuses specifically on yes/no logical puzzles.
Stars
163
Forks
15
Language
Jupyter Notebook
License
Apache-2.0
Category
Last pushed
Oct 16, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/mazzzystar/TurtleBench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems