GAIR-NLP/benbench
Benchmarking Benchmark Leakage in Large Language Models
This project helps evaluate if large language models (LLMs) have been trained on specific benchmark datasets, which can make their performance seem better than it is. It takes an LLM and a benchmark dataset as input and outputs an analysis of potential data leakage, including a 'Benchmark Transparency Card' to understand the model's training data. This is for researchers, academics, or anyone evaluating and comparing the true capabilities of different LLMs.
No commits in the last 6 months.
Use this if you need to accurately assess if an LLM's high performance on a mathematical reasoning benchmark is due to genuine capability or because it was trained on the benchmark's data.
Not ideal if you are looking for a tool to develop new LLM architectures or improve existing model performance on specific tasks.
Stars
60
Forks
3
Language
JavaScript
License
—
Category
Last pushed
May 20, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/GAIR-NLP/benbench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems