tatsu-lab/alpaca_eval
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.
This project helps evaluate how well your instruction-following language model (like a chatbot) performs compared to others. You input your model's responses to a set of instructions, and it provides a win-rate score against a reference model, indicating its quality. This tool is for AI researchers and developers who are building or fine-tuning large language models and need to quickly assess their performance.
1,957 stars. No commits in the last 6 months.
Use this if you are developing or fine-tuning instruction-following language models and need a fast, affordable, and highly correlated automatic evaluation method to guide your iteration cycles.
Not ideal if you need a definitive evaluation for high-stakes decisions like model release, as automatic evaluators can have biases and may not cover all potential risks.
Stars
1,957
Forks
305
Language
Jupyter Notebook
License
Apache-2.0
Category
Last pushed
Aug 09, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/tatsu-lab/alpaca_eval"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Related models
eth-sri/matharena
Evaluation of LLMs on latest math competitions
HPAI-BSC/TuRTLe
TuRTLe: A Unified Evaluation of LLMs for RTL Generation 🐢 (MLCAD 2025)
nlp-uoregon/mlmm-evaluation
Multilingual Large Language Models Evaluation Benchmark
haesleinhuepf/human-eval-bia
Benchmarking Large Language Models for Bio-Image Analysis Code Generation
ShuntaroOkuma/adapt-gauge-core
Measure LLM adaptation efficiency — how fast models learn from few examples