Yifan-Song793/GoodBadGreedy
The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinism
This project helps evaluate how different generation strategies impact the performance of large language models (LLMs) in various tasks. It takes in LLM outputs generated using different decoding methods (like greedy or sampling) and benchmark evaluation results, then reveals how consistent an LLM's performance is and when one method outperforms another. This is for researchers, MLOps engineers, and product managers who are integrating LLMs and need to understand their reliability and optimal configuration.
No commits in the last 6 months.
Use this if you need to understand the variability of LLM outputs and how different generation settings (like greedy vs. sampling) affect performance on specific benchmarks for tasks like reasoning or code generation.
Not ideal if you are looking for a general-purpose LLM evaluation framework that doesn't focus on non-determinism or best-of-N sampling strategies.
Stars
30
Forks
1
Language
Python
License
—
Category
Last pushed
Jul 17, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/Yifan-Song793/GoodBadGreedy"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
EvolvingLMMs-Lab/lmms-eval
One-for-All Multimodal Evaluation Toolkit Across Text, Image, Video, and Audio Tasks
vibrantlabsai/ragas
Supercharge Your LLM Application Evaluations 🚀
open-compass/VLMEvalKit
Open-source evaluation toolkit of large multi-modality models (LMMs), support 220+ LMMs, 80+ benchmarks
EuroEval/EuroEval
The robust European language model benchmark.
Giskard-AI/giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents