aigc-apps/PertEval
[NeurIPS '24 Spotlight] PertEval: Unveiling Real Knowledge Capacity of LLMs via Knowledge-invariant Perturbations
This toolkit helps AI researchers and developers understand what an LLM truly "knows" by testing its knowledge in a very specific way. You feed it existing multiple-choice benchmark questions, and it generates slightly altered versions of these questions that shouldn't change the underlying knowledge required to answer. The output is a robust score of the LLM's real knowledge capacity and insights into why it might fail.
No commits in the last 6 months.
Use this if you need to rigorously evaluate the fundamental knowledge of large language models on close-ended benchmarks, going beyond surface-level accuracy to identify genuine understanding versus spurious correlations.
Not ideal if you are looking to evaluate LLMs on open-ended tasks, creative generation, or conversational ability, as this tool focuses specifically on probing knowledge via multiple-choice questions.
Stars
14
Forks
2
Language
Jupyter Notebook
License
Apache-2.0
Category
Last pushed
Oct 30, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/aigc-apps/PertEval"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
google-deepmind/long-form-factuality
Benchmarking long-form factuality in large language models. Original code for our paper...
gnai-creator/aletheion-llm-v2
Decoder-only LLM with integrated epistemic tomography. Knows what it doesn't know.
sandylaker/ib-edl
Calibrating LLMs with Information-Theoretic Evidential Deep Learning (ICLR 2025)
nightdessert/Retrieval_Head
open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality
MLD3/steerability
An open-source evaluation framework for measuring LLM steerability.