fannie1208/FactTest
[ICML2025] "FactTest: Factuality Testing in Large Language Models with Finite-Sample and Distribution-Free Guarantees"
This tool helps researchers and AI practitioners systematically assess how truthful a Large Language Model (LLM) is when it generates text. You provide the LLM you want to test and a calibration dataset, and it produces a statistical measure of its factual accuracy, backed by strong statistical guarantees. It's designed for those who need to rigorously quantify and report the factuality of LLMs.
No commits in the last 6 months.
Use this if you need to scientifically test and report the factuality of an LLM with reliable, statistically-sound metrics.
Not ideal if you're looking for a simple, quick way to get a subjective sense of an LLM's general truthfulness without deep statistical analysis.
Stars
9
Forks
1
Language
Python
License
—
Category
Last pushed
May 29, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/fannie1208/FactTest"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
google-deepmind/long-form-factuality
Benchmarking long-form factuality in large language models. Original code for our paper...
gnai-creator/aletheion-llm-v2
Decoder-only LLM with integrated epistemic tomography. Knows what it doesn't know.
sandylaker/ib-edl
Calibrating LLMs with Information-Theoretic Evidential Deep Learning (ICLR 2025)
nightdessert/Retrieval_Head
open-source code for paper: Retrieval Head Mechanistically Explains Long-Context Factuality
MLD3/steerability
An open-source evaluation framework for measuring LLM steerability.