yinzhangyue/SelfAware
Do Large Language Models Know What They Don’t Know?
This project provides a dataset and tools to assess if large language models (LLMs) can recognize when they don't have the answer to a question. It takes a list of questions (some answerable, some not) and an LLM's responses, then evaluates how accurately the LLM identifies unanswerable questions. The primary user is anyone working with or evaluating the reliability and knowledge boundaries of AI chatbots and conversational agents.
102 stars. No commits in the last 6 months.
Use this if you need to objectively measure an AI model's ability to 'know what it doesn't know' and avoid making up incorrect information.
Not ideal if you're looking for a tool to train or fine-tune an LLM, as this project focuses on evaluation rather than model development.
Stars
102
Forks
5
Language
Python
License
Apache-2.0
Category
Last pushed
Nov 08, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/yinzhangyue/SelfAware"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
MadryLab/context-cite
Attribute (or cite) statements generated by LLMs back to in-context information.
microsoft/augmented-interpretable-models
Interpretable and efficient predictors using pre-trained language models. Scikit-learn compatible.
Trustworthy-ML-Lab/CB-LLMs
[ICLR 25] A novel framework for building intrinsically interpretable LLMs with...
poloclub/LLM-Attributor
LLM Attributor: Attribute LLM's Generated Text to Training Data
THUDM/LongCite
LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA