lifan-yuan/PLMCalibration
Code for ACL 2023 paper "A Close Look into the Calibration of Pre-trained Language Models"
This project helps researchers and developers understand how reliable the predictions of Pre-trained Language Models (PLMs) are. It takes a PLM and a text classification dataset as input, then outputs detailed metrics showing how 'calibrated' the model's confidence scores are. Scientists and machine learning engineers working with language models for tasks like sentiment analysis or spam detection would use this.
No commits in the last 6 months.
Use this if you need to rigorously evaluate the confidence scores of your large language models, especially in research or when deploying models where trust in predictions is critical.
Not ideal if you are looking for a tool to directly improve or 'calibrate' your deployed language models without deep analysis of the underlying calibration dynamics.
Stars
11
Forks
1
Language
Python
License
MIT
Category
Last pushed
May 09, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/nlp/lifan-yuan/PLMCalibration"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
n-waves/multifit
The code to reproduce results from paper "MultiFiT: Efficient Multi-lingual Language Model...
princeton-nlp/SimCSE
[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821
yxuansu/SimCTG
[NeurIPS'22 Spotlight] A Contrastive Framework for Neural Text Generation
alibaba-edu/simple-effective-text-matching
Source code of the ACL2019 paper "Simple and Effective Text Matching with Richer Alignment Features".
Shark-NLP/OpenICL
OpenICL is an open-source framework to facilitate research, development, and prototyping of...