SORRY-Bench/sorry-bench
Benchmark evaluation code for "SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal" (ICLR 2025)
This tool helps AI safety researchers and developers systematically evaluate how well large language models (LLMs) refuse unsafe or inappropriate requests. You input an LLM (either a local model or an API-based one like GPT or Claude) and it outputs detailed compliance rates across 44 safety categories. This allows practitioners to quickly understand an LLM's safety behaviors and identify areas for improvement.
No commits in the last 6 months.
Use this if you need a comprehensive, granular, and efficient way to benchmark the safety refusal capabilities of various large language models, including how they handle diverse linguistic styles and mutations.
Not ideal if you are looking to fine-tune an LLM for safety or to develop new safety refusal techniques, as this tool focuses on evaluation rather than training.
Stars
77
Forks
5
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Mar 01, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/SORRY-Bench/sorry-bench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
stanfordnlp/axbench
Stanford NLP Python library for benchmarking the utility of LLM interpretability methods
aidatatools/ollama-benchmark
LLM Benchmark for Throughput via Ollama (Local LLMs)
LarHope/ollama-benchmark
Ollama based Benchmark with detail I/O token per second. Python with Deepseek R1 example.
qcri/LLMeBench
Benchmarking Large Language Models
THUDM/LongBench
LongBench v2 and LongBench (ACL 25'&24')