lechmazur/generalization
Thematic Generalization Benchmark: measures how effectively various LLMs can infer a narrow or specific "theme" (category/rule) from a small set of examples and anti-examples, then detect which item truly fits that theme among a collection of misleading candidates.
This project helps evaluate how well large language models (LLMs) can grasp a very specific, underlying concept from a few examples, and then apply that understanding to identify the single best match among several similar, but incorrect, options. You provide the LLM with a small set of positive examples, a small set of closely related 'anti-examples', and a list of candidates. It then tells you which candidate best fits the precise concept. This is for professionals who use or develop LLMs and need to rigorously test their ability to infer subtle patterns.
No commits in the last 6 months.
Use this if you need to objectively measure an LLM's capacity for precise conceptual inference and its ability to distinguish subtle differences between categories.
Not ideal if you're looking for a benchmark to test general knowledge, common-sense reasoning, or broad categorization abilities.
Stars
63
Forks
2
Language
—
License
—
Category
Last pushed
Sep 22, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/lechmazur/generalization"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems