adobe-research/NoLiMa
Official repository for "NoLiMa: Long-Context Evaluation Beyond Literal Matching"
NoLiMa helps evaluate how well large language models (LLMs) can find specific, relevant information buried deep within very long documents, even when the question doesn't use the exact same words as the answer. It takes a question and a lengthy text document as input and measures the model's ability to extract the correct answer, providing a score for different document lengths. This tool is designed for AI researchers and practitioners who are developing or comparing long-context LLMs.
186 stars. No commits in the last 6 months.
Use this if you need to rigorously test whether a large language model can truly understand and retrieve information from extremely long texts, especially when it requires inferring meaning beyond simple keyword matching.
Not ideal if you are looking for an off-the-shelf solution for short-context question answering, or if your primary concern is literal keyword matching performance in information retrieval.
Stars
186
Forks
16
Language
Python
License
—
Category
Last pushed
Jul 17, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/adobe-research/NoLiMa"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Higher-rated alternatives
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems