RulinShao/RAG-evaluation-harnesses

An evaluation suite for Retrieval-Augmented Generation (RAG).

/ 100

Emerging

This project helps evaluate how well your Retrieval-Augmented Generation (RAG) system performs on various question-answering tasks. You provide your RAG model's retrieved documents and the questions, and it outputs performance scores. This tool is for researchers, developers, or MLOps engineers who are building and fine-tuning RAG systems and need to rigorously benchmark their effectiveness.

No commits in the last 6 months.

Use this if you are developing a RAG system and need to systematically test its accuracy and performance across established benchmarks.

Not ideal if you are looking for a tool to deploy or manage your RAG system in a production environment, as this focuses solely on evaluation.

RAG-evaluation LLM-benchmarking NLP-research AI-model-testing information-retrieval

Stale 6m No Package No Dependents

Maintenance 2 / 25

Adoption 6 / 25

Maturity 16 / 25

Community 11 / 25

How are scores calculated?

Stars

Forks

Language

Python

License

MIT

Featured in

You're Shipping AI You Can't Measure

Compare

RAG-evaluation-harnesses and open-rag-eval RAG-evaluation-harnesses and XRAG RAG-evaluation-harnesses and rag-evaluator RAG-evaluation-harnesses and rageval RAG-evaluation-harnesses and RAG-Evaluator

Higher-rated alternatives

vectara/open-rag-eval

RAG evaluation without the need for "golden answers"

DocAILab/XRAG

XRAG: eXamining the Core - Benchmarking Foundational Component Modules in Advanced...

HZYAI/RagScore

⚡️ The "1-Minute RAG Audit" — Generate QA datasets & evaluate RAG systems in Colab, Jupyter, or...

AIAnytime/rag-evaluator

A library for evaluating Retrieval-Augmented Generation (RAG) systems (The traditional ways).

microsoft/benchmark-qed

Automated benchmarking of Retrieval-Augmented Generation (RAG) systems

Explore RAG Tools

All categories Trending RAG directory Insights