modelscope/evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
This tool helps AI model developers and researchers objectively assess how well large language models (LLMs), vision-language models (VLMs), and other generative AI models perform. You provide various models and datasets, and it generates detailed comparison reports and performance metrics, including stress test results and interactive visualizations. It helps you understand a model's strengths and weaknesses across different tasks and benchmarks.
2,501 stars. Used by 1 other package. Actively maintained with 34 commits in the last 30 days. Available on PyPI.
Use this if you need to thoroughly benchmark and compare multiple large AI models (LLMs, VLMs, AIGC) against standard datasets or custom criteria to determine their effectiveness for specific applications.
Not ideal if you are a casual user looking for a simple API to integrate a pre-trained model without needing deep performance analysis or custom evaluation.
Stars
2,501
Forks
285
Language
Python
License
Apache-2.0
Category
Last pushed
Mar 11, 2026
Commits (30d)
34
Dependencies
38
Reverse dependents
1
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/modelscope/evalscope"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Compare
Related tools
izam-mohammed/ragrank
🎯 Your free LLM evaluation toolkit helps you assess the accuracy of facts, how well it...
Kareem-Rashed/rubric-eval
Independent framework to test, benchmark, and evaluate LLMs & AI agents locally.
justplus/llm-eval
大语言模型评估平台，支持多种评估基准、自定义数据集和性能测试。支持基于自定义数据集的RAG评估。
relari-ai/continuous-eval
Data-Driven Evaluation for LLM-Powered Applications
cleanlab/tlm
Score the trustworthiness of outputs from any LLM in real-time