logic-star-ai/swt-bench
[NeurIPS 2024] Evaluation harness for SWT-Bench, a benchmark for evaluating LLM repository-level test-generation
This tool helps developers and researchers evaluate how well large language models can automatically generate tests for software issues found on GitHub. You provide a codebase and an issue description, and the LLM attempts to produce a 'reproducing test' that fails before a bug fix and passes afterward. The output is a performance report detailing the LLM's success rate and other metrics, useful for comparing different LLMs or development approaches.
Use this if you are a software developer, researcher, or AI engineer who needs to benchmark and compare the effectiveness of large language models at generating automated tests for real-world software bugs.
Not ideal if you are looking for a general-purpose bug-reporting tool or an environment to write manual software tests.
Stars
72
Forks
17
Language
Python
License
MIT
Category
Last pushed
Jan 15, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/logic-star-ai/swt-bench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Featured in
Related tools
sierra-research/tau2-bench
τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment
xlang-ai/OSWorld
[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
bigcode-project/bigcodebench
[ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI
scicode-bench/SciCode
A benchmark that challenges language models to code solutions for scientific problems
THUDM/AgentBench
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)