microsoft/repoclassbench
[ICML DMLR 2024] Repo that contains code for the paper titled: "Class-Level Code Generation from Natural Language Using Iterative, Tool-Enhanced Reasoning over Repository".
This project helps software developers evaluate how well large language models (LLMs) can generate complex, class-level code within existing software projects. It takes a natural language description of a desired class and an existing codebase, then assesses the LLM's ability to generate functional code that integrates correctly with the repository's dependencies. The target users are software engineers and researchers working on advanced code generation tools and benchmarks.
No commits in the last 6 months.
Use this if you are developing or evaluating AI models for generating software code and need a robust way to measure their performance on realistic, class-level coding tasks within a larger repository.
Not ideal if you are looking for a tool to generate code directly for your software projects, as this is a benchmark for evaluating code generation models, not an end-user code generation application.
Stars
17
Forks
4
Language
Python
License
MIT
Category
Last pushed
Apr 29, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/microsoft/repoclassbench"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
MMMU-Benchmark/MMMU
This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal...
pat-jj/DeepRetrieval
[COLM’25] DeepRetrieval — 🔥 Training Search Agent by RLVR with Retrieval Outcome
lupantech/MathVista
MathVista: data, code, and evaluation for Mathematical Reasoning in Visual Contexts
x66ccff/liveideabench
[𝐍𝐚𝐭𝐮𝐫𝐞 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬] 🤖💡 LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea...
ise-uiuc/magicoder
[ICML'24] Magicoder: Empowering Code Generation with OSS-Instruct