lt-asset/REPOCOD
For our ACL25 Paper: Can Language Models Replace Programmers? RepoCod Says ‘Not Yet’ - by Shanchao Liang and Yiran Hu and Nan Jiang and Lin Tan
REPOCOD is a specialized benchmark designed to assess how well large language models can generate code for real-world software projects. It takes code generation models as input and outputs their performance scores, specifically evaluating their ability to handle complex, multi-file programming tasks. This tool is for AI researchers and developers who are building or evaluating advanced code-generating AI.
No commits in the last 6 months.
Use this if you are developing or evaluating large language models intended for complex, real-world software development tasks that require understanding across multiple code files.
Not ideal if you are looking for a simple benchmark for basic, single-file code generation problems, or if you are not working on advanced LLM development.
Stars
26
Forks
3
Language
Python
License
BSD-3-Clause
Category
Last pushed
Aug 27, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/lt-asset/REPOCOD"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
MMMU-Benchmark/MMMU
This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal...
pat-jj/DeepRetrieval
[COLM’25] DeepRetrieval — 🔥 Training Search Agent by RLVR with Retrieval Outcome
lupantech/MathVista
MathVista: data, code, and evaluation for Mathematical Reasoning in Visual Contexts
x66ccff/liveideabench
[𝐍𝐚𝐭𝐮𝐫𝐞 𝐂𝐨𝐦𝐦𝐮𝐧𝐢𝐜𝐚𝐭𝐢𝐨𝐧𝐬] 🤖💡 LiveIdeaBench: Evaluating LLMs' Scientific Creativity and Idea...
ise-uiuc/magicoder
[ICML'24] Magicoder: Empowering Code Generation with OSS-Instruct