SeekingDream/DyCodeEval
Official repository of the ICML2025 paper “Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination”
This project offers a new way to test how well code-generating AI models (Code LLMs) can actually solve programming problems, even if they've seen similar code during training. It takes existing coding challenges like HumanEval or MBPP and dynamically rewrites them into new, diverse versions. The output is a more reliable assessment of a Code LLM's true reasoning ability, free from the influence of memorized data, for AI researchers and practitioners evaluating these models.
255 stars.
Use this if you need to rigorously evaluate the reasoning capabilities of Code LLMs and want to avoid misleading results caused by models having already seen the test data.
Not ideal if you are looking for a tool to develop or fine-tune Code LLMs, as its primary purpose is evaluation rather than model creation.
Stars
255
Forks
22
Language
Python
License
—
Category
Last pushed
Dec 23, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/SeekingDream/DyCodeEval"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ExtensityAI/symbolicai
A neurosymbolic perspective on LLMs
TIGER-AI-Lab/MMLU-Pro
The code and data for "MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding...
deep-symbolic-mathematics/LLM-SR
[ICLR 2025 Oral] This is the official repo for the paper "LLM-SR" on Scientific Equation...
microsoft/interwhen
A framework for verifiable reasoning with language models.
zhudotexe/fanoutqa
Companion code for FanOutQA: Multi-Hop, Multi-Document Question Answering for Large Language...