jie-jw-wu/human-eval-comm
HumanEvalComm: Evaluating Communication Skill of Code LLM and LLM Agent
This project helps evaluate how well Large Language Models (LLMs) can clarify ambiguous, inconsistent, or incomplete coding problems. You feed it a coding problem description (modified from the HumanEval benchmark) and it assesses the LLM's ability to ask relevant clarifying questions and generate correct code. This is useful for AI researchers and developers who are building or evaluating LLMs for code generation tasks.
Use this if you need to benchmark the communication and code generation skills of various LLMs when faced with imperfect problem specifications.
Not ideal if you're looking for a tool to automatically fix or debug your own code, or to generate production-ready code from vague requirements.
Stars
11
Forks
4
Language
Python
License
—
Category
Last pushed
Feb 08, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/jie-jw-wu/human-eval-comm"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
oripress/AlgoTune
AlgoTune is a NeurIPS 2025 benchmark made up of 154 math, physics, and computer science...
xjywhu/Awesome-Multimodal-LLM-for-Code
Multimodal Large Language Models for Code Generation under Multimodal Scenarios
juyongjiang/CodeUp
CodeUp: A Multilingual Code Generation Llama-X Model with Parameter-Efficient Instruction-Tuning
JHansiduYapa/Fine-Tuning-a-Small-Language-Model-for-Cypher-Query-Generation
This project fine-tunes Unsloth's Gemma-3 4B IT (4-bit) model to translate natural language into...
Gen-Verse/ReasonFlux
[NeurIPS 2025 Spotlight] LLM post-training suite — featuring ReasonFlux, ReasonFlux-PRM, and...