logikon-ai/cot-eval
A framework for evaluating the effectiveness of chain-of-thought reasoning in language models.
This tool helps AI researchers and developers systematically assess how effectively large language models (LLMs) use Chain-of-Thought (CoT) reasoning. You provide an LLM, a specific task like logical reasoning, and a CoT prompting strategy. The tool then generates reasoning traces and evaluates the model's performance on both original and perturbed versions of the task, providing metrics on CoT effectiveness and potential data contamination.
No commits in the last 6 months.
Use this if you are a researcher or AI developer working on large language models and need to rigorously evaluate their reasoning capabilities and identify potential training data contamination.
Not ideal if you are an end-user looking for a ready-to-use LLM application, rather than a framework for model evaluation.
Stars
19
Forks
3
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Feb 06, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/logikon-ai/cot-eval"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
SimonAytes/SoT
Official code repository for Sketch-of-Thought (SoT)
xuyige/SoftCoT
ACL'2025: SoftCoT: Soft Chain-of-Thought for Efficient Reasoning with LLMs. and preprint:...
Fr0zenCrane/UniCoT
[ICLR 2026] Uni-CoT: Towards Unified Chain-of-Thought Reasoning Across Text and Vision
nicolay-r/THOR-ECAC
The official fork of THoR Chain-of-Thought framework, enhanced and adapted for Emotion Cause...
andrewginns/CoT-at-Home
Who needs o1 anyways. Add CoT to any OpenAI compatible endpoint.