3B-Group/ConvRe
🤖ConvRe🤯: An Investigation of LLMs’ Inefficacy in Understanding Converse Relations (EMNLP 2023)
This project helps evaluate how well large language models (LLMs) understand relationships between concepts, especially when those relationships are expressed in a converse (opposite) way. It takes in structured facts (like "x has part y") and assesses an LLM's ability to interpret them correctly in both normal and converse forms, showing if the model truly grasps the meaning or just uses shortcuts. This tool is for AI researchers, natural language processing engineers, or anyone developing and evaluating LLMs.
No commits in the last 6 months.
Use this if you need to rigorously test the semantic understanding capabilities of a large language model, particularly its ability to discern between normal and converse relational meanings.
Not ideal if you are looking for a tool to train LLMs or apply them directly to a specific real-world text analysis task, as this is solely an evaluation benchmark.
Stars
24
Forks
—
Language
Python
License
—
Category
Last pushed
Oct 10, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/llm-tools/3B-Group/ConvRe"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
filipnaudot/llmSHAP
llmSHAP: a multi-threaded explainability framework using Shapley values for LLM-based outputs.
microsoft/automated-brain-explanations
Generating and validating natural-language explanations for the brain.
CAS-SIAT-XinHai/CPsyCoun
[ACL 2024] CPsyCoun: A Report-based Multi-turn Dialogue Reconstruction and Evaluation Framework...
wesg52/universal-neurons
Universal Neurons in GPT2 Language Models
ICTMCG/LLM-for-misinformation-research
Paper list of misinformation research using (multi-modal) large language models, i.e., (M)LLMs.