frankaging/ReCOGS
ReCOGS: How Incidental Details of a Logical Form Overshadow an Evaluation of Semantic Interpretation
This project helps researchers and practitioners in natural language processing (NLP) to more accurately evaluate how well language models understand sentence meanings. It takes the COGS benchmark dataset, which contains sentences and their logical forms (semantic representations), and reformats it in several ways to reduce biases. The output is a modified dataset that more truthfully measures a model's ability to generalize semantic understanding, not just its ability to mimic specific logical form syntax.
No commits in the last 6 months.
Use this if you are a natural language processing researcher or practitioner evaluating the semantic interpretation capabilities of language models, especially if you use or are considering using the COGS benchmark.
Not ideal if you are looking for a tool to build or deploy general-purpose language understanding applications, as this is a research tool focused on evaluation methodology.
Stars
10
Forks
4
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Jul 10, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/frankaging/ReCOGS"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
lucidrains/x-transformers
A concise but complete full-attention transformer with a set of promising experimental features...
kanishkamisra/minicons
Utility for behavioral and representational analyses of Language Models
lucidrains/simple-hierarchical-transformer
Experiments around a simple idea for inducing multiple hierarchical predictive model within a GPT
lucidrains/dreamer4
Implementation of Danijar's latest iteration for his Dreamer line of work
Nicolepcx/Transformers-in-Action
This is the corresponding code for the book Transformers in Action