CheongWoong/impact_of_cooccurrence
A repository for analyzing the impact of co-occurrence statistics on factual knowledge of large language models (EMNLP 2023 Findings).
This project helps researchers understand how word co-occurrence in large text datasets affects the factual knowledge stored in large language models. It takes pre-training data and knowledge probing datasets as input to calculate statistics like term document index and co-occurrence matrices. The output includes analysis results and baselines for probing how well language models capture factual information. It is designed for AI researchers and NLP scientists studying language model behavior and knowledge representation.
No commits in the last 6 months.
Use this if you are an AI researcher investigating the mechanisms by which large language models acquire and represent factual knowledge from their training data, particularly focusing on the role of word co-occurrence statistics.
Not ideal if you are looking for a tool to build or fine-tune a practical large language model for an application, as this is a research framework for analysis.
Stars
10
Forks
—
Language
Jupyter Notebook
License
MIT
Category
Last pushed
Dec 13, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/CheongWoong/impact_of_cooccurrence"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
galilai-group/stable-pretraining
Reliable, minimal and scalable library for pretraining foundation and world models
CognitiveAISystems/MAPF-GPT
[AAAI-2025] This repository contains MAPF-GPT, a deep learning-based model for solving MAPF...
UKPLab/gpl
Powerful unsupervised domain adaptation method for dense retrieval. Requires only unlabeled...
larslorch/avici
Amortized Inference for Causal Structure Learning, NeurIPS 2022
svdrecbd/mhc-mlx
MLX + Metal implementation of mHC: Manifold-Constrained Hyper-Connections by DeepSeek-AI.