LSquaredM/mutual_info_scaling_law
(NeurIPS 2025) Official Code for L²M: Mutual Information Scaling Law for Long-Context Language Modeling
This project helps AI researchers understand how effectively large language models (LLMs) process and retain information over very long texts. It takes text data and LLM configurations as input, then measures how much information the model captures about distant parts of the text. The primary users are researchers and practitioners focused on improving the performance of LLMs in handling extended contexts.
Use this if you are a researcher studying the theoretical underpinnings of long-context language models and want to reproduce or extend experiments on mutual information scaling laws.
Not ideal if you are looking for a tool to directly fine-tune or deploy an LLM for a specific application.
Stars
13
Forks
—
Language
Python
License
—
Category
Last pushed
Nov 09, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/LSquaredM/mutual_info_scaling_law"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
jncraton/languagemodels
Explore large language models in 512MB of RAM
microsoft/unilm
Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
haizelabs/verdict
Inference-time scaling for LLMs-as-a-judge.
albertan017/LLM4Decompile
Reverse Engineering: Decompiling Binary Code with Large Language Models
bytedance/Sa2VA
Official Repo For Pixel-LLM Codebase