ahans30/goldfish-loss
[NeurIPS 2024] Goldfish Loss: Mitigating Memorization in Generative LLMs
This project offers a simple technique to train large language models (LLMs) in a way that significantly reduces their tendency to 'memorize' and inadvertently reproduce specific parts of their training data. It takes an existing LLM training setup as input and outputs a fine-tuned or pre-trained LLM that is less prone to revealing sensitive or private information from its training set. This tool is for AI researchers and engineers who are pre-training or fine-tuning generative LLMs, especially those concerned with data privacy and model safety.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher involved in developing and deploying large language models and need to reduce the risk of your models memorizing and leaking training data.
Not ideal if you are not directly involved in the low-level training or fine-tuning of large language models or if you are not using AMD compute nodes in a SLURM-managed distributed environment.
Stars
97
Forks
8
Language
Python
License
Apache-2.0
Category
Last pushed
Nov 17, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/ahans30/goldfish-loss"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
kyegomez/LIMoE
Implementation of the "the first large-scale multimodal mixture of experts models." from the...
dohlee/chromoformer
The official code implementation for Chromoformer in PyTorch. (Lee et al., Nature Communications. 2022)
yinboc/trans-inr
Transformers as Meta-Learners for Implicit Neural Representations, in ECCV 2022
bloomberg/MixCE-acl2023
Implementation of MixCE method described in ACL 2023 paper by Zhang et al.
ibnaleem/mixtral.py
A Python module for running the Mixtral-8x7B language model with customisable precision and...