GPT2 Pretraining Fine-tuning Transformer Models

Tools for pretraining, fine-tuning, and implementing GPT-2 models from scratch, including language-specific variants and inference optimization. Does NOT include downstream applications like question-answering or summarization, nor other model architectures beyond GPT-2 variants.

There are 149 gpt2 pretraining fine-tuning models tracked. 7 score above 50 (established tier). The highest-rated is tabularis-ai/be_great at 65/100 with 350 stars.

Get all 149 projects as JSON

curl "https://pt-edge.onrender.com/api/v1/datasets/quality?domain=transformers&subcategory=gpt2-pretraining-fine-tuning&limit=20"

Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.

#	Model	Score	Tier	Stars	Language
1	tabularis-ai/be_great A novel approach for synthesizing tabular data using pretrained large language models	65	Established	350	Python
2	EleutherAI/gpt-neox An implementation of model parallel autoregressive transformers on GPUs,...	58	Established	7,399	Python
3	shibing624/textgen TextGen: Implementation of Text Generation models, include LLaMA, BLOOM,...	55	Established	979	Python
4	ai-forever/ru-gpts Russian GPT3 models.	51	Established	2,093	Python
5	AdityaNG/kan-gpt The PyTorch implementation of Generative Pre-trained Transformers (GPTs)...	51	Established	725	Python
6	zemlyansky/gpt-tfjs GPT in TensorFlow.js	50	Established	33	JavaScript
7	keith2018/TinyGPT Tiny C++ LLM inference implementation from scratch	50	Established	106	C++
8	kyegomez/GPT4o Community Open Source Implementation of GPT4o in PyTorch	48	Emerging	26	Shell
9	kakaobrain/kogpt KakaoBrain KoGPT (Korean Generative Pre-trained Transformer)	48	Emerging	1,014	Python
10	EleutherAI/gpt-neo An implementation of model parallel GPT-2 and GPT-3-style models using the...	47	Emerging	8,286	Python
11	kyegomez/Lets-Verify-Step-by-Step "Improving Mathematical Reasoning with Process Supervision" by OPENAI	47	Emerging	114	Python
12	0hq/WebGPT Run GPT model on the browser with WebGPU. An implementation of GPT inference...	44	Emerging	3,784	JavaScript
13	turtlesoupy/this-word-does-not-exist This Word Does Not Exist	43	Emerging	1,021	Python
14	cdpierse/script_buddy_v2 Script Buddy v2 is a film script text generation tool built using film...	43	Emerging	47	Jupyter Notebook
15	eric-ai-lab/MiniGPT-5 Official implementation of paper "MiniGPT-5: Interleaved Vision-and-Language...	43	Emerging	863	Python
16	mytechnotalent/RE-GPT Inspired by Andrej Karpathy’s "Let’s Build GPT", this project guides you...	43	Emerging	27	Jupyter Notebook
17	ai-forever/mgpt Multilingual Generative Pretrained Model	42	Emerging	207	Jupyter Notebook
18	hyperonym/basaran Basaran is an open-source alternative to the OpenAI text completion API. It...	42	Emerging	1,290	Python
19	datadreamer-dev/DataDreamer DataDreamer: Prompt. Generate Synthetic Data. Train & Align Models. 🤖💤	41	Emerging	1,100	Python
20	erogol/BlaGPT Experimental playground for benchmarking language model (LM) architectures,...	41	Emerging	102	Python
21	saqib1707/gpt2-from-scratch PyTorch Implementation of GPT-2	40	Emerging	31	Python
22	soumyadip1995/BabyGPT Something in the middle of Karpathy's mingpt model and video lectures, ...	39	Emerging	24	Jupyter Notebook
23	TatevKaren/BabyGPT-Build_GPT_From_Scratch BabyGPT: Build Your Own GPT Large Language Model from Scratch Pre-Training...	39	Emerging	116	Python
24	wpeebles/G.pt Official PyTorch Implementation of "Learning to Learn with Generative Models...	39	Emerging	345	Python
25	EagleW/Stage-wise-Fine-tuning Code for Stage-wise Fine-tuning for Graph-to-Text Generation	38	Emerging	26	Lex
26	iVishalr/GPT A minimal and efficient Pytorch implementation of OpenAI's GPT (Generative...	38	Emerging	18	Jupyter Notebook
27	arrmansa/Basic-UI-for-GPT-J-6B-with-low-vram A repository to run gpt-j-6b on low vram machines (4.2 gb minimum vram for...	38	Emerging	113	Jupyter Notebook
28	kyegomez/CNNGPT This CNN-based language model leverages causal and dilated convolutions,...	38	Emerging	4	Python
29	potamides/uniformers Token-free Language Modeling with ByGPT5 & Friends!	37	Emerging	12	Python
30	arrmansa/Gpt-Neo-Limited-Vram-Cuda A notebook that runs GPT-Neo with low vram (6 gb) and cuda acceleration by...	37	Emerging	14	Jupyter Notebook
31	losttech/Torch.MinGPT A C# implementation of GPT	37	Emerging	20	C#
32	sytelus/nanuGPT Simple, reliable and well tested training code for quick experiments with...	37	Emerging	13	Python
33	readme-generator/alreadyme-ai-research Generate README.md with GPT-3 few-shot learning	37	Emerging	27	Python
34	dreamingjudith/KoGPT2-personachat Fine-tuned KoGPT2 chatbot demo with translated PersonaChat (ongoing)	36	Emerging	13	Jupyter Notebook
35	arrmansa/Basic-UI-for-GPT-Neo-with-low-vram A basic ui for running gpt neo 2.7B on low vram (3 gb Vram minimum)	35	Emerging	36	Jupyter Notebook
36	FareedKhan-dev/gpt4o-from-scratch Implementation of a GPT-4o like Multimodal from Scratch using Python	35	Emerging	78	Jupyter Notebook
37	EvilFreelancer/rugpt3-custom Pre-training custom ruGPT3 model on books written by F.M. Dostoevski	34	Emerging	7	Python
38	tanulsingh/Humour.ai-Language-model-that-can-crack-Jokes Language Model that makes you Laugh .	33	Emerging	41	Python
39	jseeio/gpt2-tfjs GPT2 with Tensorflow.js	33	Emerging	4	JavaScript
40	StarxSky/ANE-GPT-New New ANE GPT	32	Emerging	5	Python
41	EdvardOlsen/Horoscope_generator This is a horoscope generating code	32	Emerging	4	Python
42	ant-louis/belgpt2 🇧🇪 BelGPT-2: the 1st GPT model pretrained in French.	31	Emerging	34	Python
43	Agora-Lab-AI/OmniByteGPT An implementation of an all-new foundation model architecture that trains on...	31	Emerging	9	Python
44	JarvisPei/FuseGPT The implementation for the paper, FuseGPT: Learnable Layers Fusion of...	31	Emerging	4	Python
45	trekhleb/homemade-gpt-js A minimal TensorFlow.js re-implementation of Karpathy's minGPT (Generative...	30	Emerging	88	TypeScript
46	fcakyon/gpt2-shakespeare A tutorial on GPT2 language model training with texts from Shakespeare	30	Emerging	1	Jupyter Notebook
47	mrseanryan/gpt-local Local GPT (llama 2 or dolly or gpt etc.) via Python - using ctransforers project	30	Emerging	2	Python
48	Atenrev/forocoches-language-generation This is a PyTorch implementation of a decoder only transformer inspired on...	29	Experimental	7	Python
49	procesaur/Scratch2LM Training transformer models (e.g. RoBERTa, GPT2 and GPT-J) from scratch.	29	Experimental	7	Python
50	uSaiPrashanth/gpt-j-finetune Parallelizes finetuning of gpt-j on P3 dataset across multiple gpu nodes	29	Experimental	1	Python
51	YashrajBaila7/GPT2LM A implimentation of GPT2 varient.	29	Experimental	1	Python
52	Any-Winter-4079/Nano-GPT-Speedrun-Track This repo represents my Nano-GPT speedrun playground, which started coding...	29	Experimental	6	Python
53	kabachuha/nanoGPKANT Testing KAN-based text generation GPT models	27	Experimental	18	Jupyter Notebook
54	Navy10021/KRLawGPT KRLawGPT : Generative Pre-trained Transformer for producing Korean Legal Text	26	Experimental	9	Python
55	mytechnotalent/MicroGPT MicroGPT is a clean, educational implementation of the GPT (Generative...	25	Experimental	2	Python
56	Andras7/gpt2-pytorch Extremely simple and understandable GPT2 implementation with minor tweaks	25	Experimental	21	Python
57	s-omranpour/Shirin-Sokhan A Persian Poet Transformer! (finetuned GPT2 on Ganjoor data)	25	Experimental	5	Jupyter Notebook
58	codiceSpaghetti/numpyGPT A from-scratch GPT built with NumPy and Python’s standard library. No...	24	Experimental	20	Python
59	SIC98/GPT2-python-code-generator GPT2 finetuning with transformers 🤗	24	Experimental	28	Jupyter Notebook
60	pablo-reyes8/implementing-gpt Clean-room GPT-2/GPT-3 implementation: tokenizers, architecture blocks,...	24	Experimental	1	Python
61	Vadimbuildercxx/NumpyGPT A lightweight educational implementation of GPT (Generative Pre-trained...	23	Experimental	4	Python
62	chizkidd/microGPT Minimal char-level GPT inspired by @karpathy's microGPT: multi-dataset...	23	Experimental	2	Jupyter Notebook
63	fattorib/Little-GPT GPT* - Training faster small transformers using ALiBi, Parallel Residual...	22	Experimental	21	Python
64	RahulSChand/gpt2_squad GPT2 training on squad dataset	22	Experimental	2	Python
65	kyegomez/TinyGPTV Simple Implementation of TinyGPTV in super simple Zeta lego blocks	22	Experimental	16	Python
66	pronzzz/atomgpt AtomGPT is a chaotic, evolutionary implementation of a Generative...	21	Experimental	—	Python
67	aarxshi/DsaGPT A minimal GPT-style transformer built from scratch for DSA-style Q&A	21	Experimental	—	Jupyter Notebook
68	jaketae/lm-identifier A toolkit for identifying pretrained language models from potentially...	21	Experimental	9	Python
69	Eden-Eldith/WiggleGPT WiggleGPT is an language model that integrates bio-inspired neural...	20	Experimental	1	Python
70	marlo-z/reversal_curse_analysis Code for 'Towards a Theoretical Understanding of the 'Reversal Curse' via...	20	Experimental	5	Python
71	lorenzomaiuri-dev/quantum-gpt A hybrid Quantum-Classical Transformer implementation based on nanoGPT,...	20	Experimental	1	Python
72	alperiox/bookbot A toy project for my generative AI studies on text data. Train generative...	19	Experimental	1	Python
73	hackerlibs/inline-completion-model PromptlyCode inline completion model by PyTorch	19	Experimental	4	Python
74	bellthomas/gpt.local A work-in-progress, from-scratch implementation of a generative pre-trained...	19	Experimental	4	Python
75	iangitonga/gten A minimal library to run transformer neural networks on CPU.	19	Experimental	3	C
76	NJX-njx/microgpt 🔬 The most atomic GPT-2 implementation in 265 lines of pure Python & CUDA. A...	19	Experimental	21	HTML
77	neuronalin/gpt-from-scratch-pytorch A decoder-only GPT-style Transformer built from scratch with PyTorch —...	19	Experimental	—	Python
78	shreydan/shakespeareGPT understanding language modeling by training a small GPT on Shakespeare plays.	19	Experimental	13	Jupyter Notebook
79	Sairamg18814/GUBBALA-V3-TRUE Revolutionary Self-Evolving Language Model - 100% self-contained AI trained...	19	Experimental	2	Python
80	zTgx/DeepText A GPT Model To Generate Text	19	Experimental	2	Python
81	tahmidmir/Graph-RAG Fine-tuning GPT-2 on domain-specific articles related to skin cancer, using...	18	Experimental	2	Jupyter Notebook
82	Divyansh900/PyCodeGen A python code generation model with 75M parameter built from the ground up...	18	Experimental	1	Python
83	dedsecurity/dpt Repo for offsite scale work	18	Experimental	2	Python
84	Agora-Lab-AI/NeoCore NeoCore™ - Next Generation CPU-Native Transformer.	18	Experimental	2	Python
85	neemiasbsilva/developing-nanoGPT2-fineweb Developing a cusstom nano GPT-2 from scratch using PyTorch on the Fineweb dataset.	18	Experimental	2	Python
86	Amir-Hofo/GPT2 Implementation of the GPT-2 architecture using PyTorch, trained on the...	17	Experimental	—	Python
87	sartq333/story-GPT a simple GPT model pre-trained from scratch on tiny stories dataset	17	Experimental	1	Jupyter Notebook
88	J3lly-Been/gpt2-story-generation This project fine-tunes GPT-2, a popular pre-trained transformer model, to...	17	Experimental	1	Jupyter Notebook
89	Uni-Creator/NanoGPT NanoGPT is a lightweight GPT-style language model designed for text...	17	Experimental	1	Python
90	juletx/gpt2-eus Pretraining GPT2 model on Basque language	17	Experimental	1	Python
91	Harsh-2909/gpt-from-scratch The "GPT from Scratch" project is an endeavor to implement the Generative...	17	Experimental	1	Jupyter Notebook
92	baumandm/lorem-insight Tool to generate lorem ipsum-style Insights for Insights Explorer	17	Experimental	1	Python
93	jiseokson/PageBrain Light-weight LLM Serving with PagedAttention	17	Experimental	15	Python
94	lin826/nanoGPT-demo Training and finetuning local GPTs.	17	Experimental	—	Python
95	Adam-Bowen/nanoGPT 🧠 nanoGPT (Andrej Karpathy's Zero to Hero)	17	Experimental	1	Jupyter Notebook
96	jndiogo/gptbench A python package to experiment with GPT-like transformer models	17	Experimental	1	Jupyter Notebook
97	inkybubble/mi_01_attention_patterns_scratch MI-01 - Attention Patterns from Scratch: Finding Previous-Token and...	17	Experimental	—	Python
98	SynthWomb/Synthia SynthiaGPT leverages Google's Gemini & the Hugging Face Transformers library...	17	Experimental	1	Python
99	Med-Karim-Ben-Boubaker/gpt-2-from-scratch A repository that shows the code behind different LLMs architectures and...	16	Experimental	3	Python
100	buhsnn/eli5-gpt2-language-model Decoder-only Transformer (GPT-2 style) trained from scratch on the ELI5...	14	Experimental	1	Jupyter Notebook
101	Sparsh-2007/GPT-From-Scratch Implementation of a GPT-style LLM from scratch, following "Build a Large ...	14	Experimental	1	Jupyter Notebook
102	Jake1402/Torch-GPTs A way for users to train, and interact with their own mini language models...	14	Experimental	1	Python
103	RWKV-Wiki/rwkv-wiki.github.io RWKV Wiki website (archived, please visit official wiki)	13	Experimental	11	Shell
104	Ojas025/almostGPT A GPT implementation for training and generating text on custom datasets	13	Experimental	—	Jupyter Notebook
105	diixo/build-gpt A PyTorch library with educational re-implementation of GPT-models: GPT2, LLaMA	13	Experimental	—	Python
106	sumony2j/SeedGPT-22M SeedGPT is a lightweight, 22M-parameter Transformer LLM for efficient text...	13	Experimental	—	Python
107	ENOT-AutoDL/gpt-j-6B-tensorrt-int8 GPT-J 6B inference on TensorRT with INT-8 precision	13	Experimental	11	Python
108	Akhan521/GPT-From-Scratch 🧸 A fully custom GPT-style language model built from scratch using PyTorch...	13	Experimental	—	Python
109	ademyanchuk/gpt2-diy From-scratch reproduction of GPT-2 following Andrej Karpathy's "Zero to Hero" series.	13	Experimental	—	Jupyter Notebook
110	n9e6y/PPG Persian Poetry Generator: A fine-tuned GPT-2 model for generating Persian...	13	Experimental	—	Jupyter Notebook
111	chandan11248/GPT-2 Learning and implementing GPT-2 from scratch, including architecture...	13	Experimental	—	Jupyter Notebook
112	btboilerplate/GPT-2 Fine-tunes the GPT-2 language model on Shakespearean text to generate...	13	Experimental	—	Jupyter Notebook
113	TomaszKaleczyc/scifi_book_generator The purpose of this project is to build a decoder only transformer...	13	Experimental	—	Python
114	BenBenyamin/GPT2 My implementation GPT2 from scratch using the original GPT2 and GPT3 papers	13	Experimental	—	Python
115	oskarfernlund/noskGPT Simple transformer-based language model which generates Shakespearian dialogue.	13	Experimental	—	Python
116	ayus1234/Text-Generation-with-GPT-2 A comprehensive toolkit for fine-tuning GPT-2 language models and generating...	13	Experimental	—	Python
117	taljindergill78/AI-Indian-Recipe-Generator AI-powered system that generates authentic Indian recipes using GPT-2 and...	13	Experimental	—	Python
118	eshaaaan/tinygpt 🤖 Simplify understanding of large language models with TinyGPT, featuring a...	13	Experimental	—	Python
119	Wojtekb30/GPT-2-B200-pre-trainier Code for pre-training a GPT-2 model on (eight) NVIDIA DGX B200 GPUs and...	13	Experimental	—	Python
120	thewh1teagle/g2p-byt5 g2p with byt5	13	Experimental	—	Python
121	jaydeepthik/Nano-GPT Simple GPT with multiheaded attention for char level tokens, inspired from...	12	Experimental	7	Jupyter Notebook
122	blackbird71SR/CharacterLLMs Character Level Large Language Models including Transformer based nano GPT...	12	Experimental	5	Jupyter Notebook
123	brendandagys/ChadGPT From-scratch GPT experiments in PyTorch, covering attention mechanisms,...	11	Experimental	—	Jupyter Notebook
124	oskarvanderwal/sinterklaas-keybasebot Code for running a simple Keybase bot that generates Sinterklaas poems on command.	11	Experimental	—	Python
125	somosnlp/gpt-2-spanish Pre-train a Spanish GPT-2 model from scratch using the Spanish OSCAR dataset.	11	Experimental	3	—
126	Asura3301/nanogpt2 Replica of OpenAI GPT-2 model from scratch using PyTorch \| Personal Projects	11	Experimental	—	Jupyter Notebook
127	sathishkumar67/GPT2-Turbo Implementation of GPT2-Turbo	11	Experimental	—	Python
128	mfaizan-ai/gpt_nano Implementation of lightweight transformer model for character level text generation	11	Experimental	4	Python
129	LuluW8071/Building-LLM-from-Scratch GPT-2 Pre-Training from scratch with Flash Attention	11	Experimental	1	Jupyter Notebook
130	MecBonjourno/GPT-Ignition GPT - Ignition: Understanding, Studying & Mastering AI, GPT, LLMs, AGENTS,...	11	Experimental	3	Jupyter Notebook
131	LEv145/rugpt-docker-compose Docker compose for ru-gpts	11	Experimental	4	Python
132	mklblm/Study-NanoGPT A personal repo for the studying, implementing and training of Andrej...	11	Experimental	—	Jupyter Notebook
133	alexrosen45/IpsumGPT A modular implementation of a causal transformer with scaled dot-product...	11	Experimental	—	Python
134	PranavKohli24/GPT_Arch Implementation of GPT Neural network transformer Architecture	11	Experimental	2	Jupyter Notebook
135	Pranavh-2004/GPT-From-Scratch Exploring transformers by building a GPT model from scratch using nanoGPT,...	11	Experimental	—	Jupyter Notebook
136	bhuvanchennoju/GPT-from-scratch This repo contains a implementation of small scaled GPT styled LLM, and...	11	Experimental	—	Jupyter Notebook
137	abir9hassini/Generating-New-Recipes-using-GPT The project is about generating new recipes, the recipes are split into...	11	Experimental	—	Jupyter Notebook
138	nikhilgrad/GPT-from-Scratch This is a basic version of a GPT (Generative Pretrained Transformer) model...	11	Experimental	—	Jupyter Notebook
139	2003HARSH/Transformer-based-Decoder-only-Language-Model This repository explores building a character-level transformer decoder in...	11	Experimental	—	Jupyter Notebook
140	Nishant2018/Text-Generation-GPT-1-and-GPT-2 Generative Pre-trained Transformer 1 and Generative Pre-trained Transformer 2 models	11	Experimental	3	Jupyter Notebook
141	youshitsune/shakespeare This is AI that generates Shakespeare alike text.	11	Experimental	—	Python
142	nikhil-xb/WillyGPT A GPT2 model made from scratch on PyTorch (Inspired by Andrej Karpathy)	11	Experimental	—	Python
143	azizbarank/Dutch-GPT-2-Text-Generation Using the Dutch version of the GPT-2 model that was obtained by retraining...	11	Experimental	—	Python
144	jaugusto/gpt-neo-exploring A serie of notebooks exploring GPT-Neo	11	Experimental	—	Jupyter Notebook
145	harpertoken/models retrain GPT with updates.	11	Experimental	—	Python
146	sidharth72/GPT-124M-CPP A Reproduction of GPT-2 (124M) using C++	11	Experimental	1	Jupyter Notebook
147	mihaicristianfarcas/GPT Generative Transformer built from scratch using PyTorch.	11	Experimental	—	Jupyter Notebook
148	iamb0ttle/GPT-2-Fine-Tuning OpenAI's GPT-2 Fine Tunning Code	11	Experimental	—	Jupyter Notebook
149	manya-imran/Generative-Pre-trained-Transformers This publication is a project that covers the use of transformers and prompt...	10	Experimental	2	—

Comparisons in this category

gpt-neox and gpt-neo (58 vs 47) gpt-tfjs and WebGPT (50 vs 44)