clabrugere/scratch-llm
Implements a LLM similar to Meta's Llama 2 from the ground up in PyTorch, for educational purposes.
This project offers a clear, basic implementation of a large language model like Meta's Llama, built using PyTorch. It helps developers and researchers understand how these models work internally by showing the mechanics of components like positional encoding and attention. The project takes text data, processes it, and demonstrates the core computational steps that lead to a trained language model.
No commits in the last 6 months.
Use this if you are a machine learning engineer or researcher who wants to learn the fundamental building blocks and internal workings of a Llama-like large language model from scratch, without optimization complexities.
Not ideal if you need a high-performance, production-ready language model for real-world applications or require advanced training and inference optimizations.
Stars
38
Forks
9
Language
Python
License
MIT
Category
Last pushed
Feb 07, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/clabrugere/scratch-llm"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
facebookresearch/LayerSkip
Code for "LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding", ACL 2024
FareedKhan-dev/train-llm-from-scratch
A straightforward method for training your LLM, from downloading data to generating text.
kmeng01/rome
Locating and editing factual associations in GPT (NeurIPS 2022)
datawhalechina/llms-from-scratch-cn
仅需Python基础，从0构建大语言模型；从0逐步构建GLM4\Llama3\RWKV6， 深入理解大模型原理