ruimalheiro/training-custom-llama
Llama-style transformer in PyTorch with multi-node / multi-GPU training. Includes pretraining, fine-tuning, DPO, LoRA, and knowledge distillation. Scripts for dataset mixing and training from scratch.
This project helps machine learning engineers and researchers train custom large language models (LLMs) from scratch or fine-tune existing ones. You can input various text datasets and configuration settings, then output a specialized LLM ready for deployment. It's designed for individuals and teams working with large-scale NLP tasks and requiring high-performance computing.
Use this if you need to build or adapt a Llama-style language model using your own datasets and require multi-node or multi-GPU training capabilities.
Not ideal if you're looking for a low-code solution to apply pre-trained models without needing to customize architecture, training methods, or manage distributed computing infrastructure.
Stars
21
Forks
3
Language
Python
License
Apache-2.0
Category
Last pushed
Mar 10, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/ruimalheiro/training-custom-llama"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
unslothai/unsloth
Fine-tuning & Reinforcement Learning for LLMs. 🦥 Train OpenAI gpt-oss, DeepSeek, Qwen, Llama,...
huggingface/peft
🤗 PEFT: State-of-the-art Parameter-Efficient Fine-Tuning.
modelscope/ms-swift
Use PEFT or Full-parameter to CPT/SFT/DPO/GRPO 600+ LLMs (Qwen3.5, DeepSeek-R1, GLM-5,...
oumi-ai/oumi
Easily fine-tune, evaluate and deploy gpt-oss, Qwen3, DeepSeek-R1, or any open source LLM / VLM!
linkedin/Liger-Kernel
Efficient Triton Kernels for LLM Training