gpustack/gguf-packer-go
Deliver LLMs of GGUF format via Dockerfile.
This project helps machine learning engineers and MLOps professionals efficiently package and distribute Large Language Models (LLMs) in the GGUF format. You provide your base LLM, a Dockerfile describing how to quantize and configure it, and get out a ready-to-deploy Docker image containing your optimized LLM. This streamlines the process of getting LLMs from development to production environments.
No commits in the last 6 months.
Use this if you need a robust, standardized way to package and distribute your GGUF-formatted LLMs using existing Docker workflows and cloud-native practices.
Not ideal if you prefer using Ollama's native Modelfile format or do not use Docker for your model deployment.
Stars
15
Forks
4
Language
Go
License
MIT
Category
Last pushed
Oct 24, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/gpustack/gguf-packer-go"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
ModelCloud/GPTQModel
LLM model quantization (compression) toolkit with hw acceleration support for Nvidia CUDA, AMD...
intel/auto-round
🎯An accuracy-first, highly efficient quantization toolkit for LLMs, designed to minimize quality...
pytorch/ao
PyTorch native quantization and sparsity for training and inference
bodaay/HuggingFaceModelDownloader
Simple go utility to download HuggingFace Models and Datasets
NVIDIA/kvpress
LLM KV cache compression made easy