m3dev/kannon
Kannon is a wrapper for the gokart library that allows gokart tasks to be easily executed in a distributed and parallel manner on multiple kubernetes jobs.
This tool helps data engineers and ML engineers execute complex, multi-step data processing or machine learning workflows more efficiently. It takes definitions of tasks and their dependencies, written in Python, and orchestrates their parallel execution across a Kubernetes cluster. The output is a completed workflow, with each task's results stored as defined.
No commits in the last 6 months. Available on PyPI.
Use this if you need to run large-scale data pipelines or machine learning training jobs that involve many interconnected steps and require distributed processing on Kubernetes.
Not ideal if your tasks are simple, don't require parallel execution, or you are not already using Kubernetes for your computational workloads.
Stars
26
Forks
3
Language
Python
License
MIT
Category
Last pushed
Jan 17, 2025
Commits (30d)
0
Dependencies
2
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/mlops/m3dev/kannon"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
skypilot-org/skypilot
Run, manage, and scale AI workloads on any AI infrastructure. Use one system to access & manage...
dstackai/dstack
dstack is an open-source control plane for running development, training, and inference jobs on...
ray-project/kuberay
A toolkit to run Ray applications on Kubernetes
kubeflow/kale
Kubeflow’s superfood for Data Scientists
volcano-sh/volcano
A Cloud Native Batch System (Project under CNCF)