crackcell/hpipe
Workflow engine for various computing systems.
This tool helps data professionals manage complex data processing tasks, especially those involving large datasets for things like ad click-through rate predictions. It takes in various data processing jobs and orchestrates them into a unified workflow, providing status tracking and the ability to resume from where it left off. Data engineers, data scientists, and operations engineers who work with big data pipelines will find this useful.
No commits in the last 6 months.
Use this if you need to integrate and manage different types of data processing jobs, particularly those using Hadoop Streaming or Hive, into a reliable, trackable workflow.
Not ideal if your data processing tasks are simple, isolated, or do not involve large-scale distributed computing systems.
Stars
26
Forks
9
Language
Go
License
GPL-2.0
Category
Last pushed
Mar 30, 2017
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/data-engineering/crackcell/hpipe"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
rocketlaunchr/dataframe-go
DataFrames for Go: For statistics, machine-learning, and data manipulation/exploration
Xenios91/Byte-Chomp
A Golang tool for obtaining data on Golang binaries in csv format
MaxHalford/tuna
:fish: A streaming ETL for fish
yaricom/dfutils
The collection of utilities to work with QFrame implementation of the data frames in GO language...