Data Pipeline Frameworks Data Engineering Tools

Prefect is a workflow orchestration framework for building resilient data...

Open Source Feature Flags, Experimentation, and Product Analytics

Transform, query, and download geospatial data on the web.

Python ETL framework for stream processing, real-time analytics, LLM...

An orchestration platform for the development, production, and observation...

data load tool (dlt) is an open source Python library that makes data...

Meltano: the declarative code-first data integration engine that powers your...

locopy: Loading/Unloading to Redshift and Snowflake using Python.

Hop Orchestration Platform

Apache Superset is a Data Visualization and Data Exploration Platform

The leading data integration platform for ETL / ELT data pipelines from...

Clean APIs for data cleaning. Python implementation of R package Janitor

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability,...

The Public Utility Data Liberation Project provides analysis-ready energy...

Change data capture for a variety of databases. Please log issues at...

Quilt is a Scientific Data Management Platform on AWS that helps teams and...

ingestr is a CLI tool to copy data between any databases with a single...

Apache DevLake is an open-source dev data platform to ingest, analyze, and...

A dbt adapter for Databricks.

Egeria core

Flink CDC is a streaming data integration tool

Blazing-fast Data-Wrangling toolkit

OLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg or Plain...

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp...

A Spark plugin for reading and writing Excel files

Support agile DataOps Based on Flink, DataX and Flink-CDC, Chunjun with Web-UI

A high-performance observability data pipeline.

Database Reporting Tool and Tasks (.Net)

Dataform is a framework for managing SQL based data operations in BigQuery

Know your data better！Datavines is Next-gen Data Observability Platform,...

A fast and versatile ETL tool that can transfer data between RDBMS and NoSQL...

Data Contracts engine for the modern data stack. https://www.soda.io

Data pipelines for cloud config and security data. Build cloud asset...

Event streaming platform for agents, apps, and analytics. Continuously...

A local-first workflow engine built the way it should be: declarative,...

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream,...

Relational data pipelines for the science lab

The leader in Customer Data Infrastructure

Fast, Simple and a cost effective tool to replicate data from Postgres to...

lakeFS - Data version control for your data lake | Git for data

ApeCloud's Data Transfer Suite, written in Rust. Provides ultra-fast data...

A wrapper around the python-irodsclient to allow for easy interaction with...

Scalable and efficient data transformation framework - backwards compatible with dbt.

visual data prep powered by python

Wikidata and Wiktionary language data extraction

Automation fetching data for Vietlott. Just for fun.

The premier open source Data Quality solution

The dbt-native data observability solution for data & analytics engineers....

A utility library for comparing and synchronizing different datasets.

🎲 Business Logic Code in a flow!

Consolidate, enrich and republish the data gathered by warn-scraper

Sling is a CLI tool that extracts data from a source storage/database and...

⚡ Fastest SQL ETL pipeline in a single C++ binary, built for stream...

Snowflake Snowpark Python API

Command-line interface for downloading WARN Act notices of qualified plant...

Server application to serve U.S. federal spending data via a RESTful API

Desbordante is a high-performance data profiler that is capable of...

Open-source data framework for biology. Context and memory for datasets and...

dbt-sugar is a CLI tool that allows users of dbt to have fun and ease...

Inspection of tabular (csv, xls-like) files to guess the columns' content

ReplicaDB is open source tool for database replication, designed for...

Conduit streams data between data stores. Kafka Connect replacement. No JVM required.

Fancy stream processing made operationally mundane

A collection of tools for extracting FHIR resources and analytics services...

The best place to learn data engineering. Built and maintained by the data...

为DataX(https://github.com/alibaba/DataX) 提供远程多语言调用（ThriftServer，HttpServer）...

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

log data pre processing, generation and shipping in python

The Fabric Metadata-Driven Framework (FMD) is a cutting-edge accelerator...

First open-source data discovery and observability platform. We make a life...

A new Airflow Provider for Fivetran, maintained by Astronomer and Fivetran

A Python package to load complex XML files into a relational database

A curated list of awesome system integration software and resources.

PyAirbyte brings the power of Airbyte to every Python developer.

Official Neo4j JDBC Driver

Contribute to dlt verified sources 🔥

go-etl is a toolset for data extraction, transformation and loading.

DataOps Data Quality TestGen is part of DataKitchen's Open Source Data...

Lightweight REST for Java

Home of the Open Data Contract Standard (ODCS).

LinkedPipes ETL is an RDF based, lightweight ETL tool

One framework to develop, deploy and operate data workflows with Python and SQL.

Community-driven, simple, yet powerful framework for fast, cost-effective...

Core package of the Metafacture tool suite for metadata processing.

A COBOL parser and Mainframe/EBCDIC data source for Apache Spark

Configurable Extract, Transform, and Load

Zero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No...

Data Engineering Zoomcamp is a free 9-week course on building...

Privacy and Security focused Segment-alternative, in Golang and React

glue-utils makes AWS Glue jobs less repetitive, more type-safe, and easier...

Community supported integrations for the Dagster platform.

TAK Compatible, browser based Common Operation Picture & Situational Awareness tool

Frontend & BFF (Backend for frontend) for Olake. This includes the UI code...

Monopoly is a Python library & CLI that converts bank statement PDFs to CSV.

Drop-in replacement for Apache Spark UI

Declarative text based tool for data analysts and engineers to extract,...

A deterministic orchestrator for composable micro-workflows with reusable modules

High Performance Storage and Streaming Solution for Data Acquisition Systems

Data Pipeline Automation Framework to build MCP servers, data APIs, and data...

Documentation repository for the Egeria project.

SimTradeData is a utility library supporting SimTradeDesk, SimTradeLab and...

The Boring query platform - Connect and query anything

A package supporting the conversion from Synthea CSV to OMOP CDM

Unified Data Foundation with Microsoft Fabric with Options to Integrate with...

In-memory Java DataFrame library

A simple and easy to use Data Validation library for Python.

Next generation compute platform for the post-modern data stack

The Open-Source Enterprise Data Platform in a single Portal

GovHub - Transformando Dados em Valor para Gestão Pública

A lightweight ETL (extract, transform, load) library and data integration...

Open Source Business Intelligence with Malloy Semantic Layer :tada:

🥪🦘 An open source sandbox project exploring dbt workflows via a fictional...

OpenSnowcat Enricher (Apache 2.0 License)

Powerful, developer-experience centric, blazingly fast and extensible job...

Helm charts

Tiozin, your friendly ETL framework

Flowfile is a visual ETL tool and Python library combining drag-and-drop...

Ingestão, preparação e disponibilização gratuita de dados de CNPJs de...

Library and CLI for profiling tabular data

(Spatial) data harmonisation with hale»studio (formerly HUMBOLDT Alignment Editor)

The data-validation toolkit for enhanced dbt (data build tool) PR review

Representing BIM Data as Parquet

Installer for DataKitchen's Open Source Data Observability Products. Data...

Home of the Open Data Product Standard (ODPS).

Use @xonsh wherever you go through the SSH without installation on the host.

Addax Admin is a web-based management console for Addax ETL jobs, offering...

Data Engineering Pilipinas is a community for data engineers, data analysts,...

Dataform Tools - VS Code extension to run and visualise Dataform data...

RKI Metadata Exchange | Software development toolkit for the MEx project...

end-to-end data engineering project to get insights from PyPi using python,...

Transformation, web frontend, and API for the hbz catalog as LOD

A high-performance rules engine for IFTTT-style automation in Rust with...

Pipeline de Dados do Gov-Hub

bboss-datatran 由 bboss 开源的数据采集&流批一体化工具，提供数据采集、清洗转换处理以及流批一体化计算功能；...

SeaTunnel Web is a visual platform for building, managing, and monitoring...

End-to-end NFL data pipeline that scrapes PFF grades and Pro Football...

Complete structured archive of every CIA World Factbook edition from...

A set of Data Engineering tools online for public use

Pull and standardize data on cloud compute resources.

High-performance streaming SQL query engine designed for real-time data...

Backend service for Scribe data downloads

Quality python data sources for pyspark 4.x

A Python library that acts as a client to download, pre-process and...

A tool for converting FERC filings published in XBRL into SQLite databases

Indonesian Stock Exchange API wrapper for trading data integration.

A comprehensive map of companies that hire for tech jobs in Greece.

One ETL tool to rule them all

Git all remotes. git cli tool that targets both Github and Gitlab

Apache DevLake is an open-source dev data platform to ingest, analyze, and...

Dagster Labs' open-source data platform, built with Dagster.

A machine-readable, human-editable database of the Yu-Gi-Oh! Trading Card...

Watchmen Platform is a low code data platform for data pipeline, meta data...

A lightweight Go framework for building stateful, real-time data pipelines....

In this repository, you will find varies demo and presentations I have...

Data transformation framework for LinkML data models

Resilient data pipeline framework running on Apache Spark

OpenSnowcat Collector, an open source fork of Snowplow (Apache 2.0 License)

A collection of tools for the BNA.

Stop Guessing. Start Dominating Your Market. The only data platform built...

Airflow extensions for communicating with Wherobots Cloud

opendataworks...

Powerful, whiteboard-style ETL

Focusing on building industry-leading ETL engines.

Embedded workflow orchestration library for Rust and Python. Build...

An end-to-end, serverless AI platform built on Google Cloud that...

Stay up to date with hiero organisation activity and contributor diversity

Professional forex tick data preprocessing with unified DuckDB storage,...

Harvesting & Semantic search for open data portals

Build scalable data pipelines on YTsaurus with automatic stage management,...

This project hosts scripts to generate flat tables used for reporting purposes.

Categorical Query Language IDE

An ML-first temporal graph of Bitcoin's on-chain fund flows.

Frontend for Syncmaster, no-code ETL tool. WIP

Streamlit Connection to Explore Foreign Currency Exchange rates 💰 in real-time

DevOps Data Application Platform...

Dimensional Data Pipeline & Semantic Data Engineering Framework

A python package to create a database on the platform using our moj data...

Open-source data contract enforcement — define, sync dbt, validate, block,...

DataOps Observability is part of DataKitchen's Open Source Data...

ETL / ELT Framework powered by DuckDB, designed to seamlessly integrate and...

JSONCleanseETL是一款专业的数据清洗和转换工具，旨在为用户提供高效处理JSON格式数据的解决方案。...

Parse judgements from the Transformation Engine and load them into MarkLogic...

Converts AsyncApi and JsonSchema to PyArrow schema

Data Cleaning for Pyspark

An automatically-updated collection of wikitexts from Yugipedia. Part of YAML Yugi.

Public repo of Beyond Finance's technical assessment for Data Engineering candidates

VS Code extension for visualizing SQL Server database object dependencies...

No-code ETL tool, based on onETL + PySpark

Extract transform load CLI tool for extracting small and middle data volume...

Codebase for CCAO data infrastructure construction and management

Go client for Chalk

Convert your audio files into DuckDB or Parquet files

A minimal zero dependency dataframe library

Hands-on Spark internals and performance engineering.

End-to-end data warehouse exercises for students - build a modern ELT...

This repository is a place for the Data Warehousing course at the...

DataOps Observability Integration Agents are part of DataKitchen's Open...

Packages for the Tenzir ecosystem.

In-process virtual filesystem with hard quota for Python

Fetchez is a lightweight, modular, and highly extendable Python framework...

Códigos de captura e tratamento de dados da SMTR

The High-Performance Analytics Engine — Free for Personal Use

Data quality validation, profiling, anomaly detection, and YAML-driven...

Library for data quality monitoring based on duckdb.

A self-updating global dashboard that aggregates 40+ open data sources...

RushDB is an Instant Database for Modern Apps & AI. Built on top of Neo4j.

Cool DE Projects

(ETL) Extract, transform, load/download and augment images and annotations...

Airflow provider package for SAP HANA

Open-source workflow engine. Orchestrate and observe computational workflows...

Mox-Data.com is a cloud-based data ingestion tool used to process raw data...

Modern, AI-native and agentic Pythonic data transformation platform.

Official action to install Bruin CLI in Github Actions.

Open Data Stack Platform: a collection of projects and pipelines built with...

End-To-End Data Engineering Project. Made to learn some common data...

Mapping intelligence for enterprise data migrations: schema mapping,...

This site uses the amazing Astro.build project. I added **Google Docs** ...

Simple HWM Store backend

Little helper to write IAM policies

PHP ETL engine with pluggable steps: extractors, transformers, loaders

A versatile, high-performance Change Data Capture (CDC) engine built in...

A documentation-centric DuckDB based ETL tool, implementing transformations...

DW de e-commerce (Kimball/Star Schema) em SQL Server, com scripts, dados...

Basic ETL Entity classes for onETL

Library for describing data transformation pipelines by compositing simple...

tracebloc data pipeline for training/test dataset setup

Open Source Business Intelligence

Fine-grained control over analytics events and logs via remote configuration

simple lib of ETL building blocks

Airflow DAGS for migrating and managing ILS data into FOLIO along with other...

Horizon HWM Store for onETL

POC for real-time monitoring and alert system for home hemodialysis,...

🛠️ Streamline task management with this full-stack solution featuring...

Extensions to Kiba ETL

argo-kedro is a kedro-plugin for executing Kedro pipelines on Argo Workflows.

A resilient, fault‑tolerant telemetry analytics pipeline designed to...

Production-style financial data engineering pipeline that standardizes NSE...

Semantic cost-linting and performance warnings extension for Databricks in VS Code

Transform data with queries

PardoX: The Hyper-Fast Data Engine

Pipeline that extracts data from the Spotify API to build a more detailed...

A sophisticated cross-browser extension for bulk Google Classroom downloads,...

elevata is an Architecture Runtime for modern data platforms —...

A hash you can do algebra on — composable verification for ordered data over...

Enterprise-grade multimodal redaction runtime that detects and removes...

A simple, self-contained CLI for performance-focused data streaming & anonymization.

Projeto da API do primeiro semestre de 2026

Macro-aware, explainable equity analytics system using Bronze–Silver–Gold...

Deterministic key and join discovery for structured datasets

PHP ETL engine for building extract–transform–load pipelines with pluggable...

Data engineering demo project for Danish Parliament (Folketing) open data —...

A movie journal coupled with open IMDb data, and a Flask web-app for easy...

Big-data pipeline detecting wash trading, pump & dump, and spoofing in trade...

asyncIO, Github Actions, GCP, dbt, Terraform, Docker

Code Snippets, Designs, and other things about building a Data Analytics...

A data lineage tool detects table dependencies from rendered SQL statements.

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

Automated weekly tech trend reports — built with Claude Code + Claude Cowork

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

My Solutions to Danny Ma's 8 Week SQL Challenge — built in T-SQL on SQL Server

High-performance, declarative stream processor for delimited text data.