chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS
This project demonstrates how to build and automate an ETL pipeline written in Python and schedule it using open source Apache Airflow orchestration tool on AWS EC2 instance.
This project helps data professionals automate the extraction, transformation, and loading of data from web APIs into cloud storage. It takes raw JSON data from an API, performs basic cleaning, and stores it in Amazon S3 buckets. This is ideal for data engineers or analysts who need to set up reliable, scheduled data feeds for reporting or further analysis.
No commits in the last 6 months.
Use this if you need to regularly pull data from a web API, clean it slightly, and store it in an AWS S3 data lake for subsequent use.
Not ideal if your data sources are not web APIs, you don't use AWS, or your data transformation needs are highly complex.
Stars
19
Forks
6
Language
Python
License
—
Category
Last pushed
Aug 21, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/data-engineering/chayansraj/Python-ETL-pipeline-using-Airflow-on-AWS"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
elyra-ai/pipeline-editor
Common pipeline-editor components used in different clients (e.g. Elyra application, Web browser...
orchest/orchest
Build data pipelines, the easy way 🛠️
stitchfix/hamilton
A scalable general purpose micro-framework for defining dataflows. THIS REPOSITORY HAS BEEN...
Codex56799/dataengineering
🚀 Build a containerized data engineering workflow for NYC Yellow Taxi Trip Data using Apache...
msmenegol/datapark
Datapark: a self-hosted data platform