lorien/awesome-web-scraping
List of libraries, tools and APIs for web scraping and data processing.
This is a curated collection of resources to help you gather information from websites automatically. It provides lists of software, services, and how-to guides for extracting data from web pages. Anyone needing to collect publicly available data from the internet for analysis, competitive intelligence, or research would find this useful.
7,822 stars. Actively maintained with 11 commits in the last 30 days.
Use this if you are looking for specific tools or methods to programmatically extract information from various websites.
Not ideal if you need a fully automated, managed data feed without any technical involvement or manual setup.
Stars
7,822
Forks
877
Language
Makefile
License
—
Category
Last pushed
Mar 20, 2026
Commits (30d)
11
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/lorien/awesome-web-scraping"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.