luminati-io/Awesome-Web-Scraping
A list of libraries, tools, and APIs for web scraping and data processing. Find everything you need for extracting, managing, and processing data from the web, from HTTP libraries to browser automation tools and proxy services.
This collection helps you extract specific information from websites, transforming unstructured web content into organized, ready-to-use data. It provides the tools and guides needed to overcome common hurdles like anti-bot measures and CAPTCHAs. Anyone who needs to gather large amounts of publicly available data from the web for analysis, research, or business intelligence will find this useful.
No commits in the last 6 months.
Use this if you need to programmatically collect data from multiple websites to build datasets for market research, competitor analysis, lead generation, or academic studies.
Not ideal if you're looking for a simple point-and-click tool or a fully managed service that does the scraping for you without any coding.
Stars
13
Forks
6
Language
—
License
—
Category
Last pushed
Apr 06, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/luminati-io/Awesome-Web-Scraping"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.