testdrivenio/concurrent-web-scraping
Building a Concurrent Web Scraper with Python and Selenium
This project helps operations engineers, data analysts, or marketers quickly gather information from many web pages. It takes a list of URLs and efficiently extracts specific data points, delivering structured information ready for analysis or reporting. It's designed for anyone needing to collect large amounts of publicly available web data.
No commits in the last 6 months.
Use this if you need to rapidly collect data from numerous websites, such as product prices, job postings, or news articles, for competitive analysis, market research, or data aggregation.
Not ideal if you only need to scrape a few pages, as the setup for concurrent scraping might be more complex than necessary, or if you require interaction with dynamic elements on a page that are not explicitly covered by the existing scripts.
Stars
33
Forks
27
Language
HTML
License
—
Category
Last pushed
Dec 22, 2021
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/testdrivenio/concurrent-web-scraping"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.