ScrapingBee/scrapy-scrapingbee
JavaScript support and proxy rotation for Scrapy with ScrapingBee.
This tool helps web scraping developers extract data from websites that use JavaScript or have strong anti-bot measures. It takes a list of URLs and, through integration with the ScrapingBee API, provides the full, rendered HTML content of those pages, bypassing common scraping hurdles. Web scraping engineers, data scientists, or anyone building data collection systems would use this to reliably get data from complex websites.
No commits in the last 6 months. Available on PyPI.
Use this if you are a Python developer building a web scraper with Scrapy and need to handle JavaScript-heavy sites or require robust proxy rotation without managing it yourself.
Not ideal if you are not using Scrapy for your web scraping project or if you prefer to manage proxies and headless browsers manually.
Stars
93
Forks
6
Language
Python
License
—
Category
Last pushed
May 14, 2024
Commits (30d)
0
Dependencies
1
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/ScrapingBee/scrapy-scrapingbee"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.