ivan-sincek/scrapy-scraper
Web crawler and scraper based on Scrapy and Playwright's headless browser.
This tool helps you collect information from websites, even those that load content dynamically with JavaScript. You provide a starting URL or a list of URLs, and it will browse the sites, extract links and content, and can even save screenshots or JavaScript files. It's designed for anyone needing to systematically gather data from the web, like competitive intelligence analysts or researchers.
Available on PyPI.
Use this if you need to extract structured data from many websites, including modern web applications that heavily rely on JavaScript to display their content.
Not ideal if you only need to download static files or scrape simple, non-JavaScript-heavy pages, as simpler tools might be sufficient and faster.
Stars
17
Forks
6
Language
Python
License
MIT
Category
Last pushed
Mar 10, 2026
Commits (30d)
0
Dependencies
9
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/ivan-sincek/scrapy-scraper"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.