spider-rs/spider-py
Spider ported to Python
This tool helps web developers quickly gather all the links from a website, which is essential for tasks like site auditing, content analysis, or building custom search indexes. You provide a website address, and it efficiently returns a comprehensive list of all discovered URLs. This is ideal for developers needing fast and scalable web data collection.
104 stars.
Use this if you are a web developer who needs a high-performance tool to programmatically extract all links from large websites using Python.
Not ideal if you need to extract specific data fields from web pages (like product prices or article authors) rather than just links, or if you prefer a no-code solution.
Stars
104
Forks
16
Language
Rust
License
MIT
Category
Last pushed
Jan 20, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/spider-rs/spider-py"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
seleniumbase/SeleniumBase
APIs for browser automation, testing, and bypassing bot-detection.
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers....
intoli/user-agents
A JavaScript library for generating random user agents with data that's updated daily.
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In...
Kaliiiiiiiiii-Vinyzu/patchright
Undetected version of the Playwright testing and automation library.