postmodern/spidr
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
Spidr is a versatile Ruby library that helps developers navigate and extract information from websites programmatically. You provide it with a starting URL or domain, and it systematically follows links to gather details like page content, links, and HTTP headers. This is ideal for developers building tools for web scraping, site mapping, or content indexing.
836 stars.
Use this if you are a Ruby developer needing to automate the crawling of websites to collect data or analyze site structure.
Not ideal if you need a no-code solution or are not comfortable writing Ruby code to interact with web data.
Stars
836
Forks
108
Language
Ruby
License
MIT
Category
Last pushed
Jan 12, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/postmodern/spidr"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.