atymri/WebCrawler
WebCrawler is a C# console application that recursively scans a website starting from a given URL, collects all discovered links, and saves them to a file. It’s useful for site mapping, link analysis, and content discovery.
This tool helps web administrators, SEO specialists, or content auditors map out all the links on a website. You provide a starting URL, and it systematically explores all pages, collecting every discovered link. The output is a simple text file listing all the valid URLs found across the site.
Use this if you need a comprehensive list of all URLs on a specific website for site audits, content inventory, or identifying broken links.
Not ideal if you need to extract specific data from web pages (like product prices or article content) rather than just a list of links.
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/atymri/WebCrawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.