pzaino/thecrowler
A Content Discovery and Development Platform. Empowering Cybersecurity, AI, Marketing, and Finance professionals and researchers to discover, analyze, and interact with the web in all its dimensions.
The CROWler helps cybersecurity researchers, intelligence teams, and marketers gather specific information from the web. It takes your defined rules for what to look for and how to interact with websites, then uses real browsers to crawl, scrape data, and detect specific technologies or vulnerabilities. The output is structured data and intelligence tailored to your investigation, perfect for professionals needing deep web analysis and full control over their data collection.
Use this if you need to perform advanced, highly customizable web crawling, data extraction, and intelligence gathering for cybersecurity, market research, or competitive analysis, requiring full ownership and auditability of your data.
Not ideal if you need a simple, point-and-click solution for basic web scraping without advanced detection or self-hosting requirements.
Stars
52
Forks
11
Language
Go
License
Apache-2.0
Category
Last pushed
Mar 29, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/perception/pzaino/thecrowler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python.
Altimis/Scweet
A simple and unlimited twitter scraper : scrape tweets, likes, retweets, following, followers,...
lexiforest/curl_cffi
Python binding for curl-impersonate fork via cffi. A http client that can impersonate browser...
plabayo/rama
modular service framework to move and transform network packets
scrapinghub/spidermon
Scrapy Extension for monitoring spiders execution.