any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.
AnyCrawl helps you gather large amounts of information from websites and search engines, turning unstructured web content into organized, ready-to-use data. It can crawl an entire website or extract specific details from individual pages and search engine results, making it ideal for market researchers, content strategists, or data analysts.
2,763 stars. Actively maintained with 29 commits in the last 30 days.
Use this if you need to systematically collect web pages, extract specific data fields, or get structured search results from multiple engines for analysis or feeding into AI models.
Not ideal if you only need to manually browse a few websites or are looking for a simple, point-and-click scraping tool without any technical setup.
Stars
2,763
Forks
289
Language
TypeScript
License
MIT
Category
Last pushed
Mar 08, 2026
Commits (30d)
29
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/any4ai/AnyCrawl"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...
lightfeed/extractor
Using LLMs and AI browser automation to robustly extract web data