vishwajeetdabholkar/eGet-Crawler-for-ai
Web scraping framework built for AI applications. Extract clean, structured content from any website with dynamic content handling, markdown conversion, and intelligent crawling capabilities. Perfect for RAG applications and AI training data pipelines. Features async processing, browser management, and Prometheus monitoring.
This tool helps AI developers and data scientists automatically extract clean, structured information from any website, even those with dynamic content. It takes website URLs as input and provides well-formatted text (like markdown), HTML, and structured data ready for use in AI models, such as for training or as context for Retrieval Augmented Generation (RAG) applications.
No commits in the last 6 months.
Use this if you need to build comprehensive knowledge bases or collect high-quality training data for AI applications by extracting content from a variety of websites, including those with complex layouts or dynamic elements.
Not ideal if you only need to extract data from a few static web pages or if you don't have experience integrating with APIs for data ingestion.
Stars
53
Forks
19
Language
Python
License
Apache-2.0
Category
Last pushed
Sep 09, 2025
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/vishwajeetdabholkar/eGet-Crawler-for-ai"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...