paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file for each page, designed for LLM RAG
This tool helps you quickly gather content from an entire website, converting each web page into a well-structured Markdown file. You provide a starting website URL, and it generates a collection of organized Markdown documents. It's designed for data scientists, AI/ML engineers, or content managers who need to prepare web data for AI model training or knowledge retrieval systems.
431 stars. No commits in the last 6 months. Available on PyPI.
Use this if you need to create a large, structured collection of web page content in Markdown format for training AI models, building knowledge bases for chatbots, or implementing retrieval-augmented generation (RAG).
Not ideal if you only need to extract small snippets of text, require real-time data streaming, or are looking for a tool to perform deep data analysis or analytics on the scraped content itself.
Stars
431
Forks
54
Language
Python
License
MIT
Category
Last pushed
Aug 13, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/paulpierre/markdown-crawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
lightfeed/extractor
Using LLMs and AI browser automation to robustly extract web data