sigoden/rag-crawler
Crawl a website to generate knowledge file for RAG
This tool helps you quickly gather all the text content from a website or a GitHub repository containing Markdown files. It takes a starting URL and produces a structured knowledge file (JSON or separate Markdown files) of the site's content. This is useful for anyone building or managing knowledge bases, documentation, or AI-powered assistants that need up-to-date information from web sources.
No commits in the last 6 months. Available on npm.
Use this if you need to extract and organize web content into a usable format for building a search engine, a chatbot's knowledge base, or an internal documentation portal.
Not ideal if you need to extract complex data structures, interact with web forms, or require advanced browser automation for your content gathering.
Stars
50
Forks
11
Language
TypeScript
License
MIT
Category
Last pushed
Apr 03, 2025
Commits (30d)
0
Dependencies
6
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/sigoden/rag-crawler"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Related tools
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...