BjornMelin/ai-docs-vector-db-hybrid-scraper
Retrieval-augmented docs ingestion stack: Firecrawl + Crawl4AI + Qdrant vector search with FastAPI and MCP interfaces for AI engineers.
This is an ingestion and retrieval system that helps AI engineers build reliable applications requiring up-to-date information. It takes documentation from various web sources, processes it into searchable data, and makes it available for AI agents and applications. The system outputs both dense and sparse embeddings, providing robust search and generation capabilities for RAG workflows.
Use this if you are an AI engineer who needs to create dependable data pipelines for AI models that query and generate content based on extensive, current documentation.
Not ideal if you are looking for a simple, off-the-shelf chatbot or a solution that doesn't require deep integration into an AI engineering workflow.
Stars
10
Forks
3
Language
Python
License
MIT
Category
Last pushed
Jan 27, 2026
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/BjornMelin/ai-docs-vector-db-hybrid-scraper"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...