arkeodev/scraper
RAG-based Web Scraping
This tool helps you quickly extract information from websites by automatically scraping their content and then letting you ask natural language questions about it. You input a website URL and your question, and it provides a concise answer based on the site's content. It's ideal for market researchers, content strategists, or anyone needing to get specific answers from web pages without manually reading through them.
No commits in the last 6 months.
Use this if you need to quickly understand the key details from a specific webpage or collection of pages by asking questions, rather than sifting through all the text yourself.
Not ideal if you need to extract highly structured data like tables or product listings, or if you require extensive, deep analysis of website code or components.
Stars
14
Forks
2
Language
Python
License
MIT
Category
Last pushed
Jul 22, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/rag/arkeodev/scraper"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
any4ai/AnyCrawl
AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts...
kreuzberg-dev/html-to-markdown
High performance and CommonMark compliant HTML to Markdown converter. Maintained by the...
ScrapeGraphAI/Scrapegraph-ai
Python scraper based on AI
adbar/trafilatura
Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping,...
paulpierre/markdown-crawler
A multithreaded 🕸️ web crawler that recursively crawls a website and creates a 🔽 markdown file...